Nutrition Monitoring: Data Serve Many Purposes, But Users Recommend
Improvements (Letter Report, 06/20/95, GAO/PEMD-95-15).

The National Nutrition Monitoring and Related Research Program consists
of a network of surveys, surveillance systems, and research activities
that serves various purposes.  It provides researchers and
decisionmakers with data for assessing the safety of the nation's food
supply, targeting food assistance to low-income families, and studying
the relationship between diet and disease. The program has been
criticized, however, for the lack of coordination among the various
activities and its poor coverage of populations at risk of nutritional
problems. GAO surveyed users of nutrition-monitoring data.  This report
(1) describes the users and the major uses of nutrition-monitoring data
and (2) summarizes user satisfaction with nutrition-monitoring
activities and the changes that users believe are likely to increase
their use of, or confidence in, the data.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  PEMD-95-15
     TITLE:  Nutrition Monitoring: Data Serve Many Purposes, But Users 
             Recommend Improvements
      DATE:  06/20/95
   SUBJECT:  Nutrition research
             Nutrition surveys
             Data collection operations
             Monitoring
             Food supply
             Data integrity
             Interagency relations
             Health research programs
             Demographic data
IDENTIFIER:  Special Supplemental Food Program for Women, Infants, and 
             Children
             USDA Continuing Survey of Food Intakes by Individuals
             Hispanic Health and Nutrition Examination Survey
             National Health and Nutrition Examination Survey
             HNIS Nationwide Food Consumption Survey
             National Health Interview Survey
             National Nutrition Monitoring and Related Research Program
             WIC
             HANES
             PHS Pregnancy Nutrition Surveillance System
             PHS Pediatric Nutrition Surveillance System
             PHS Navajo Health and Nutrition Survey
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Ranking Minority Member, Committee on Science, House of
Representatives

June 1995

NUTRITION MONITORING - DATA SERVE
MANY PURPOSES; USERS RECOMMEND
IMPROVEMENTS

GAO/PEMD-95-15

Nutrition Monitoring Data Serve Many Uses

(973403)


Abbreviations
=============================================================== ABBREV

  ARS - Agricultural Research Service
  BRFSS - Behavioral Rish Factor Surveillance Survey
  CDC - Centers for Disease Control and Prevention
  CSFII - Continuing Survey of Food Intake by Individuals
  DHKS - Diet and Health Knowledge Survey
  FDA - Food and Drug Administration
  HHANES - Hispanic Health and Nutrition Examination Survey
  HHS - Department of Health and Human Services
  HNS - Health and Nutrition Survey
  NCCDPHP - National Center for Chronic Disease Prevention and Health
     Promotion
  NCHS - National Center for Health Statistics
  NFCS - Nationwide Food Consumption Survey
  NHANES - National Health and Nutrition Examination Survey II and
     III
  NHEFS - National Health and and Nutrition Examination Survey I
  NHIS - National Health Interview Survey
  NNMRR - National Nutrition Monitoring and Related Research
  NNMRRP - National Nutrition Monitoring and Related Research Program
  NTIS - National Technical Information Service
  PedNSS - Pediatric Nutrition Surveillance System
  PHS - Public Health Service
  PNSS - Pregnancy Nutrition Surveillance System
  USDA - U.S.  Department of Agriculture
  WIC - Special Supplemental Food Program for Women, Infants, and
     Children

Letter
=============================================================== LETTER


B-260632

June 20, 1995

The Honorable George E.  Brown, Jr.
Ranking Minority Member
Committee on Science
House of Representatives

Dear Mr.  Brown: 

The National Nutrition Monitoring and Related Research Program
(NNMRRP) is a network of surveys, surveillance systems, and research
activities designed to serve multiple purposes.  It provides
researchers and decisionmakers with data for assessing the safety of
the nation's food supply, targeting food assistance to low-income
families, and studying the relationship between diet and disease,
among other uses.  However, past evaluations of federal nutrition
monitoring have criticized it on several counts, including the lack
of coordination among the various activities and its poor coverage of
populations at risk of nutritional problems.  Through the National
Nutrition Monitoring and Related Research Act of 1990 (P.L. 
101-445), the Congress established objectives for addressing these
problems. 

This report is the second in a series of three responding to your
request for information on the NNMRRP.  In our first report,
published last year, we detailed the activities that make up the
NNMRRP, the history of concerns about the data collection systems,
and agency progress toward meeting the objectives of the NNMRR Act.\1
In this report, we summarize the results of our survey of users of
nutrition monitoring data.  Specifically, the objectives of this
report are to (1) describe users and major uses of nutrition
monitoring data and (2) summarize the satisfaction of users with
selected nutrition monitoring activities and the changes that users
identified as likely to increase their use of or confidence in the
data.  The survey results presented here serve as a foundation for
our follow-up report on the features of a model nutrition monitoring
program.\2


--------------------
\1 Nutrition Monitoring:  Progress in Developing a Coordinated
Program (GAO/PEMD-94-23; May 27, 1994). 

\2 See GAO/PEMD-95-19, forthcoming. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

The data users who responded to our survey were located in a variety
of settings, including governmental, academic, and business.  These
users reported that they provide analyses to the general public as
well as to decisionmakers in their organizational settings and to
other audiences.  They also reported using nutrition monitoring data
for an extensive range of purposes, from identifying
nutrition-related problems and designing programs to address the
problems to informing basic research. 

Although most of the data users who responded to our survey were
satisfied with the degree to which the data meet their information
and data quality needs, a majority also suggested changes that would
increase their use of or confidence in the data.  Their
recommendations include the need for improved dietary intake methods,
more continuous data collection, better coverage of subpopulations
and small geographic areas, improved timeliness and documentation of
the data, and increased dissemination of the data in formats that
facilitate access and analysis. 


   BACKGROUND
------------------------------------------------------------ Letter :2

The U.S.  nutrition monitoring system has included more than 70
separate data collection activities conducted by several different
federal agencies.  Major components of the system include the
national health and nutrition surveys administered by the National
Center for Health Statistics (NCHS) within the Centers for Disease
Control and Prevention (CDC), the state-based surveillance systems
managed by the National Center for Chronic Disease Prevention and
Health Promotion (NCCDPHP), and national surveys operated by the
Agricultural Research Service (ARS).  Table 1 lists the data
collection activities addressed in our survey. 

Although the system has been praised for being comprehensive, it has
also been criticized for the redundancy of some of the monitoring
activities, the prolonged data collection and delays in data release,
the poor coverage of subpopulations, and the lack of compatibility in
data assessment and sampling methods across different surveys.\3 In
response to these concerns, the National Nutrition Monitoring and
Related Research Act of 1990 required the Secretaries of the
Departments of Agriculture (USDA) and Health and Human Services (HHS)
to implement a coordinated program of nutrition monitoring and
established an Interagency Board to facilitate the process.  The
Board developed preliminary plans for meeting the goals of the
legislation and published them in a 10-year comprehensive plan in
June 1993.\4



                                     Table 1
                     
                     Data Collection Systems in Our Survey\a

                                                  Time period
Agency      Data collection system                covered         Short name
----------  ------------------------------------  --------------  --------------
HHS/PHS/    National Health and Nutrition         1982-84, 1986,  NHEFS
CDC/NCHS    Examination Survey I:                 1987, 1992
            Epidemiological Follow-up

            National Health and Nutrition         1976-80         NHANES II
            Examination Survey II

            National Health and Nutrition         1988-94         NHANES III
            Examination Survey III

            Hispanic Health and Nutrition         1982-84         HHANES
            Examination Survey

            National Health Interview Survey      1986            NHIS-Vitamin
            Supplement on Vitamin
            and Mineral Supplements

            National Health Interview Survey      1987, 1992      NHIS-Cancer
            Supplement on Cancer Epidemiology
            and Cancer Control

HHS/PHS/    Behavioral Risk Factor Surveillance   Continuously    BRFSS
CDC/        System                                since 1984
NCCDPHP

            Pregnancy Nutrition Surveillance      Continuously    PNSS
            System                                since 1988

            Pediatric Nutrition Surveillance      Continuously    PedNSS
            System                                since 1973

HHS/PHS/    Navajo Health and Nutrition Survey    1991, 1992      Navajo HNS
IHS

HHS/PHS/    Health and Diet Survey                Biannually      Health and
FDA                                               since 1982      Diet

USDA/ARS    Nationwide Food Consumption Survey    Every 10 years  NFCS
                                                  since 1936,
                                                  1977-78, 1987-
                                                  88

            Continuing Survey of Food Intake by   1985-86, 1989-  CSFII
            Individuals                           91

            Diet and Health Knowledge Survey      1989, 1990,     DHKS
                                                  1991, 1993
--------------------------------------------------------------------------------
\a For more information on the scope and design of these systems, see
Directory of Federal and State Nutrition Monitoring Activities,
prepared by the Interagency Board for Nutrition Monitoring and
Related Research, HHS Pub.  No.  (PHS) 92-1255-1 (1992). 


--------------------
\3 See GAO/PEMD-94-23 for a discussion of these concerns and the
NNMRRP activities intended to address them. 

\4 Ten-Year Comprehensive Plan for the National Nutrition Monitoring
and Related Research Program, 58 Fed.  Reg.  111 (June 11, 1993), pp. 
32752-806. 


   SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3

To obtain information on the current uses of nutrition monitoring
data and identify the kinds of changes that are needed to increase
the utility of the data, we conducted a survey of potential users of
nutrition monitoring data.  The survey focused on the 14 NNMRRP data
collection activities listed in table 1, selected because they are
major activities or because they addressed a major concern, such as
the need for data on subpopulations.  These activities collect three
kinds of nutrition data:  nutritional and health status; food
consumption and dietary intake; and dietary knowledge, attitudes, and
behavior.  Two other areas of nutrition monitoring--food composition
and food supply and demand--were not addressed by the survey. 

Because we found no comprehensive list of people who use these data,
we developed a complex sampling plan to obtain lists of potential
users from a variety of sources.\5 Our focus was on obtaining the
views of primary data users, defined as those who have conducted
analyses in the past 5 years rather than relied on information
already processed and interpreted by others.  We limited our focus to
these users because we expected them to have a greater familiarity
with the strengths and limitations of each data collection system. 
(The data collection and sampling design are detailed in appendix I.)

Our sample design cast a wide net with the intention of obtaining
information from a variety of users.  However, because we aggregated
samples of different sizes from multiple lists, the survey results
cannot be used to characterize the average user in general or the
typical user in each of the organizational settings.  Moreover, we
asked users to consider their experiences with individual data
systems only, rather than with the NNMRRP as a whole.  Users first
identified which of the 14 data collection activities they had used
in the last 5 years and then focused on the two they used most
frequently. 

We conducted our review between December 1993 and December 1994 in
accordance with generally accepted government auditing standards. 


--------------------
\5 Out of a total of 1,585 potential data users sampled from the
lists, 1,180 (or 74.5 percent) responded; an industry group helped us
find another 10 respondents.  Among all respondents, 93 provided
insufficient information and 123 pooled their responses with those of
another respondent.  Among the rest of the respondents, 344 were not
users of nutrition monitoring data, 190 were secondary users (using
information that had already been analyzed), and 440 were primary
users. 


   PRINCIPAL FINDINGS
------------------------------------------------------------ Letter :4


      DATA ARE USED IN A VARIETY
      OF SETTINGS FOR MULTIPLE
      PURPOSES
---------------------------------------------------------- Letter :4.1

The 440 primary users who responded to our survey worked in federal,
state, and local government; academic institutions; for-profit
businesses, such as food industries; and other settings, such as
hospitals.  These users also represented a variety of occupations. 
While data users in federal or academic settings were more likely to
identify themselves as engaged in basic research, those in state or
local settings were more likely to indicate program planning and
management as their primary occupation.  (As noted above, these data
users who responded to our survey are not necessarily representative
of users in general or of the users in each organizational setting.)

Some use of almost every data collection system was reported in each
of the organizational settings.  (The exception is the Navajo HNS,
which was reported as used by only a small number of respondents in
the federal government and "other" category.) While state and local
government respondents were more likely to use the state-based
surveillance systems, federal government and academic respondents
were more likely to report using the national surveys.  (Appendix IV
presents more information on users, the data sets they use, and their
organizational setting.)

Across the different settings, the nutrition monitoring data
supported a variety of uses from identifying nutritional problems to
planning programs to address the problems, evaluating food- and
nutrition-related programs and policies, informing basic and
methodological research, and supporting state and local surveillance
activities.  Table 2 provides specific examples of the decisions
respondents stated they made based on the data.  (Tables showing the
percent of respondents indicating a specific purpose for which they
used a data collection system are provided in appendix II.)



                                     Table 2
                     
                     Respondents' Examples of Decisions Made
                                Based on the Data

Category                                Reported use
--------------------------------------  ----------------------------------------
Problem identification                  Determine prevalence of high blood
                                        cholesterol in U.S. adults

                                        Calculate exposure estimates for
                                        regulatory issues involving food
                                        additives

                                        Assess damage from Exxon Valdez oil
                                        spill

Policy-making or program planning       Refocus on diabetes in minority
                                        populations by the American Diabetes
                                        Association

                                        Support goals and activities for
                                        improved nutrition status of population
                                        in state cancer plan

                                        Develop national guidelines for
                                        screening and management of iron-
                                        deficiency anemia

                                        Confirm need for addition of calcium to
                                        infant and toddler foods

                                        Decide the size of target populations
                                        for new pharmaceutical products

                                        Place breast-feeding coordinators in
                                        areas of greatest need

Policy or program management and        Increase funding for Healthy Heart
evaluation                              Programs

                                        Modify year 2000 objectives for blood
                                        pressure to include Mexican Americans

                                        Document the need for use of iron-
                                        fortified infant formula and then
                                        document the success of the policy
                                        implementation

                                        Calculate sales tax consequences of
                                        cashing out food stamps

                                        Conclude that children have too much fat
                                        in their diets, but the excess is not
                                        caused by participation in child feeding
                                        programs

Research related to nutrition           Implement a clinical trial to prevent
                                        diabetes through diet modification

                                        Plan study of unusually high anemia
                                        levels in Alaskan Natives, which led to
                                        new cause of iron-deficiency anemia
                                        (bacterial)

                                        Determine which foods to include on a
                                        food-frequency questionnaire for Puerto
                                        Rican elderly

Support of monitoring activities        Choose knowledge and attitude indicators
by states and localities                for a state survey because reference
                                        values from national surveys are
                                        available

                                        Revise weighing and measuring policy to
                                        increase accuracy in clinics

                                        Use data in community needs assessment
                                        for counties to develop plans for
                                        services
--------------------------------------------------------------------------------
The primary users who responded to the survey also identified the
customers for their analyses.  As shown in table 3, users in each
organizational setting identified a range of end users of the data. 
In general, customers in their own organizational setting were most
commonly indicated, but the general public was also frequently
identified as a customer for the primary users' analyses of the
nutrition data. 



                                Table 3
                
                 End Users of Most Frequently Used Data
                                Systems


                            Feder                Academ  Busine  Other
End users                      al  State  Local      ic      ss     \a
--------------------------  -----  -----  -----  ------  ------  -----
Federal government            79%    14%      0     17%     22%    18%
State government               38     81    32%      19      14     23
Local government               26     80     75       7       3     23
Universities                   55     50     11      70      14     40
Hospital or health care        28     50     25      17      16     50
Researchers                    68     34     14      74      38     48
For-profit business            27     10     11      14      68     20
Nonprofit, noncharitable       26     49     18      15      16     45
Charitable organization        13     24     18       8       5     18
Media                          43     40     21      18      22     30
General public                 50     53     43      30      30     53
Other\b                         9     10     11       7      11      8
----------------------------------------------------------------------
\a Other settings include hospitals, nonprofit organizations, and
other charitable organizations. 

\b Other end users include trade associations, labor groups, tribal
governments, community action agencies, medical practitioners,
minority groups, regulators, university students, and so on. 


      DESPITE GENERAL
      SATISFACTION, USERS
      SUGGEST CHANGES
---------------------------------------------------------- Letter :4.2

The majority of the primary users responding to our survey reported
that the data collection systems meet their information and data
quality needs to at least a moderate extent.  However, despite this
and the evidence that the data are used for a variety of purposes, a
majority of respondents stated that changes are needed to increase
their confidence in or substantially increase their use of the data. 
A somewhat higher proportion of users of USDA data systems than of
HHS systems indicated a need for change.  (Users' satisfaction is
summarized in table IV.3.)

We asked primary users to identify what changes are needed in the
systems that they use most frequently.  Common themes in their
comments were

  -- continuous or more frequent data collection;

  -- more detailed information on racial, ethnic, and age groups;

  -- data that can support estimates for small geographic areas;

  -- improved timeliness and documentation of the data; and

  -- increased dissemination of the data in general and in formats
     that facilitate access and analysis. 

Specific comments from the users that illustrate these themes are
provided in table 4.  More detailed summaries are in appendix III. 



                                     Table 4
                     
                     Comments Illustrating Users' Suggestions
                                    for Change

                                                                        Data
Category                      Comment                                   system
----------------------------  ----------------------------------------  --------
Frequency of data collection  Consider sampling subsets of variables    NHANES
                              in NHANES and NFCS more frequently and
                              conducting full survey occasionally--
                              i.e.,
                              every 5-10 years.

                              NFCS could be done every 5 years since N  FCS
                              food availability changes
                              so much.

Coverage of racial, ethnic,   Include very young and old and enough     NHANES
and age groups                minorities to make conclusions about the
                              different groups.

                              Expand sample size to provide sufficient  BRFSS
                              number of minority respondents.

                              Need up-to-date analysis for all age      NFCS
                              groups. Had to use the different
                              databases because one did not provide
                              all age groups.

Coverage of geographic areas  More specific regional coverage would be  NHANES
                              highly useful for assessing
                              the diffusion of dietary and other
                              health behaviors.

                              Geographic area coverage should be more   CSFII
                              specific to allow analysis
                              and interpretation of data for
                              individual states.

                              It would be valuable to be able to        BRFSS
                              provide county-level data for use by
                              local health departments.

Timeliness and documentation  Would like faster turnaround from CDC to  PedNSS
                              states for annual PedNSS tables.

                              If results could be published more        NHANES
                              frequently, it would help us
                              see how well interventions are working.

                              Need more documentation, especially of    CSFII
                              what was done in the
                              survey, how it was done, and how the
                              statistical analyses were done.

Dissemination                 Data can be made more accessible and      NHANES
                              more timely using modern
                              technology and user-friendly systems.

                              Put on CD-ROM, include software that      DHKS
                              facilitates use, establish bulletin
                              board with updates as new data become
                              available.
--------------------------------------------------------------------------------

   CONCLUSIONS
------------------------------------------------------------ Letter :5

The NNMRRP data systems provide an important resource, serving a wide
set of purposes in a variety of settings.  Moreover, the data users
are mostly satisfied with the quality of the data and the degree to
which their data needs are met.  Despite this evidence of
satisfaction with the data systems, those who responded to our survey
had numerous suggestions for improving the data collection activities
of the NNMRRP.  These suggestions are consistent with many of the
past criticisms of the nutrition monitoring system.  Understanding
who uses the data and for what purposes is essential to developing
and implementing an effective nutrition monitoring system.  Our study
begins this process and provides a useful framework of purposes for
nutrition monitoring data. 


   AGENCY COMMENTS
------------------------------------------------------------ Letter :6

We provided the Board and responsible agencies with summary survey
data so they could begin revising their data collection activities as
we continued with our analyses and prepared this report.  A draft of
this report was then sent to USDA, HHS, and members of the National
Nutrition Monitoring Advisory Council for review and comment.  USDA
and HHS provided written comments, which are included in appendixes
VI and VII. 

In general, officials from these Departments agreed with our
principal findings and conclusions.  USDA noted that our survey
results will be useful as they plan future monitoring activities, and
HHS indicated that our report provides a good overview of the user
survey.  HHS officials, however, thought that our report did not
sufficiently describe all of the major federal uses of nutrition
monitoring data, and they provided further detail about these uses. 
Both HHS and USDA also presented additional information about actions
taken that respond to concerns raised by survey respondents regarding
information and data quality needs.  Technical comments made by HHS
and USDA officials and the members of the Advisory Council that
reviewed the report have been incorporated where appropriate. 

As arranged with your office, we will be sending copies of this
report to the Director of the Office of Management and Budget, the
Interagency Board on Nutrition Monitoring and Related Research, the
agencies

responsible for data collection, and to other interested parties.  We
will also make copies available to others upon request.  If you have
any questions or would like additional information, please call me at
(202) 512-3092.  Major contributors to this report are listed in
appendix VIII. 

Sincerely yours,

Kwai-Cheung Chan
Director of Program Evaluation
 in Physical Systems Areas


QUESTIONNAIRE DESIGN, SAMPLING
PLAN, AND ANALYSIS DECISIONS
=========================================================== Appendix I

QUESTIONNAIRE DESIGN

This section describes how we developed our questionnaire and
provides an overview of the questionnaire content. 


      SELECTING DATA COLLECTION
      SYSTEMS
------------------------------------------------------- Appendix I:0.1

Our survey queried respondents about only 14 of the approximately 70
data collection activities listed in the Directory of Federal and
State Nutrition Monitoring Activities.  (See table 1 on p.  3.) All
14 systems met our criteria of focusing on (1) dietary, nutritional,
and health status; (2) food consumption; or (3) dietary knowledge,
attitudes, and behavior.  Information about the food composition
databases or activities for monitoring the food supply was not
gathered.  This allowed us to concentrate on survey-based data
collection activities. 

An additional criterion was that the data collection system be an
ongoing program.  For example, periodic surveys like the Nationwide
Food Consumption Survey were included, while one-shot evaluations of
food assistance programs were excluded.  We made three exceptions to
this criterion.  We chose the NHIS-Vitamin and Mineral Supplements
and NHIS-Cancer Epidemiology and Cancer Control because of their
large size (nearly 11,800 and 45,000 interviews, respectively).  We
also collected information on the Navajo Health and Nutrition Survey
because of the need for data on subpopulation groups expressed in
public comments to a draft of the 10-year comprehensive plan. 


      DEFINING USERS
------------------------------------------------------- Appendix I:0.2

Most of the questions in the survey were directed only to primary
users of the data from the 14 selected activities.  We defined a
primary data user as one who directly accesses these data.  This
includes those who request analyses from others as well as those who
access the data systems themselves.  In contrast, secondary users are
those who use nutrition monitoring information that has already been
processed and interpreted by others in reports, articles,
publications, or other documents.  We chose this definition to target
the questionnaire to respondents with firsthand experience with the
design and content of the data collection activities and the
strengths and limitations of the data. 


      STRUCTURING THE
      QUESTIONNAIRE
------------------------------------------------------- Appendix I:0.3

We sent our survey to both known and potential users.  Primary users
of the data could not be identified in advance, so in the first
section, we screened out secondary users and nonusers of the 14 data
collection systems.  Then we asked the remaining respondents--the
primary data users--to identify the data collection system they used
most frequently and the next most frequently in the past 5 years. 

A major portion of the survey was dedicated to determining how the
respondents used the data.  Through literature reviews and a series
of expert panel meetings, we developed an inventory of the uses of
federal nutrition monitoring data.  As shown in appendix II, specific
uses were categorized in five main areas:  (1) problem
identification, (2) policy-making and program planning, (3) policy
and program evaluation and management, (4) research related to
nutrition, and (5) support of state and local nutrition monitoring
activities.  Respondents also had the opportunity to record up to
five additional purposes for which they used the data.  To ascertain
the validity of the uses they identified, we asked them to list at
least one report, article, or other document produced with the data. 

We also obtained information on the extent to which the data
collection systems met the respondent's information and data quality
needs.  We asked whether changes are needed to better meet their
needs for the data.  Of those indicating a need for changes, we asked
for their suggestions on improving the (1) data elements collected,
(2) data collection methods, (3) units of analysis, (4) time of data
collection, (5) population group coverage, (6) geographic area
coverage, and (7) ease of use. 

SAMPLING PLAN

This section details the sampling approach and provides information
on the sources from which we obtained names of people to survey. 


      OVERVIEW
------------------------------------------------------- Appendix I:0.4

We had no way of identifying all the users of the federal nutrition
monitoring data, so we chose a nonrandom sampling approach to
maximize the heterogeneity of the individuals surveyed.  From a
variety of sources, we obtained lists of known and potential users of
the data and also of contacts in organizations likely to contain data
users. 

We mailed out a total of 1,614 surveys.  Addresses were incorrect for
29, so the sample size was reduced to 1,585.  We received 1,180
responses, or 74.5 percent.  An additional 10 responses came from a
confidential industry mailing list, increasing our total responses to
1,190.  Of those, 344 were nonusers, 190 were secondary users, and
440 were primary users.  In addition, 123 indicated that their
responses were included in with other respondents, and 93 did not
provide useful information because they were ill, retired, or failed
to complete the questionnaire.  Primary users tended to respond
early, and nonusers tended to respond only after one or two
follow-ups. 


      IDENTIFYING USERS AND
      POTENTIAL USERS
------------------------------------------------------- Appendix I:0.5

We selected users both as individuals likely to use nutrition
monitoring data and as members of organizations likely to contain one
or more data users.  We asked the former to answer only for their own
uses and the latter to direct the survey to the most appropriate user
within their organization, who would also answer only for his or her
own use.  We did not distinguish between individual and
organizational respondents in our analyses. 


         INDIVIDUALS
----------------------------------------------------- Appendix I:0.5.1

We identified actual and potential individual users of the 14
nutrition monitoring activities from a variety of sources, including
lists maintained by federal agencies of people who had requested
data, referrals of likely users from the Interagency Board and other
federal contacts, lists of people attending workshops and
conferences, and professional association membership lists.  Tables
I.1-I.3 identify the sources of our lists and the number of people
surveyed. 

Table I.1 provides the number individuals we surveyed, by data
collection activity, who requested data from the National Technical
Information Service (NTIS) or directly from the federal agencies
administering any of the 14 systems.  The two largest groups were the
NHANES III and Cancer Risk Survey data requesters. 



                               Table I.1
                
                 Data Requesters Included in Our Sample

Source                                                          Number
agency      Data system                                       surveyed
----------  ----------------------------------------------  ----------
USDA/HNIS   CSFII 1989                                               7
            CSFII 1990                                               1
            NFCS 1987-88                                             7
            NFCS (household)                                         5
            NFCS (low income)                                        9
            NFCS (household and individual intakes)                  6
HHS/NCHS    NHANES I                                                16
            NHANES II                                               24
            NHANES III                                              58
            NHANES I Epidemiological Follow-up                      13
            NHANES I Follow-up Group Members                        12
            Hispanic HANES                                           7
            NHANES (unspecified)                                     9
            Cancer Risk Survey                                      42
            Vitamin and Mineral Survey                               5
HHS/FDA     Health and Diet Survey                                   6
NTIS        CSFII                                                   10
            Nutrient Data Base                                       8
            NHANES I                                                 2
            NHANES II                                                9
            NHANES I Epidemiological Follow-up                       7
            NHIS                                                    10
            NFCS 1977-78                                             6
            NFCS 1987-88                                            10
----------------------------------------------------------------------
To ensure that we had full coverage of federal government users, we
asked the Interagency Board for the names of directors of agency
divisions mentioned in the 10-year comprehensive plan.  Through other
referrals, we added the names of 26 potential users within those
agencies.  From lists of attendees at three federally-sponsored,
nutrition-related workshops, we identified another set of known or
likely users that we surveyed.  Members of associations for nutrition
professionals were another source of potential users we surveyed. 
Finally, we obtained lists of local government officials working in
nutrition.  (Table I.2 provides the source and application of these
additional potential users that we surveyed.)



                               Table I.2
                
                  Additional Sources of Nutrition Data
                                 Users

                                                                Number
Source              Affiliation                               surveyed
------------------  ----------------------------------------  --------
Likely federal      Division directors (list provided by the        23
 users               Interagency Board)
                    Potential federal users identified              26
                     through referrals
Nutrition-related   Dietary Consensus Conference                    58
 workshop
 attendees
                    Food Insecurity Conference                      61
                    USDA Household User Group                       14
Association         American Dietetic Association\a                203
 members
                    American Institute of Nutrition\b              268
                    Society for Nutrition Education\c              107
Lists of local      CityMatCH members (Urban Maternal and          144
 government          Child Health directors)
 officials
                    National Association of County Health           27
                     Officials\d
----------------------------------------------------------------------
\a From more than 65,000 members, we identified a subgroup of 2,030
employed in education and research, and then we drew a 1-in-10
sample. 

\b From nearly 3,000 members, we drew a 1-in-10 sample. 

\c We drew a 1-in-3 sample of members working in higher education,
industry, public health, and county extension education. 

\d From their National Directory of Local Health Departments, we
chose the directors from 30 counties containing 1 million or more
inhabitants, or 23.6 percent of the 1990 U.S.  population (according
to the Census Bureau). 


         ORGANIZATIONS
----------------------------------------------------- Appendix I:0.5.2

To capture any state and local officials we may have missed, we
targeted organizations that were likely to have at least one or more
state and local nutrition monitoring data users.  We asked
organizations to direct the survey to the most appropriate or
experienced officials, who would respond only for their own use of
the data, not for the organization as a whole.  (See table I.3.)

To cover the fragmented groups in nutrition research and policy
analysis, we built in some redundancy within the sampling plan. 
Respondents who received more than one survey, however, were counted
only once in our analyses. 



                               Table I.3
                
                  Additional State and Local Nutrition
                               Data Users

                                                                Number
Affiliation              Respondents                          surveyed
-----------------------  -----------------------------------  --------
Association of State     Child health and nutrition                 45
 and Territorial Public   officials
 Health Officials
                         Chronic disease officials                  42
                         Health education officials                 53
Maternal and Child       Maternal and child health directors        52
 Health Association
Center on Budget and     WIC directors                              44
 Policy Priorities
State Government Yellow  Health department directors                52
 Book
Surveillance system      BRFSS                                      47
 contacts
                         PedNSS                                     39
                         PNSS                                       18
----------------------------------------------------------------------

      LIMITATIONS OF THE SAMPLING
      PLAN
------------------------------------------------------- Appendix I:0.6

While our approach allowed us to cast a wide net and contact as many
data users as possible, it also has some limitations.  The major one
is our inability to determine the degree to which the survey
respondents are representative of primary data users in general. 
With a nonprobability sample, we cannot generalize beyond our
respondents to the universe of all users of the 14 data collection
activities the questionnaire addresses.  In addition, we cannot make
any inferences about the extent of use across groups.  Our ability to
identify primary users within groups varied, so differences in
reported use may be a function of our sample design rather than of
actual differences in use.  For example, we were able to target
federal users of the data more closely than users in other sectors,
but it would be inappropriate to compare the extent of their use to
that by other groups of respondents. 

A further limitation of our survey design is that we asked
respondents to comment on individual data collection activities (such
as DHKS) and not on the specific survey components within each
activity (such as the 1989, 1990, 1991, or 1992 DHKS).  Some users
who are familiar with only a specific survey component and not all
the survey components may have made suggestions to us for changes
that have already been addressed by the agencies in later versions of
the data collection activity. 

ANALYSIS DECISIONS


      CONSTRUCTION OF ANALYSIS
      GROUPS
------------------------------------------------------- Appendix I:0.7

Our six groups of primary users--federal, state, local, academic,
business (for-profit), and other--were constructed according to
respondents' self-reports.  Healthcare (hospital, nursing home) was
the dominant group within the "other" category, which also included
nonprofit businesses and charitable organizations.  Overall, the
groups were diverse, and no single subgroup dominated any group. 

The federal group in our sample consisted of at least 23 different
agencies.  Those with the largest number of respondents were the
National Center for Health Statistics (17) and the Human Nutrition
Information Service (11).  They are responsible for the main data
systems, and they provided an insider's view to the strengths and
weaknesses of these systems.  The state group included 45 states that
responded and the District of Columbia and Guam.  The largest number
of respondents from one state was eight, or 6 percent, so no one
state had a large influence on the whole group.  The local group had
one user each from 28 different counties. 

The academic group came from 67 different cities, and some cities,
such as Boston and Chicago, were represented by more than one school. 
The largest number of respondents from one school was five, or 5
percent.  Five of the 37 in the business group did not give their
addresses; the others came from 26 different cities, and the largest
number from one city was three.  The 40 respondents in the "other"
group came from 33 cities, and the largest number from one city was
three. 


USES OF NUTRITION MONITORING DATA
========================================================== Appendix II

The data collected by the nutrition monitoring systems are not only
used across a variety of organizational settings, but they also
support a range of uses.  With the assistance of our expert
consultants, we developed an inventory of the purposes that federal
nutrition monitoring data serve.  Specific purposes were categorized
in five overarching areas:  (1) problem identification, (2)
policy-making and program planning, (3) policy and program evaluation
and management, (4) research related to nutrition, and (5) support of
state and local nutrition monitoring activities. 

For the two data collection activities used most frequently (see
table II.1), respondents to our survey were asked to indicate the
purposes the data had served.  The data users were also asked to
write in specific decisions that the data had supported.  Although
our respondents presented a variety of purposes and decisions, one
noted that the data were not timely enough, or sufficiently on
target, to truly inform decisions.  We classified the written
comments into the five major categories of purposes. 



                               Table II.1
                
                Two Most Frequently Used Data Collection
                               Activities



                              Numbe  Perce  Numbe  Perce  Numbe  Perce
Data collection activity          r     nt      r     nt      r     nt
----------------------------  -----  -----  -----  -----  -----  -----
NHEFS                            34      8     20      6     54     12
NHANES II                        46     10     55     16    101     23
NHANES III                       54     12     36     11     90     20
HHANES                           11      2     17      5     28      6
NIHS-Vitamin                      2      1      1      0      3      1
NHIS-Cancer                       9      2      4      1     13      3
BRFSS                            64     15     21      6     85     19
PNSS                             29      7     44     13     73     17
PedNSS                           68     15     28      8     96     22
Health and Diet                   8      2      5      1     13      3
NFCS                             69     16     41     12    110     25
CSFII                            43     10     46     14     89     20
DHKS                              3      1     20      6     23      5
======================================================================
Total                           440  101\a  338\b   99\a  440\c
----------------------------------------------------------------------
\a Because of rounding, total percentages do not add up to 100. 

\b Some respondents had used only one data system; thus, the number
of the second most frequently used system was less than 440. 

\c Total sample; not column total. 

PROBLEM IDENTIFICATION

The expert panels distinguished seven kinds of problems that the data
might be used to identify.  (See table II.2.) As shown in the table,
problem identification was a commonly indicated purpose supported by
the data.  However, there is some variation in the kinds of problems
examined by the different data sets.  For example, NHANES data--with
its emphasis on health--is used to examine chronic degenerative
diseases, as well as deviations in nutritional status. 



                               Table II.2
                
                 Respondents' Indication of Use of Data
                       for Problem Identification


Data collection activity         1     2     3     4     5     6     7
----------------------------  ----  ----  ----  ----  ----  ----  ----
NHEFS                          70%   48%    6%   52%    6%    4%   14%
NHANES II                       49    45    11    55     9     2    10
NHANES III                      55    47    21    57     6     5    14
HHANES                          65    42     4    54     4     0     8
NHIS-Cancer                     31    39     0    39     8     0     8
BRFSS                           55    41     1    74     1     0    13
PNSS                            14    86    15    80     0     1     6
PedNSS                          17    88    16    80     0     1     5
Health and Diet                 39    39     8    62     8    23    15
NFCS                            29    44    22    56    18    11    43
CSFII                           30    44    23    61    21    14    42
DHKS                            36    27    23    77    32    23    27
Median percent                  38    44    13    59     7     3    14
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
indicate if they had used the data to identify or estimate the risk,
incidence, prevalence, duration, or cost of any of the following
problems: 

1. Chronic degenerative diseases and their relationship to diet and
nutritional status;

2. Nutritional deficiency diseases and health-related issues;

3. Hunger and food insecurity, including its relationship to diet and
its periodicity;

4. Deviations in nutritional status (e.g., obesity) and diet quality
across the life-cycle and across population groups;

5. Food safety problems over which consumers have little control
(e.g., contaminants);

6. Food safety problems over which consumers have some control (e.g.,
microbiological problems resulting from food preparation, handling,
or consumption activities); and

7. Other food quality problems (availability, accessibility, and
composition). 

POLICY-MAKING OR PROGRAM PLANNING

Within the overarching objective of policy-making or program
planning, the expert panel identified seven purposes.  (See table
II.3).  As shown in the table, most of the program purposes reported
were fairly common uses of nutrition monitoring data sets.  One
exception was "to compare cost-effectiveness" of different kinds of
policy or program interventions, which is not surprising since the
nutrition data do not provide this information.  Many of the uses
that respondents wrote in response to the request for four specific
decisions informed by the data appeared to fit under policy-making or
program planning. 



                               Table II.3
                
                 Respondents' Indication of Use of Data
                 for Policy-making or Program Planning


Data collection activity         1     2     3     4     5     6     7
----------------------------  ----  ----  ----  ----  ----  ----  ----
NHEFS                          51%   30%   42%   20%    2%   19%   42%
NHANES II                       58    36    44    28     4    23    43
NHANES III                      64    52    55    31     8    27    61
HHANES                          58    46    46    36     4    27    54
NHIS-Cancer                     46    31    31    31     0    15    77
BRFSS                           72    58    47    53     4    46    78
PNSS                            85    70    65    85     7    60    89
PedNSS                          86    69    66    80    11    60    89
Health and Diet                 77    54    54    46     8    39    54
NFCS                            66    44    52    38     8    22    48
CSFII                           76    51    57    40     6    23    51
DHKS                            74    48    73    52     0    23    57
Median percent                  69    50    53    39     5    25    56
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
write in four specific decisions informed by the data, many of which
fit within the following purposes: 

1. Define and quantify the extent and distribution of a food- or
nutrition-related problem or the risk of the problem;

2. Assess the importance of a problem or risk of the problem relative
to other problems;

3. Identify determinants of a food- or nutrition-related problem or
risk of the problem;

4. Identify policy and programmatic responses to the problem or risk
of the problem;

5. Compare cost-effectiveness of responses;

6. Justify the selection of a response (prevention, intervention to
mitigate, or intervention to deal with the consequences); and

7. Serve as a basis for targeting prevention or intervention
resources or both. 

PROGRAM EVALUATION AND MANAGEMENT

The inventory of purposes served by nutrition monitoring data
included nine program evaluation and management purposes.  (See table
II.4.) As shown in the table, two of the more commonly indicated
purposes in this area were measuring changes in deficiency diseases
and assessing achievement of specific dietary objectives.  For
example, several of the uses described by the survey respondents
focused on monitoring or modifying Healthy People 2000 objectives. 
In contrast, measuring changes in food safety problems was one of the
least common purposes indicated in the survey. 



                               Table II.4
                
                 Respondents' Indication of Use of Data
                  for Program Evaluation or Management


Data collection activity     1    2    3    4    5    6    7    8    9
-------------------------  ---  ---  ---  ---  ---  ---  ---  ---  ---
NHEFS                      47%   8%  31%  35%   8%  33%  12%  15%   8%
NHANES II                   37    8   23   43    7   46   14   22   13
NHANES III                  46   13   33   42    7   50   13   28   18
HHANES                      39    8   23   27    0   31    8   23   12
NHIS-Cancer                 25    8    8   33    0   42   17   33    8
BRFSS                       44    1   26   46    4   77    1   41   38
PNSS                        16    7   63   66    1   78    3   40   64
PedNSS                      20    7   75   73    0   79    3   30   63
Health and Diet             33    0   46   39    8   77   25   69   46
NFCS                        19   17   20   50   13   51   28   42   27
CSFII                       19   17   19   55   13   57   26   48   25
DHKS                        36   18   14   46   10   57   23   57   30
Median percent              35    8   24   45    7   54   14   37   26
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked if
they had used the data for the following: 

1. Measure changes in chronic degenerative diseases, their
relationship to diet and nutritional status, and the risk of such
diseases;

2. Measure changes in hunger and food insecurity;

3. Measure changes in deficiency diseases and health-related issues;

4. Measure changes in deviation in nutritional status and diet
quality;

5. Measure changes in food safety problems;

6. Measure achievement of specific dietary objectives;

7. Evaluate food supply and nutrient supplements and fortificants;

8. Measure changes in food- and nutrition-related behaviors and their
precursors and determinants; and

9. Assess targeting and coverage of food- and nutrition-related
programs. 

RESEARCH RELATED TO NUTRITION

The expert panelists included seven research purposes for nutrition
monitoring data in the inventory.  (See table II.5.) Across the
different data sets, respondents commonly indicated that the data
were used to increase basic research knowledge of the determinants of
problems and options for intervention. 



                               Table II.5
                
                 Respondents' Indication of Use of Data
                              for Research


Data collection activity         1     2     3     4     5     6     7
----------------------------  ----  ----  ----  ----  ----  ----  ----
NHEFS                          27%   14%   54%   44%   35%   19%   10%
NHANES II                       29    26    50    48    33    22     9
NHANES III                      40    31    53    45    36    21     7
HHANES                          54    44    62    50    42    39     8
NHIS-Cancer                     39    23    54    23    23     8     0
BRFSS                           33    44    40    33    18     8     5
PNSS                            41    62    52    40    33    13     3
PedNSS                          35    51    44    35    32    13     4
Health and Diet                 15    15    39    39    23     8     0
NFCS                            28    23    43    44    27    36    23
CSFII                           26    30    51    49    40    41    22
DHKS                            27    18    64    68    36    46    19
Median percent                  31    28    52    44    33    20     8
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
indicate if they had used the data for the following: 

1. Improve sampling and statistical methods for gathering data from
people at different life-cycle stages or for minority or other
subpopulations, especially those at risk of food- or
nutrition-related problems;

2. Improve methods for informing decisionmakers so analysis results
are timely, pertinent, and understandable;

3. Increase basic research knowledge of the determinants of problems
and options for intervention;

4. Increase basic research knowledge of the relationships between
food, nutrition, and health;

5. Identify and stimulate needed research and development on
monitoring methods;

6. Conduct other kinds of basic research (e.g., on the distribution
of human nutrient requirements, databases on food cost and food
preparation, individual variability); and

7. Conduct food composition research and improve food composition
databases. 


A few data users conduct food composition research; however, the food
composition databases were not included as a focus of the survey. 
Specific research uses supported by the data included identifying
foods for food-frequency questionnaires, establishing cut-points for
defining research subjects, and developing survey instruments. 

SUPPORT OF MONITORING ACTIVITIES
BY STATES AND LOCALITIES

Under the general goal of supporting the monitoring activities of
states and localities, the expert panelists identified two specific
purposes for the nutrition monitoring data:  supporting state and
local surveillance activities and supporting technical assistance. 
At least some portion of the users of each of the data collection
activities identified one of these purposes as a way in which they
use the data.  (See table II.6.) Of the two, the latter was more
commonly indicated.  This pattern also appears in the respondents'
comments, many of which focus on identifying the need for technical
assistance and improving the quality of data collection.  (See
appendix III.)



                               Table II.6
                
                 Respondents' Indication of Use of Data
                for Activities by States and Localities


Data collection activity                              1              2
----------------------------------------  -------------  -------------
NHEFS                                               10%            10%
NHANES II                                             5             13
NHANES III                                           12             26
HHANES                                                8             19
NHIS-Cancer                                           8             15
BRFSS                                                12             54
PNSS                                                 22             63
PedNSS                                               19             62
Health and Diet                                      15             15
NFCS                                                 11             16
CSFII                                                 8             16
DHKS                                                 14             23
Median percent                                       12             18
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
indicate if they had used the data for the following: 

1. Support state and local surveillance of and responses to food- and
nutrition-related crises; and

2. Support development and provision to states and localities of
technical assistance in data collection, analysis, and
interpretation. 


SUMMARIES OF THE SUGGESTIONS MADE
BY USERS
========================================================= Appendix III

SYSTEMS UNDER NCCDPHP

The National Center for Chronic Disease Prevention and Health
Promotion administers three surveillance systems that collect
information on health and nutritional status:  the Pediatric
Nutrition Surveillance System, the Pregnancy Nutrition Surveillance
System, and the Behavioral Risk Factor Surveillance System.  The
three systems are overseen by the National Center, but are actually
implemented by the states that participate in the surveillance
programs.  Table III.1 describes each system's target population and
data collection methods. 



                                   Table III.1
                     
                        CDC's NCCDPHP Surveillance Systems

                                Type and source of
System  Target population       sample                  Data collection method
------  ----------------------  ----------------------  ------------------------
PedNSS  Low-income, high-risk   Participants in         Clinic staff record data
        children                publicly-funded         at checkups (body
                                prenatal nutrition and  measurements, blood test
                                food assistance         results, and demographic
                                programs                data)

PNSS    Low-income, high-risk   Participants in         Clinic staff record
        pregnant women          publicly-funded         health status, blood
                                prenatal nutrition and  test results, risk
                                food assistance         behaviors, and
                                programs                demographic data

BRFSS   Adults, age 18 and      Random telephoning of   Telephone interviews
        over                    households              (body measurements, risk
                                                        behaviors, food choices)
--------------------------------------------------------------------------------
These surveillance systems vary in their purposes, methods of data
collection, and types of respondents, yet we found common themes in
the recommendations made by primary users of the systems.  For all
three systems, users suggested providing

  -- more data on dietary intake,

  -- better controls on the quality of the data collected,

  -- more detail on subpopulation groups in the reporting of the
     data,

  -- increased ability to look at substate geographic divisions,

  -- improved timeliness of CDC's return of the data,

  -- simplified reports that are more readily used at the local
     level, and

  -- additional technical and financial assistance in data collection
     and interpretation. 

User recommendations specific to each of the systems are presented in
the tables below. 


      COMMENTS ON PEDNSS AND PNSS
----------------------------------------------------- Appendix III:0.1

Because PedNSS and PNSS collect data on several similar issues, they
are listed together in table III.2.  In addition to these comments,
some respondents complimented CDC on the quality of PedNSS,
specifically for the automated system and for its coordination of the
data collection with the WIC Program. 



                                   Table III.2
                     
                      User Suggestions for Improving PedNSS
                                     and PNSS

Type of change                Comment
----------------------------  --------------------------------------------------
Data elements                 Collect more data on

                              Infant feeding practices, particularly breast-
                              feeding or type of formula (and provide
                              better analysis)

                              Pregnancy risk information (PNSS)

                              Dietary intake (food frequency, 7-day records,
                              "usual" intake)

                              Food security and hunger (PedNSS)

                              Demographics

                              Other indicators--blood lead levels, serum
                              cholesterol, immunizations, height and
                              weight at 2 and 3 years, household smoking
                              (PedNSS) and physical activity and
                              risk behaviors (PNSS)

Data collection methods       Improve data collection quality control (training,
                              uniform reporting, better software, and
                              standardization of measurements)

                              Use other sources of information (vital records,
                              private physicians)

                              Maintain cultural sensitivity

                              Streamline and simplify questions

                              Stop changing the data requirements (PNSS only)

                              Use more biochemical measures

                              Develop methods to obtain data from more sources
                              than public clinics (e.g., scannable forms that
                              private physicians could complete)

Units of analysis             Maintain records by individual child, not by
                              clinic visit (PedNSS only) to avoid duplication of
                              records

Time of data collection       Facilitate analyses of changes over time by
                              linking all records to the individual child
                              (PedNSS only)

Population group coverage     Expand beyond participants in WIC and other
                              publicly-funded programs to include non-low-
                              income women and children

                              Collect and report more data by subgroup (race,
                              ethnicity, age, sex, income)

Geographic area coverage      Enable reporting by substate divisions

                              Improve national estimates by including all states
                              (currently, states choose whether they
                              participate)

Ease of use                   Provide

                              Improved timeliness of reporting

                              Simplified report format and content

                              Reports that are more accessible for local users

                              Improved flexibility of the PedNSS automated
                              system and exportability of the data

                              An automated system for PNSS that is similar to
                              PedNSS

                              Technical training and funding assistance to
                              states to implement systems
--------------------------------------------------------------------------------

      COMMENTS ON BRFSS DATA
----------------------------------------------------- Appendix III:0.2

User recommendations for BRFSS are listed below in table III.3. 



                                   Table III.3
                     
                       User Suggestions for Improving BRFSS

Type of change          Comment
----------------------  --------------------------------------------------------
Data elements           Collect more data on

                        Dietary intake (in general and to link to behavior)

                        Specific dietary elements (fat, food groups, fiber,
                        nutritional supplements, alcohol, ethnic foods)

                        Households (number of adult smokers)

                        Ethnicity (state-specific)

                        Improve

                        Questions on dietary fat to obtain a better measure

                        Correspondence of health indicators with health
                        objectives (percent of fat, salt intake, label reading)

                        Add questions on cholesterol, diabetes, and disease risk

                        Make questions more culturally sensitive and relevant

Data collection         Address data validity and other quality control issues
methods                 (translation for non-English-speakers, applicability to
                        adults in households without telephones, nonuniformity
                        across states)

                        Develop a method to gather more complete dietary data

Units of analysis       Maintain records by individual, with more data on the
                        individual's household

Time of data            Collect nutrition data
collection

                        Continually or at least every 2 years

                        At times most representative of year-round habits (make
                        seasonal adjustments)

Population group        Provide more detail on
coverage

                        Subpopulations in general (increase sample)

                        Racial and ethnic groups (and those specific to a state)

                        Specific age groups

                        High-risk populations

                        Include populations without telephones

Geographic area         Improve national estimates by including all states
coverage

                        Increase sample sizes within states for better estimates

                        Provide information for substate divisions (counties,
                        cities, rural areas), which will assist in planning and
                        evaluating community interventions

Ease of use             Provide

                        Improved timeliness of data (not only for state and
                        local users, but also for researchers, who must obtain
                        permission from each state)

                        An automated system for state analyses of data

                        Improved documentation

                        Facilitated access to the data for nonstate users

                        Technical assistance in data interpretation (especially
                        dietary fat data)
--------------------------------------------------------------------------------
SYSTEMS UNDER NCHS

Users of four of the data collection activities run by the National
Center for Health Statistics commented on these systems in our
survey:  the National Health and Nutrition Examination Survey, the
NHANES I Epidemiological Follow-up Study, the Hispanic Health and
Nutrition Examination Survey, and the National Health Interview
Survey on Cancer Epidemiology and Cancer Control.  Table III.4
provides some summary information about these surveys.\1



                                             Table III.4
                               
                                CDC's NCHS Data Collection Activities

Activity          Target population           Type and source of sample   Data collection method
----------------  --------------------------  --------------------------  --------------------------
NHANES            Civilian,                   Stratified, multistage,     In-person interviews,
                  noninstitutionalized        probability cluster sample  including a single 24-
                  population age 2 months     of households;              hour recall and physical
                  and older                   oversampling of children,   examinations
                                              elderly, African-
                                              Americans, and Mexican-
                                              Americans

NHEFS             All persons between 25 and  Same as for NHANES, with    In-person interviews,
                  74 years old who completed  tracing of age group of     physical measurements,
                  a medical examination at    interest                    review of hospital and
                  NHANES I in 1971-75                                     other records

HHANES            Civilian,                   Stratified, multistage,     In-person interviews and
                  noninstitutionalized        probability cluster sample  physical examinations
                  Hispanics (Mexican-         of the target populations
                  Americans, Cubans, Puerto
                  Ricans) age 6 months-74
                  years residing in
                  households in three
                  defined U.S. geographic
                  areas

NHIS-Cancer       Civilian,                   Stratified, multistage,     In-person interviews
                  noninstitutionalized U.S.   cluster sample, including
                  population age 18 years     one randomly selected
                  and older                   person 18 years or older
                                              in each NHIS household;
                                              oversampling of Hispanics
                                              and African-Americans in
                                              the last implementation
                                              (1990)
----------------------------------------------------------------------------------------------------
Our survey asked respondents to identify themselves as users of
NHANES II, which was conducted from 1976 to 1980, or NHANES III,
which started in 1988 and was completed in 1994.  (NHANES I was
conducted between 1971 and 1975.) Because the surveys are very
similar in their design, no distinction is made between suggestions
made by users of NHANES II and those made by users of NHANES III in
the discussion below.  HHANES differs from NHANES in its focus on
three Hispanic subpopulations, but is otherwise similar in
methodology.  The data collected by NHEFS, unlike that for NHANES and
HHANES, allow for the study of changes over time through follow-up
surveys (in 1982-84, 1986, 1987, and 1992) with all persons between
25 and 74 years of age who had completed a medical examination for
NHANES I.  All three use both food-frequency questions and 24-hour
recall to collect dietary intake data. 

In contrast, NHIS-Cancer relies solely on food-frequency questions. 
Even though it is not one of the major nutritional data collection
systems, NHIS-Cancer was included in our survey because it measured
nutrition variables such as frequency of eating selected food items,
vitamin and mineral supplement intake, and knowledge of the
relationship between diet and cancer. 

Although the target populations and methods used for the NCHS data
collection activities vary, some common themes emerged in the
analysis of the comments from the users of the different systems. 
Users suggested providing

  -- more information on health habits and outcomes;

  -- more detailed data on food consumption;

  -- improved dietary intake methods, whether food-frequency
     questions or 24-hour recalls;

  -- a focus on the individual unit of analysis, with information
     linking the individual to the family or household unit;

  -- continuous or more frequent data collection;

  -- more detailed information on racial, ethnic, and age groups;

  -- data that can support estimates for smaller geographic areas;

  -- improved timeliness and documentation of the data; and

  -- increased dissemination of the data in general and in formats
     that facilitate access and analysis. 

The specific comments made under these general themes and on other
subjects are detailed in table III.5. 



                                   Table III.5
                     
                     User Suggestions for Improving NCHS Data
                                Collection Systems

Type of change          Comment
----------------------  --------------------------------------------------------
Data elements           Collect more information on

                        Health-related habits (physical activity, smoking,
                        alcohol use)

                        Medical history

                        Health outcomes in general (arthritis, skin diseases,
                        food allergies, cancer, and for elderly, hearing loss)

                        Cause of death (NHIS-Cancer)

                        Dietary intake

                        Demographics (occupation as a source of nondietary
                        exposure to cancer)

                        Environmental risk factors

                        Nonrespondents

Data collection         Conduct more research on data collection methods, in
methods                 particular

                        Measuring for race, ethnicity, and age (minorities,
                        adolescents)

                        Validating portion size (absolute amounts and percent of
                        calories

                        Using biochemical analyses (larger samples)

                        Improve automation and processing

                        Standardize techniques

                        Use multiple measures (telephone and in-person
                        interviews and mailed questionnaires)

                        Expand the use of food-frequency questions

                        Include Hispanic foods and newer versions of common
                        foods

                        Translate questionnaires for non-English-speaking
                        persons and use fully bilingual interviewers

                        Obtain dietary data on more than a single day (multiple
                        24-hour data or 3-day records)

Units of analysis       Retain individual as most important unit for nutrition
                        issues (NHANES, HHANES, NHEFS)

                        Link individual data to family or household unit (NHIS-
                        Cancer)

                        Account for non-Hispanics in household (HHANES)

Time of data            Continuously collect nationally representative NHANES
collection              data while collecting subpopulation data\a

                        Shorten NHANES to conduct more frequently if not
                        continuously (NHIS is a model for continuous
                        collection.)

                        Shorten cycles of surveys to produce more frequent
                        updates

                        Conduct more methods research and data analysis between
                        surveys

                        Increase frequency for nutritionally vulnerable groups

                        Conduct longitudinal follow-up on chronic diseases

                        Account for seasonality

Population group        Provide
coverage

                        Better and more coverage of racial, ethnic, and age
                        groups

                        Clearer criteria on definition of race

                        Comparable age-sex groups for racial and ethnic groups

                        Generalizability (HHANES)

Geographic area         Provide
coverage

                        More specific regional coverage

                        More refinement of geographic detail (rural; urban;
                        standard metropolitan statistical areas; Alaska, Hawaii,
                        Puerto Rico, and Indian reservations)

                        Use small-area estimation models

                        Provide state-level estimates

Ease of use             Provide

                        More timely release of data

                        More and better documentation of complex sample designs

                        On-line documentation Improved advertising of
                        availability of different data

                        Training in using complex sample designs

                        Data in common statistical package format

                        Occupation data coded for risk categories

                        Anthropometric data using 15th and 85th percentiles as
                        well as 25th and 75th
--------------------------------------------------------------------------------
\a One of the criticisms of HHANES is that the data were collected at
a different time from the NHANES data, and thus, the health and
nutritional status of the Hispanic groups cannot be compared to that
of the nation as a whole. 

SYSTEMS UNDER USDA

Our survey asked users of three USDA data collection activities to
comment on changes to the surveys that would increase their use of
the data.  The three USDA surveys addressed are the Nationwide Food
Consumption Survey (now called the Household Food Consumption
Survey), the Continuing Survey of Food Intakes by Individuals, and
the Diet and Health Knowledge Survey.  Table III.6 describes each
activity's target population and data collection methods. 



                                   Table III.6
                     
                        USDA's Data Collection Activities

                                                                    Data
                                                                    collection
Activity    Target population           Type and source of sample   method
----------  --------------------------  --------------------------  ------------
NFCS        Households in the 48        Stratified, multistage,     Personal
            contiguous states and       area probability sample     interview
            individuals residing in     with oversampling for low-  with the
            those households            income households           household
                                                                    food
                                                                    manager,
                                                                    including a
                                                                    7-day record
                                                                    of household
                                                                    food use;
                                                                    personal
                                                                    interview
                                                                    with
                                                                    household
                                                                    members on
                                                                    dietary
                                                                    intake,
                                                                    including 3
                                                                    consecutive
                                                                    days of
                                                                    dietary
                                                                    intake data
                                                                    collected
                                                                    with one 24-
                                                                    hour recall
                                                                    and a 2-day
                                                                    record

CSFII       Individuals in the 48       Stratified, multistage,     Personal
            contiguous states           area probability sample     interviews
                                        with oversampling for       with
                                        individuals in low-income   household
                                        households                  members on
                                                                    dietary
                                                                    intake,
                                                                    including 3
                                                                    consecutive
                                                                    days of
                                                                    dietary
                                                                    intake data
                                                                    collected
                                                                    with one 24-
                                                                    hour recall
                                                                    and a 2-day
                                                                    record

DHKS        Main meal planner or        Same as CSFII               Computer-
            preparer in households                                  assisted
            that participated in CSFII                              telephone
                                                                    interviews
                                                                    (supplemente
                                                                    d with in-
                                                                    person
                                                                    interviews
                                                                    for
                                                                    respondents
                                                                    without
                                                                    telephones)
--------------------------------------------------------------------------------
The focus of NFCS is on household use of food, including food costs,
food preparation, and food consumption.  NFCS data are intended to
inform policies related to food production and marketing, food
safety, food assistance, and nutrition education.  CSFII is intended
to complement NFCS in two ways.  First, it provides a more frequent
source of information than the decennial NFCS, and second, it focuses
on individual, rather than household, food consumption.  DHKS, a
follow-up to CSFII, is intended to support analyses of the
relationship between dietary intake and knowledge and attitudes about
dietary guidance and food safety. 

Although NFCS and CSFII vary in their target populations and
purposes, they are similar in their sampling approach (national with
oversampling for low-income population) and the focus on food
consumption.  Their similarities are reflected in the common themes
in the recommendations made by the primary users of the two data
collection systems.  DHKS users had somewhat different concerns about
data elements and data collection methods, but their comments were
otherwise consistent with remarks made about the other two systems. 

The major themes in the comments about the data systems were to
provide

  -- more specificity and detail about foods and better data on food
     composition;

  -- improved questions on dietary behavior;

  -- more information about health and demographic variables;

  -- reduced respondent burden and improved response rates;

  -- higher quality dietary recall data in general and, specifically,
     more nonconsecutive days of 24-hour recall;

  -- individual data and information on the individual's household;

  -- continuous or more frequent collection;

  -- longitudinal component;

  -- increased sample size and broadened coverage;

  -- more detail on racial, ethnic, age, and income groups;

  -- refined geographic area coverage, specifically state and
     substate data;

  -- more rapid release of the data;

  -- improved documentation; and

  -- dissemination of the data in alternative forms (for example,
     CD-ROM, formatted for use with statistical packages). 

Table III.7 provides more user suggestions and other issues from our
survey on USDA systems. 



                                   Table III.7
                     
                     User Suggestions for Improving USDA Data
                                Collection Systems

Type of change          Comment
----------------------  --------------------------------------------------------
Data elements           Collect more data on

                        Food eaten away from home

                        Food shopping access, prices, and behavior; food
                        preparation methods and facilities; food
                        storage; and safety

                        Use of salt, condiments, nutritional supplements,
                        specific foods (dairy, caffeine, water, fruits,
                        processed, seafood)

                        Improve

                        Quality and completeness of food composition data (newer
                        products, brand names, reliability of data)

                        Questions to assist linking diet and behavior (nutrition
                        knowledge and opinion, exercise, barriers and motivation
                        to change, participation in food programs)

                        Health data (by measuring rather than self-reporting
                        height, weight, health status)

                        Bring questions in line with current theory (DHKS);
                        standardize questions from year to year

Data collection         Streamline the instrument to reduce burden on
methods                 respondents

                        Use automation to improve response rates (also
                        telephones, home bar scanners)

                        Collect more days of recall data and more nonconsecutive
                        days

                        Use two periods of household records (shorten the 7
                        days) to measure better the usual food use

                        Ensure questionnaires are answered completely

                        Use multiple measures (telephone and in-person
                        interviews and mailed questionnaires)

Units of analysis       Focus on individual data (NFCS)

                        Focus on household data and individuals (CSFII)

Time of data            Need continuous survey or at least collect data more
collection              frequently

                        Collect NFCS data every 5 years and CSFII data in the
                        interim

                        Collect longitudinal data to track changes in individual
                        consumption (NFCS and CSFII)

Population group        Increase coverage of subpopulations and racial, ethnic,
coverage                and age groups

                        Increase sample size

                        Need clearer criteria for definition of race

                        Focus on high-risk groups

                        Integrate CSFII with NHANES sample

Geographic area         Need
coverage

                        More refinement of geographic detail (regions,
                        localities, areas of low density)

                        State-level estimates (allow states to collect their own
                        data and feed into national survey)

                        Use small-area estimation models

                        Provide specific estimates for major population centers

Ease of use             Collect data more frequently and allow more rapid access
                        to both published reports and raw data

                        Provide

                        User-friendly documentation

                        More detailed data on sampling design variables

                        Clear documentation on data tape and file format to
                        facilitate combining record types

                        Documentation on changes in format in food composition
                        database, codebook, and recipe file

                        Survey protocol and operations manual

                        More technical assistance (to nonnutritional
                        researchers)

                        On-line documentation

                        Data in common statistical package format

                        Lists of surveys and sources for both data and technical
                        assistance in professional journals
--------------------------------------------------------------------------------

--------------------
\1 Our survey also asked respondents if they had used and had
comments on a fifth NCHS survey--the National Health Interview Survey
on Vitamin and Mineral Supplements.  Only three respondents
identified themselves as primary users of NHIS-Vitamin, and none of
these had comments. 


CHARACTERISTICS OF THE USES OF
DATA COLLECTION ACTIVITIES
========================================================== Appendix IV

The tables in this appendix are based on the 440 responses from those
who have used at least one of the 14 data systems in the past 5
years.  Since some respondents pooled their answers, each of these
440 responses may represent one or more than one user.  The first two
tables describe some characteristics of the users in our sample. 
Table IV.1 shows the occupations that users identified themselves
with, by organizational setting.  To construct table IV.2, we asked
them what data collection activites they have used at all in the past
5 years. 



                                    Table IV.1
                     
                         Main Occupation of Respondents\a


         Federal      State      Local   Academic   Business   Other\b   Average
------  --------  ---------  ---------  ---------  ---------  --------  --------
Sample       112        125         28         98         37        40
 size

Occupation
--------------------------------------------------------------------------------
Servic        6%        14%        32%        16%        14%       33%       15%
 e
 deliv
 ery
Basic         47          6          0         57         35        28        32
 resea
 rch
Applie        29         10          7         24         49        13        21
 d
 resea
 rch
Progra        13         66         71          1          5        15        29
 m
 manag
 ement
 and
 plann
 ing
Other          7          6          0          2         16         3         5
--------------------------------------------------------------------------------
\a Column percentage totals exceed 100 percent because some users
identified more than one main occupation. 

\b Other settings include hospitals, nonprofit organizations, and
other charitable organizations. 



                                    Table IV.2
                     
                     Respondents' Use of Data Systems Within
                                the Past 5 Years\a


       Federal      State      Local   Academic   Business    Other\b    Average
---  ---------  ---------  ---------  ---------  ---------  ---------  ---------
Sam        112        125         28         98         37         40
 ple
 si
 ze

Data system
--------------------------------------------------------------------------------
NHE        31%        14%        36%        39%        30%        45%        30%
 FS
NHA         50         17         36         41         59         58         39
 NES
 II
NHA         57         17         39         29         30         48         35
 NES
 II
 I
HHA         38         10         11         21         24         23         22
 NES
NHI          9          8          7         10          3         23         10
 S-
 Vi
 ta
 mi
 n
NHI         11          5         14         16         14         20         12
 S-
 Ca
 nc
 er
BRF         13         71         39         15         11         33         33
 SS
PNS         13         47         50          6          3         33         25
 S
Ped         14         58         68          9          5         30         30
 NSS
Nav          2          0          0          0          0          5          1
 ajo
 HN
 S
Hea         13          7         11         10         19         23         12
 lth
 an
 d
 Di
 et
NFC         44         20         36         67         65         53         44
 S
CSF         46         10          7         55         46         40         35
 II
DHK         28         10          7         23         27         30         20
 S
--------------------------------------------------------------------------------
\a Column percentage totals exceed 100 percent because most users
checked two data systems. 

\b Other includes hospitals, nonprofit organizations, and other
charitable organizations. 

Tables IV.3-IV.5 show three aspects of respondents' satisfaction with
each data collection activity:  first, whether it provided for their
information needs; second, whether it met their data quality needs;
and third, whether they thought changes were needed to either
increase their confidence in or substantially increase their use of
the data system. 



                                    Table IV.3
                     
                     Users' Response to How Well the Current
                        Data Meet Their Information Needs


Data
collecti
on         Little or                                                      Sample
activity        none        Some    Moderate       Great  Very great        size
--------  ----------  ----------  ----------  ----------  ----------  ----------
NHEFS             2%         10%         45%         33%         10%          54
NHANES             3          14          37          31          15         101
 II
NHANES             2          16          34          28          19          90
 III
HHANES             0           7          37          33          22          28
NHIS-              0          23          46          15          15          13
 Cancer
BRFSS              4          30          42          19           6          85
PNSS               4          23          27          41           6          73
PedNSS             6          21          31          35           6          96
Health             8           8          33          33          17          13
 and
 Diet
NFCS               3          15          34          38          10         110
CSFII              6          13          37          36           8          89
DHKS              13           4          52          22           9          23
Median             4          15          37          33          10
--------------------------------------------------------------------------------


                                    Table IV.4
                     
                     Users' Response to How Well the Current
                             Data Meet Quality Needs


Data
collecti
on         Little or                                                      Sample
activity        none        Some    Moderate       Great  Very great        size
--------  ----------  ----------  ----------  ----------  ----------  ----------
NHEFS             4%         18%         36%         34%          8%          54
NHANES             2           8          32          36          22         101
 II
NHANES             4          11          33          34          19          90
 III
HHANES             0           7          22          37          33          28
NHIS-              0           8          23          62           8          13
 Cancer
BRFSS              6          22          42          25           5          85
PNSS               7          14          38          35           6          73
PedNSS             9          17          35          33           6          96
Health             0          17          42          25          17          13
 and
 Diet
NFCS               8          13          43          32           4         110
CSFII              5          15          42          31           7          89
DHKS               4           9          52          26           9          23
Median             4          14          37          34           8
--------------------------------------------------------------------------------


                                    Table IV.5
                     
                     Users' Response to Whether Changes Would
                      Increase Their Confidence in or Use of
                                     the Data


Da
ta
co
ll
ec
ti
on
ac
ti
vi             Probably                Probably              No basis to  Sample
ty       No         not   Uncertain         yes    Yes             judge    size
--  -------  ----------  ----------  ----------  -----  ----------------  ------
NH       8%         16%         24%         25%    27%                 0      54
 E
 F
 S
NH       14          17          14          28     26                1%     101
 A
 N
 E
 S
 I
 I
NH       13          17          16          34     16                 5      90
 A
 N
 E
 S
 I
 I
 I
HH       19          15           4          15     44                 4      28
 A
 N
 E
 S
NH        0          23           0          46     31                 0      13
 I
 S
 -
 C
 a
 n
 c
 e
 r
BR        5          20          16          30     28                 1      85
 F
 S
 S
PN        4          26          16          26     24                 3      73
 SS
Pe        6          26          11          27     29                 2      96
 d
 N
 S
 S
He       25          17          17           8     25                 8      13
 a
 l
 t
 h
 a
 n
 d
 D
 i
 e
 t
NF        3          12          13          30     38                 4     110
 CS
CS        8           9          10          36     33                 3      89
 F
 I
 I
DH        9           9          22          26     35                 0      23
 KS
Me        8          17          15          28     29                 3
 d
 i
 a
 n
--------------------------------------------------------------------------------

LIST OF EXPERTS
=========================================================== Appendix V

This appendix lists the expert advisers who assisted on this project. 
The advisers were organized into three panels:  core policy panel,
methodology panel, and data users panel. 


      CORE POLICY PANEL
------------------------------------------------------- Appendix V:0.1

Johanna Dwyer, D.Sc., R.D., Francis Stern Nutrition Center, New
England Medical Center and Tufts University Schools of Medicine and
Nutrition

Jean-Pierre Habicht, M.D., Ph.D., Division of Nutritional Sciences,
Cornell University

Catherine Woteki, Ph.D.\1


--------------------
\1 Dr.  Woteki withdrew from the panel when she was appointed to the
Office of Science and Technology Policy in the White House.  During
her participation in our work, she was the Director of the Food and
Nutrition Board of the National Academy of Sciences. 


      METHODOLOGY PANEL
------------------------------------------------------- Appendix V:0.2

Norman Bradburn, Ph.D., Director, National Opinion Research Center

Marilyn Buzzard, Ph.D., Director, Nutrition Coordinating Center,
University of Minnesota

Ricardo O.  Castillo, M.D., M.P.H., Co-Director, Pediatric
Gastroenterology, Stanford University Medical Center

Alan R.  Kristal, Dr.  P.H., Fred Hutchinson Cancer Research Center
and Department of Epidemiology, University of Washington

James Lepkowski, Ph.D., Institute for Social Research, University of
Michigan

Cheryl Ritenbaugh, Ph.D., Department of Family and Community
Medicine, University of Arizona

Laura Sims, Ph.D., Department of Nutrition and Food Science,
University of Maryland


      DATA USERS PANEL
------------------------------------------------------- Appendix V:0.3

Elizabeth Barnett, Ph.D., North Carolina Department of Environment,
Health, and Natural Resources

Doris Disbrow, Dr.  P.H., R.D., Center for Health Education

Pamela Haines, Dr.  P.H., R.D., Department of Nutrition, University
of North Carolina

Jay Hirschman, M.P.H., Senior Analyst, Food and Consumer Service,
U.S.  Department of Agriculture

Karen J.  Morgan, Ph.D., Senior Director, Nutrition and Consumer
Affairs, Nabisco Brands

Barbara Petersen, Ph.D., Technical Assessment Systems




(See figure in printed edition.)Appendix VI
COMMENTS FROM THE DEPARTMENT OF
AGRICULTURE
=========================================================== Appendix V



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)




(See figure in printed edition.)Appendix VII
COMMENTS FROM THE PUBLIC HEALTH
SERVICE
=========================================================== Appendix V



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


The following are GAO's comments on the letter from the Public Health
Service dated May 1, 1995. 

GAO COMMENTS

1. Our analysis of the responses to the open-ended questions
involved, first, sorting responses by data collection activity and
focus of the comment (such as data element, population coverage, ease
of use).  These responses were then aggregated to identify major
themes.  The comments made by FDA users were not identified as a
major theme across the many users of the different systems and, thus,
were not reported separately.  However, detailed summaries of the
responses were provided to the responsible agencies for their use. 

2. We have included a reference to the Directory of Federal and State
Nutrition Monitoring Activities for those readers who are interested
in more information on the data collection systems.  (See p.  3.)


MAJOR CONTRIBUTORS TO THIS REPORT
======================================================== Appendix VIII

PROGRAM EVALUATION AND METHODOLOGY
DIVISION

John Oppenheim, Assistant Director
Leslie Riggin, Assignment Manager
Lï¿½ Xuï¿½n Hy, Project Manager
James Joslin, Social Science Analyst
Venkareddy Chennareddy, Referencer
Elizabeth W.  Scullin, Communications Analyst

*** End of document. ***