Agent Orange: Actions Needed to Improve Communications of Air Force Ranch
Hand Study Data and Results (Letter Report, 12/17/1999, GAO/NSIAD-00-31).

Pursuant to a congressional request, GAO examined the conduct and
findings of the Air Force's Ranch Hand study on the long-term health
effects of exposure to herbicides, such as Agent Orange, in Vietnam, and
assessed the study's impact on determinations of veterans' compensation,
focusing on: (1) whether the study's findings and data have been
properly and promptly reported and disseminated; (2) the statistical
limitations of the study and whether they have been adequately reported
and communicated; (3) the measures established to monitor the study's
conduct and to prevent improper influence, particularly those involving
the Ranch Hand Study's Advisory Committee; and (4) the impact of the
study on determinations of diseases for which Vietnam veterans are
eligible to receive compensation benefits.

GAO noted that: (1) the Air Force has conducted all scheduled phases of
the Ranch Hand study and has periodically reported the results in
official reports; (2) however, there have been some delays in the
publication of morbidity and mortality findings in scientific journals
and in the update of a key report on reproductive outcomes; (3) no
journal articles on mortality or morbidity outcomes were published until
1990, even though the first mortality and morbidity reports were
published in 1983 and 1984, and the Advisory Committee repeatedly
recommended that such articles be published; (4) the Air Force reported
preliminary reproductive outcomes in 1984 but did not publish a more
detailed update until 1992 because it decided to verify the data
extensively and perform additional data analysis without releasing any
interim findings; (5) public access to study data is limited by the
small amount of data available and because its storage format is
difficult to use; (6) the public can access only data the Air Force
analyzed in 1987; (7) the Air Force intends to make all other data
available by the end of 2000; (8) the Ranch Hand study has a number of
inherent limitations, but the Air Force has not clearly or effectively
communicated these limitations to the public; (9) two limitations are
the difficulty in detecting increases in risks of rare diseases and the
fact that the study's findings cannot be generalized to all Vietnam
veterans; (10) to ensure independent review and prevent any appearance
of Air Force management bias, the study protocol mandated that: (a) an
independent group of scientists be established to monitor the study's
conduct; and (b) Air Force scientists, subject to review by the
monitoring group, have primary responsibility over the scientific
aspects of the study; (11) until 1989, the Advisory Committee did not
include any veterans' representatives, as required by the study
protocol; (12) according to documents GAO reviewed from 1984-1985, Air
Force management and the White House at the time, tried to direct some
of the Air Force scientists research; (13) although these early problems
were resolved through executive and congressional actions with the
extent of the Committee's outreach to veterans; and (14) partly on the
basis of the study's results, Congress passed legislation providing
compensation for veterans' children born with spina bifida.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-00-31
     TITLE:  Agent Orange: Actions Needed to Improve Communications of
	     Air Force Ranch Hand Study Data and Results
      DATE:  12/17/1999
   SUBJECT:  Herbicides
	     Medical research
	     Carcinogens
	     Health statistics
	     Health surveys
	     Birth defects
	     Statistical methods
	     Government information dissemination
	     Veterans
	     Veterans disability compensation
IDENTIFIER:  Vietnam War
	     Air Force Ranch Hand Study
	     Agent Orange

******************************************************************
** This file contains an ASCII representation of the text of a  **

** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

Report to the Ranking Minority Member Committee on Veterans Affairs, House
of Representatives

December 1999

AGENT ORANGE

Actions Needed to Improve
Communications of Air Force Ranch Hand Study Data and
Results
*****************

*****************

GAO/NSIAD-00-31

Letter                                                                     3

Appendixes

Appendix I:Objectives, Scope, and Methodology

                                                                         26

Appendix II:Statistical Power of the Ranch Hand Study

                                                                         28

Appendix III:Duties of the Ranch Hand Advisory Committee

                                                                         31

Appendix IV:Roles of Air Force, National Academy of Sciences, and
Department of Veterans Affairs in Evaluating Health Effects of Herbicides

                                                                         32

Appendix V:Comments From the Department of Defense

                                                                         34

Appendix VI:Comments From the Food and Drug Administration

                                                                         37

Table 1:  Minimum Detectable Effect in 1982     29

Table 2:  Minimum Detectable Effect in 1992     29

Table 3:  Minimum Detectable Effect in 2002     30

Table 4:  Summary of Roles and Criteria in Evaluating
the Health Effects of Herbicides                33

Figure 1:  Ranch Hand Study Milestones and Published
Articles and Reports                            10

Figure 2:  Communication Flow From the Advisory
Committee to Air Force Scientists Conducting the
Ranch Hand Study                                18

                                                      National Security and
                                             International Affairs Division

B-282636

December 17, 1999

The Honorable Lane Evans
Ranking Minority Member
Committee on Veterans Affairs
House of Representatives

Dear Mr. Evans:

From 1962 to early 1971, the United States sprayed herbicides, including
Agent Orange, over Vietnam. In the late 1970s, concerns began to emerge
over the long-term health problems of Vietnam veterans. Although these
veterans could have been exposed to many potential health hazards,
including pesticides, infectious diseases, and treatments for tropical
diseases, attention focused on herbicides. Several herbicides, including
Agent Orange, contain dioxin,/Footnote1/ which is known to cause a variety
of adverse health effects in animals. While there is scientific evidence
of some associations between exposure to herbicides (or the dioxin they
contain) and adverse human health conditions, the effect of this exposure
on human health remains controversial. 

There has been long-standing congressional interest in and concern about
the effects of exposure to herbicides such as Agent Orange. Congress has
held at least 26 hearings on the subject since 1978, some of them
involving scrutiny of scientific studies. One key effort to examine the
long-term health effects of servicemembers' exposure to herbicides in
Vietnam is an ongoing Air Force study known as the Ranch Hand study. This
study was designed to investigate whether exposure to herbicides in
Vietnam led or would lead to adverse health. To this end, the study
follows the health (morbidity) and mortality rates of the Ranch Hands--the
almost 1,300 Air Force personnel who sprayed herbicides from the air in
Vietnam--in contrast to those of a comparison group. This comparison group
consists of Air Force military personnel who were not involved in the
spraying and who were matched to the Ranch Hands in terms of age, race,
and military occupation. The 25-year study, which began in 1982 and is
scheduled to end in fiscal year 2006, is projected to cost over $140
million.

The study's protocol, which outlines the study's purpose, methods, and
procedures, requires the Air Force to disclose periodically the study's
findings in official reports and in a journal of stature. The protocol
also mandated a monitoring group, which currently consists of an Advisory
Committee of nine scientists (three of whom were nominated by veterans'
organizations), to oversee the conduct of the study and provide an
independent review of the study's findings. 

Findings from the Ranch Hand study, along with those of other studies
(e.g., on U.S. and foreign Vietnam veterans, chemical factory workers, or
populations exposed environmentally to dioxin), are used by the Department
of Veterans Affairs to determine whether veterans are eligible to receive
disability compensation for conditions believed to be connected to
military service. In 1984 and again in 1991, Congress revised the process
used by the Department of Veterans Affairs to determine whether Vietnam
veterans are eligible to receive compensation for health effects from
exposures to herbicides./Footnote2/ Since 1991, the Department has been
required to contract with the National Academy of Sciences to perform
reviews of scientific literature, including the Ranch Hand study, on the
associations between diseases and herbicide exposure. Largely on the basis
of these reviews, the Department of Veterans Affairs determines which
diseases show a positive association between herbicide exposure and the
disease. Vietnam veterans with such diseases are then eligible for
disability compensation.

Since its inception, the Ranch Hand study has been controversial. For
instance, news articles and statements in the Congressional Record have
alleged that government officials delayed and withheld information on the
study's findings, improperly influenced the study's design and
implementation, and failed to provide adequate veterans' representation on
the Advisory Committee. In addition, veterans' organizations such as the
American Legion and the Vietnam Veterans of America have criticized the
Department of Defense's conduct with regards to studies of Agent Orange. 

Recent congressional concerns have centered not on the scientific design
or implementation of the study but on the proper dissemination and
reporting of study results and on proper independent oversight. On the
basis of your concerns, we examined the conduct and findings of the study
and assessed the study's impact on determinations of veterans'
compensation. As agreed with your office, we did not evaluate the study's
scientific design. Specifically, we (1) assessed whether the study's
findings and data have been properly and promptly reported and
disseminated; 
(2) examined the statistical limitations of the study and whether they
have been adequately reported and communicated; (3) examined the measures
established to monitor the study's conduct and to prevent improper
influence, particularly those involving the Advisory Committee; and 
(4) assessed the impact of the study on determinations of diseases for
which Vietnam veterans are eligible to receive compensation benefits. 

To examine the conduct and findings of the Ranch Hand study, we obtained
and reviewed the study protocol, various study memoranda and
correspondence, published study reports and peer-reviewed journal
articles, executive summaries and Air Force press releases, and other
available documents related to the study. In addition, we interviewed
Ranch Hand study investigators, Advisory Committee members, Air Force
program officials, Department of Veterans Affairs officials, veterans'
representatives, and scientists involved in research on the health effects
associated with exposure to herbicides and dioxin. We also visited Brooks
Air Force Base to interview members of the Ranch Hand study team and to
review data collection and reporting procedures. Our scope and methodology
are described in more detail in appendix I.

Results in Brief

To date, the Air Force has conducted all scheduled phases of the Ranch
Hand study, including up to five full physical examinations of each
participating Ranch Hand and comparison, and has periodically reported the
results in official reports. However, there have been some delays over the
years in the publication of morbidity and mortality findings in scientific
journals and in the update of a key report on reproductive outcomes. The
pace of publication has increased in the past few years. No journal
articles on mortality or morbidity outcomes were published until 1990,
even though the first mortality and morbidity reports were published in
1983 and 1984, respectively, and the Advisory Committee repeatedly
recommended that such articles be published. Several reasons contributed
to the publishing delays, including the need to collect and analyze data
for five other morbidity and mortality reports published between 1985 and
1990. The Air Force reported preliminary reproductive outcomes in 1984 but
did not publish a more detailed update until 1992 because it decided to
verify the data extensively and perform additional data analyses without
releasing any interim findings. Although the number of published reports
has increased, public access to study data is limited by the relatively
small amount of data currently available and its storage format (magnetic
tape), which is difficult to use. Scientists and veterans' groups critical
of the study want access to all study data to verify the Air Force's
findings and to perform additional analyses. To date, the public can only
access data the Air Force analyzed in 1987. The Air Force intends to make
all other data available by the end of year 2000. 

Like many epidemiological studies,/Footnote3/ the Ranch Hand study has a
number of inherent limitations, but the Air Force has not clearly or
effectively communicated these limitations to the public. Two limitations
are the difficulty in detecting increases in risks of rare diseases
(because of the small size of the Ranch Hand population) and the fact that
the study's findings cannot be generalized to all Vietnam veterans
(because Ranch Hands and ground troops were exposed to different levels of
herbicides in different ways). In its first two morbidity reports in 1984
and 1987, however, the Air Force described the study's lack of findings
(i.e., few diseases with increased incidence among Ranch Hands) as
"reassuring," possibly leading to the misinterpretation that the study
showed herbicides to be safe. Although the Air Force no longer uses such
language, it still reports findings to the public through press releases
and executive summaries that do not make the study's limitations clear. 

During the study's first several years, some measures intended to ensure
that the study is conducted independently and without bias were not
carried out as planned. To ensure independent review and prevent any
appearance of Air Force management bias, the study protocol (published in
1982) mandated: (1) that an independent group of scientists (currently the
Advisory Committee) be established to monitor the study's conduct and 
(2) that Air Force scientists, subject to review by the monitoring group,
have primary responsibility over the scientific aspects of the study.
Shortcomings in the Advisory Committee's makeup and in the processes
through which the Committee and others communicated scientific advice were
not fully corrected until 1989. Until that year, the Committee did not
include any veterans' representatives, as required by the study protocol.
In addition, according to documents we reviewed dating from 1984 and 1985,
Air Force management and the White House at the time tried to direct
certain aspects of the Air Force scientists' research. These attempts
deviated from the protocol's requirement that Air Force scientists retain
primary responsibility over the study's scientific conduct. Furthermore,
the White House's actions bypassed review by the Advisory Committee.
Although these early problems were resolved through executive and
congressional actions, some problems remain with the extent of the
Committee's outreach to veterans. Although the Committee's public meetings
are announced in the Federal Register, the Committee has not directly
informed veterans' organizations of these meetings. In addition, the
Committee's informal process for soliciting nominations has resulted in
not all interested veterans' organizations being notified of opportunities
to nominate Committee members.

The Ranch Hand study has had significant impact on one decision regarding
compensation to Vietnam veterans. Partly on the basis of the study's
results, Congress passed legislation providing compensation for veterans'
children born with spina bifida./Footnote4/ The study has not contributed
to decisions by the Department of Veterans Affairs to compensate Vietnam
veterans for any other diseases. Because the Ranch Hand study's small
sample size provides limited potential for addressing possible links
between herbicide exposure and many forms of cancer, it has not
contributed to any National Academy of Sciences or Department of Veterans
Affairs recommendations regarding cancer. The finding of a possible
association between herbicide exposure and diabetes mellitus (diabetes),
first reported by the Ranch Hand study in 1991, has led to further review
of the issue by the Academy, as well as by Air Force and other scientists.
However, the Department of Veterans Affairs has not yet determined that
Vietnam veterans with the disease are eligible for compensation. 

We are recommending several additional actions intended to improve
communication of Air Force Ranch Hand study data and results.

Background

Before the Ranch Hand study began, official government and nongovernment
reviewers of the study protocol expressed concern that the public would
perceive the study as not credible because the fact that the Air Force
would conduct the study might give the appearance of conflict of interest.
An interagency working group that reviewed the study concluded, however,
that the appearance of conflict of interest could be properly and
adequately addressed through independent review and monitoring. Thus, an
Advisory Committee was created to regularly review and assess the conduct
of the study, review all changes to the study protocol, and provide
written comments and recommendations on the study's findings. The
Committee, chartered by the Department of Health and Human Services, was
initially responsible to the Agent Orange Working Group, an interagency
group that monitored all government activities related to herbicides such
as Agent Orange./Footnote5/ The Agent Orange Working Group was headed by
the Department of Health and Human Services and included representatives
from many government agencies and offices, including the Department of
Defense, the Department of Veterans Affairs, and the White House Office of
Science and Technology Policy. In 1989, the Department of Health and Human
Services transferred administration of the Committee to the Food and Drug
Administration. The duties of the Advisory Committee are described in more
detail in appendix III.

The process of translating scientific evidence into policy decisions on
compensation benefits for veterans has also been controversial. Scientific
studies typically strive to produce evidence of a "cause-and-effect"
relationship, and use a relatively high standard of proof (e.g., 95-
percent confidence). In 1989, a federal court ruled that the Department of
Veterans Affairs' use of similar standards in making compensation
decisions was too restrictive./Footnote6/ According to the court, Congress
had intended the Department of Veterans Affairs to require only a
"statistical association" with a disease. Furthermore, the court concluded
that veterans were to be given the benefit of the doubt in compensation
decisions. Subsequently, in 1991, Congress established the current process
for these decisions, including biennial reviews of the scientific
literature by the National Academy of Sciences./Footnote7/ The Academy has
conducted three such reviews for Agent Orange, the most recent published
in 1999. Additional information on the roles of the Ranch Hand study, the
National Academy of Sciences, and the Department of Veterans Affairs is in
appendix IV.

Publication of Study Results and Release of Study Data Were Slow in the
Early Years

Publication of study findings in peer-reviewed journals was slow to begin,
but the number of articles has increased considerably in recent years (see
fig. 1). An update of a report on reproductive outcomes was delayed to
allow the completion of additional analyses and data collection. The Air
Force has also been slow to release study data to the public. Only
analyzed data from one physical examination (not all five examinations
conducted thus far or all collected data) is currently available. In
addition, the data is available only on magnetic tape rather than a more
easily accessible format such as compact disc. 

Figure****Helvetica:x11****1:    Ranch Hand Study Milestones and Published
                                 Articles and
                                 Reports
*****************

*****************

Note: "Journal articles" are articles (not including letters, comments,
and errata) included in the National Library of Medicine's MEDLINE
database as of October 1999. "Reports" are official government reports
published by the Air Force.

*: Data currently publicly available.

Source: Air Force, MEDLINE.

Publication of Study Findings Slow to Begin
-------------------------------------------

In spite of the requirement in the study protocol as well as guidance from
the Advisory Committee that study findings be disclosed periodically in
scientific journals, no journal publications appeared until 1989, nearly 
7 years after the study had begun. Publication of study results on
morbidity or mortality did not begin in scientific journals until 1990.
The study protocol initially mandated that the Air Force issue official
reports on morbidity after each physical examination and on mortality at
least every 
5 years and that the results of the study be published in a journal of
stature. The Air Force did not publish journal articles, however, until
after it had obtained the third round of physical examinations in 1987 and
after several morbidity and mortality reports had been released. Several
reasons contributed to the delay, including the need to collect and
analyze data for five other morbidity and mortality reports published
between 1985 and 1990. The Advisory Committee repeatedly urged publication
of study findings in journals. In 1986, it recommended that results be
published in the "open literature," and in 1987 it repeated the
recommendation, stating that "first and foremost, there is an urgent need
to publish in widely read peer-reviewed medical journals." The pace of
article publication has accelerated since 1990. Eleven articles on
morbidity and mortality had been published by mid-1999, and Air Force
scientists told us that over a dozen articles are currently in submission
or preparation for publication in peer-reviewed journals. 

Update of Report on Reproductive Outcomes Was Delayed
-----------------------------------------------------

An update of a preliminary reproductive outcome report was delayed 
8 years because the Air Force performed additional data collection and
analyses without releasing any interim information. The Air Force first
reported preliminary reproductive outcomes in a baseline morbidity study
in 1984. It subsequently began verifying all medical records to check for
birth defects instead of relying on reports of interviews with Ranch
Hands, comparisons, and their spouses, which can be inaccurate. The Air
Force anticipated completion of the effort within 1 year. Although a draft
interim update, using partially verified data, was prepared later in 1984,
the Advisory Committee suggested it not be released before additional work
was done. The Air Force later expanded its verification of medical records
through age 18 (rather than up to age 1) to identify birth defects that
may not have manifested themselves earlier. This expanded verification was
not completed until late 1988. From 1987 through 1990, the Air Force (in
collaboration with the Centers for Disease Control and Prevention)
measured dioxin levels in the blood of all study participants as part of
its morbidity study and decided to analyze birth defects also using these
measurements before publishing a report. The reproductive outcome update
with all the additional analyses and data was finally published in 1992. 

Although the verification and additional analyses ultimately strengthened
the scientific basis of the reproductive outcome portion of the study, the
length of the delay, in conjunction with the lack of any interim report,
led to concerns by some veterans and scientists that the study had been
purposely delayed to suppress information. One reason underlying the
concern was the apparently high standard of verification. Through our
review of other birth defect studies and interviews with scientists, we
found that the degree of birth defects verification the Air Force
accomplished was highly unusual and virtually unprecedented for a study of
its size. We examined two other studies of birth defects in children of
Vietnam veterans. One of them verified only the health status of babies at
birth, the other verified only those with reported birth defects through
age 1, not those reported as healthy. Because the Ranch Hand study
produced evidence that led Congress to authorize compensation for Vietnam
veterans whose children have spina bifida, one veterans' organization told
us that earlier reporting, even just of preliminary findings, might have
led to earlier compensation. In addition, one scientist told us that such
preliminary information could have been useful for family planning or
prenatal care decisions. 

Public Access to Study Data Is Currently Limited 
-------------------------------------------------

Two major factors currently limit public access to study data: (1) only
one set of analyzed data--not all sets of analyzed data or all collected
data--is available and (2) data is only available on magnetic tape, not on
a more common format such as compact disc or the Internet. Two scientists
and one veterans' group told us they were interested in obtaining study
data. They said that access to study data would enable them to verify the
accuracy of the Air Force analyses and to perform additional analyses on
the data. They cited the limited amount of study data currently available
and its format among the barriers to their use of study data for these
purposes. 

Currently, only data files from Air Force analyses of 1987 physical
examination data are available, while data files from 1982, 1985, 1992,
and 1997 are not. The Air Force database consists of two types of files:
(1) raw data files, which include all the data collected at physical
examinations, and (2) analysis data files, which were used to perform
analyses for study reports. The content of raw and analysis data files
overlap, but each contains data not included in the other. The data
currently available to the public are analysis files from 1987. Raw data
files from 1987 and 1992 have been processed by the Air Force for release
and are in the process of becoming publicly available. Air Force officials
report that they expect all raw and analysis data files, including
reproductive outcome data, to be available by the end of year 2000. One
scientist and one veterans' group representative told us they believe the
Air Force should release all the study's data to permit additional
analysis of the data. According to these individuals, release of the data
is particularly important because of past allegations that government
officials may have improperly influenced the study's conduct.

Currently, Ranch Hand study scientists send data on computer diskettes to
the Defense Technical Information Center, which transfers the data to
magnetic tape and sends it to the National Technical Information Service
of the Department of Commerce. The Service said it provides the data to
the public in the same format in which it is received, and charges about
$450 to cover costs. In comparison, two other major government health
databases we examined are both available on compact disc. These are the
National Cancer Institute's "Surveillance, Epidemiology, and End Results"
database of cancer incidence throughout the United States, available free
of charge from the Department of Health and Human Services, and the
National Health and Nutrition Examination Survey, a Centers for Disease
Control and Prevention database of the health and nutritional status of
the U.S. population, that can be purchased for $20 from the Government
Printing Office. Several veterans' representatives said that they do not
have the necessary computer equipment to use magnetic tapes, and one
scientist told us that other formats such as compact discs would be easier
to use. Air Force officials also told us they are planning to release the
data on the Internet but noted that the National Technical Information
Service would remain their primary means of release.

Study Limitations Have Not Been Clearly Communicated to the Public

All scientific studies have inherent limitations. Two major limitations of
the Ranch Hand study are the difficulty in detecting increases in risks of
rare diseases (because of the small size of the Ranch Hand population) and
the fact that the study's findings cannot be generalized to all Vietnam
veterans (because Ranch Hands and ground troops were exposed to different
levels of herbicides in different ways). In spite of the study's sensitive
and controversial nature, early study reports contained language that may
have been misinterpreted to mean that the study showed herbicides were
safe. Although the Air Force no longer uses such language, recent press
releases and executive summaries still do not clearly communicate the
study's limitations to the public. 

Two Major Study Limitations
---------------------------

The small number of participants in the Ranch Hand study--currently fewer
than 1,200 Ranch Hands and less than 1,800 comparisons--makes it very
difficult to detect a statistically significant increase in the risk of
rare diseases. For instance, the lifetime risk of non-Hodgkins lymphoma 
(a cancer often linked to herbicide exposure) in Caucasian males in the
United States is about 2 percent. Even if the Ranch Hand study were to
follow participants through their entire lives (which it does not),
because of the requirement for statistical significance, it could not
confidently report any adverse effect unless the incidence of the disease
in Ranch Hands were at least 3.4 percent. Appendix II contains a more
detailed explanation of the study's statistical limitations as they relate
to the size of its study population and the detectable increased risk of
cancer.

Another limitation is that it is difficult to generalize the results of
the study to all Vietnam veterans because Ranch Hands were exposed to
herbicides in different ways than most ground troops in Vietnam. For
instance, by comparing dioxin blood levels with Ranch Hands' reported
exposures to herbicides, Air Force scientists found evidence that Ranch
Hands were exposed to herbicides primarily through their skin. Many ground-
troop veterans, however, reported exposure to herbicides through other
mediums such as contaminated food and water and contaminated clothing worn
for extended periods of time. They believe that Ranch Hands were only
minimally exposed through these mediums because they consumed
uncontaminated food and water, wore clean clothes, and could wash
regularly. These differences have not been extensively studied, so little
is known about their potential effects. 

However, blood measurements of dioxin levels suggest, though not
conclusively, that Ranch Hands' exposure levels were significantly higher
than both those of the study's comparison group and of many ground troops
who have been tested./Footnote8/ Because the comparison group was not
involved in spraying herbicides, its low dioxin levels were consistent
with lack of exposure to herbicides. Congress had proposed a program to
test veterans' blood for dioxin, under the condition that such a program
would make a contribution to scientific knowledge, but the National
Academy of Sciences stated that such testing would not be useful unless it
were part of a specific scientific study. Therefore, most Vietnam
veterans' blood dioxin levels are untested. Furthermore, according to
several scientists we interviewed and scientific documents we reviewed,
low levels of dioxin in the blood today do not guarantee that an
individual had low amounts of exposure to herbicides in Vietnam. Reasons
include (1) different herbicides contain different amounts of dioxin; (2)
the general population is also exposed to detectable levels of dioxin
masking exposures during the Vietnam War; (3) dioxin is slowly excreted
from the body, so subjects may have had higher dioxin levels in the past;
and (4) the rate at which dioxin is excreted differs, depending on the
amount of body fat, so individuals with similar dioxin levels today may
have had different exposures in the past.

Early Reports Could Have Led to Misinterpretation of Findings to Mean
Herbicides Were Safe
---------------------------------------------------------------------------

The importance of communicating the study's limitations to the public has
been stressed and highlighted by a number of scientists and officials over
the years. As far back as 1980, the predecessor of the Agent Orange
Working Group/Footnote9/ recommended that the study proceed only on
condition that the public "clearly understand that the stated health goal
in the Air Force [Ranch Hand study] Protocol may not be fully realized"
and that this "does not imply flaws in protocol design; it is to emphasize
the inherent limitation of the study size." In addition, in 1984 the White
House Office of Science and Technology Policy urged the Air Force to
respond to criticisms and perceived misunderstandings of the study by
discussing more explicitly "the meaning of the number of cases found (or
not found) ****Symbol:xbc**** for each disease included in the study,(c)
adding that (r)the discussion should be in terms of the confidence
interval of these numbers and their comparison to general population data
on these diseases.(c) 

In its early reports, and particularly in executive summaries and press
releases, the Air Force used language that may have led the results of the
Ranch Hand study to be interpreted inaccurately. As far back as 1980, a
review panel of government scientists emphasized that "it needs to be
clearly understood that failure to identify increased risk in a variety of
health parameters is to be interpreted as inconclusive and not necessarily
a true lack of effect" (emphasis in original). However, in describing the
results of the first two morbidity reports, published in 1984 and 1987,
the Air Force described the overall study conclusions as "reassuring"
because they detected few statistically significant increased risks. The
use of such language, as well as a statement by the Deputy Air Force
Surgeon General at a press conference in 1984 saying that the results
showed "nothing that would keep us from using it [herbicides such as Agent
Orange] again," may have led to the misinterpretation that the Ranch Hand
study proved that herbicides were safe. Since 1990, the Air Force has not
used this type of language. Current Air Force scientists acknowledge that
the word "reassuring" in particular may have led to improper
interpretations. 

Study Limitations Not Fully and Clearly Communicated to the Public in
Executive Summaries and Press Releases
---------------------------------------------------------------------------

We reviewed all Air Force reports, executive summaries, and press releases
on the Ranch Hand study and found that the Air Force still reports study
results to the public without consistently making clear that any lack of
conclusive evidence may be due to the study's limitations--not necessarily
a lack of long-term health effects from herbicides. While study
limitations have been addressed in the full text of the reports, they have
not been consistently explained in executive summaries and press releases,
which are more accessible to the general public. For example, the
executive summary and press release for the baseline morbidity study,
published in 1984, only stated that statistical limits exist, and gave no
further description. The executive summary of the third morbidity report,
published in 1990, stated that the study had a good chance of detecting a
statistically significant effect only if a high number of study subjects 
(i.e., 5 percent or more in the comparison group, and 10 percent or more
in the Ranch Hands) contracted a particular disease. In addition, only in
the executive summary and press release of its 1991 morbidity report did
the Air Force mention study limitations due to the differences in exposure
routes and levels between Ranch Hands and ground troops. However, in this
case, the Air Force only stressed the limits with which any positive
evidence of ill health could be applied to other Vietnam veterans and did
not mention the limits on what conclusions could be drawn from a lack of
evidence in the study. All other press releases and executive summaries
have not mentioned study limitations at all. 

Several scientists we spoke with also voiced concern about the Air Force's
public reporting of study results. One scientist with expertise in dioxin
research told us that Air Force press releases and executive summaries
continue to be misleading because they are not adequately balanced by an
account of the limitations of the study. A scientist on the Advisory
Committee also commented at a Committee meeting in August 1999 that the
draft of the upcoming Air Force morbidity report stressed the study's lack
of findings too strongly and did not adequately discuss the limits to
inferring either positive or negative effects. A former Air Force
scientist, however, told us that the Air Force's public reporting of study
findings should be viewed in the context of media and unsubstantiated
reports of major incidences of disease and death caused by herbicides.

Many Early Oversight Problems Have Been Resolved, but Some Veterans Groups
Want More Outreach

Two measures specified by the study protocol to ensure independent review
and prevent any appearance of Air Force management bias were not followed
prior to 1989. These were the requirements that the Advisory Committee
include scientists nominated by veterans' organizations and that Air Force
scientists, subject to review by the Advisory Committee, retain primary
responsibility over scientific aspects of the study. Executive agencies
(in 1985 and 1989) and Congress (in 1988 and 1990) took actions to correct
these problems. However, problems remain with how well the Advisory
Committee keeps veterans' organizations informed of its activities.

Early Problems With the Advisory Committee's Composition and the
Communication of Scientific Advice
---------------------------------------------------------------------------

Prior to 1989, the Advisory Committee did not fulfill a key requirement of
the study protocol: that it include scientists nominated by veterans'
organizations. The requirement stated that about one-third of Committee
members should be scientists nominated by veterans' organizations.
According to our review, the Committee did not include any such scientists
from 1982 through 1989. Documents we reviewed indicate that one reason was
that the Agent Orange Working Group, which was responsible for
administering the Committee, appeared to view the protocol's description
of the Advisory Committee as only giving a suggestion.

The second problem that existed before 1989 was that outside entities
tried to direct how Air Force scientists conducted certain aspects of the
study, even though the study protocol specified that Air Force scientists
"are responsible for ****Symbol:xbc**** all data analysis, and for all
data interpretation of analyses subject to review by the outside
monitoring group.(c) This problem stemmed from the fact that before 1989,
the Committee did not communicate its formal recommendations directly to
Air Force scientists conducting the study. Rather, the Committee gave its
formal recommendations first to the Agent Orange Working Group for
approval, creating opportunities for others to influence the study. Figure
2 shows the flow of communications from the Committee to study scientists
before and after 1989. 

Figure****Helvetica:x11****2:    Communication Flow From the Advisory
                                 Committee to Air Force Scientists
                                 Conducting the Ranch Hand Study

*****************

*****************

Note: Solid lines indicate formal communication (e.g., agency memoranda).
Dotted lines indicate informal communication (e.g., verbal comments or
meeting minutes).

Source: Air Force.

Documents we reviewed from the period before 1989 indicate that the Air
Force and the White House Office of Science and Technology Policy 
(a member of the Agent Orange Working Group) tried to direct certain
aspects of the Air Force scientists' research. For instance, in a
memorandum sent in October 1984, the commander of the School of Aerospace
Medicine (where the study is conducted), Brooks Air Force Base, ordered
the Air Force scientists to perform certain analyses for mortality reports
and to present the results in a specific format. The memorandum noted that
the direction came from the Air Force Deputy Surgeon General and that it
was "in accordance with the comments of the Advisory Committee." However,
according to the study protocol, Air Force scientists, not Air Force
management, have ultimate responsibility for all data analysis and
interpretations. One of the scientists involved at the time in the study
expressed concern that the actions ordered by the commander would weaken
the study or de-emphasize the significance of the results. In addition, in
a letter also sent in 1984, the White House Office of Science and
Technology Policy wrote that it had "directed that the Air Force undertake
****Symbol:xbc**** actions,(c) including the extensive verification of
reported birth defects. This verification contributed to the 8-year delay
in updating study findings on reproductive outcomes. Again, this direction
infringed on the Air Force scientists' decision-making responsibilities in
the scientific conduct of the study. Furthermore, the White House later
provided guidance directly to the Air Force, requesting that the Advisory
Committee ensure that this guidance be implemented. However, the study
protocol only authorized the Advisory Committee to provide written
comments and recommendations on the conduct of the Ranch Hand study to the
White House Office of Science and Technology Policy, not the reverse. The
protocol also required that all scientific aspects of the study undergo
review by the Advisory Committee. Both Air Force management and the
Advisory Committee later expressed concern about the appropriateness of
the White House's communication to the Air Force scientists and sought
clarification from the Agent Orange Working Group. 

Many Early Problems Solved by 1990
----------------------------------

In 1985, the Agent Orange Working Group decided that all recommendations
intended for the Air Force scientists conducting the study should be
communicated first to the Committee. This decision was made primarily in
response to concerns about conflicting advice received by the Air Force
scientists from bodies such as the White House Office of Science and
Technology Policy. We found no evidence that any recommendations made
since then have bypassed the Committee. In 1988, Congress mandated that
veterans' organizations be allowed to nominate

three of the Committee's nine members,/Footnote10/ and we verified the
Committee's compliance with this requirement. In 1989, the Department of
Health and Human Services transferred administration of the Committee from
the Agent Orange Working Group to the Food and Drug Administration, thus
eliminating the Agent Orange Working Group as an intermediary in the chain
of communication. Congress passed additional legislation in 
1990 stating that "no officer or employee of the Federal Government may
interfere in or impair direct communication between the Advisory Committee
and ****Symbol:xbc**** [the Air Force] scientists.(c)/Footnote11/ 

Remaining Problems in Informing Veterans of Committee Activities
----------------------------------------------------------------

Although procedural problems in the early years of the study have been
resolved, veterans' groups and several scientists we spoke with continue
to question the credibility of the study. In addition, representatives of
veterans' groups told us that the Committee's communication with veterans'
organizations has been intermittent and inconsistent. Two veterans' groups
told us that they had not been notified repeatedly by either the Air Force
or the Advisory Committee of public Committee meetings. Although they
acknowledged that public Committee meetings are announced in the Federal
Register, they stated that similar committees, such as those run by the
Department of Veterans Affairs or the National Academy of Sciences, are
more proactive in keeping veterans informed of their activities. One
group, the Vietnam Veterans of America, which represents veterans with the
most interest in herbicides such as Agent Orange, told us that lack of
notification persisted even after it had requested directly to the
Committee that it be informed of meetings in advance. 

In addition, the process for soliciting nominations for members of the
Advisory Committee has been relatively unstructured and has led to
incomplete dissemination of Committee membership openings, as well as
inaccuracies in the identification of veterans' nominees. The Food and
Drug Administration is responsible for soliciting nominations from
veterans' organizations. However, openings have not been widely
publicized. Rather, the Food and Drug Administration reported using an
informal process, which involves asking previous Committee consultants to
become nominees and telephone calls to some veterans' organizations. The
Vietnam Veterans of America told us that as a consequence, it has never
been asked to nominate Committee members. Our review of Committee
documents showed that veterans' organizations that have nominated
Committee members include the American Legion, the Veterans of Foreign
Wars, and the Ranch Hand Association. In addition, our review revealed
that one current Committee member was incorrectly identified as having
been nominated by a veterans' organization and that another Committee
member who was nominated by a veterans' organization was not identified as
such.

Study Has Had Limited Impact on Determination of Diseases for Which
Veterans Are Eligible to Receive Compensation

The Ranch Hand study had significant impact on the decision to provide
compensation for veterans' children born with spina bifida. The study has
not contributed to decisions by the Department of Veterans Affairs to
compensate veterans for any other diseases. The finding of a possible
association between herbicide exposure and diabetes mellitus, first
reported by the Ranch Hand study in 1991, has led to greater discussion by
the National Academy of Sciences and further study by Air Force and other
scientists, but Vietnam veterans with the disease are not yet eligible for
compensation. 

Study Contributed to the Decision to Compensate for Spina Bifida
----------------------------------------------------------------

The most significant impact of the Ranch Hand study to date has been on a
decision to allow compensation to Vietnam veterans' children born with
spina bifida. According to a scientist at the National Academy of
Sciences, because of the high quality of the Ranch Hand birth defect
study, the Academy decided in 1996 to upgrade its evaluation of evidence
for an association between herbicide exposure and spina bifida from
"insufficient/inadequate" to "limited/suggestive." Subsequently, the
Department of Veterans Affairs requested, and Congress approved,
legislation allowing the Department of Veterans Affairs to provide
compensation to Vietnam veterans' children with the disease.

Study Has Not Contributed to Decisions to Allow Compensation for Other
Diseases 
---------------------------------------------------------------------------

The Ranch Hand study has had limited impact on either National Academy of
Sciences reports or Department of Veterans Affairs decisions concerning
associations between herbicide exposure and other diseases./Footnote12/
Although findings from other studies have enabled veterans with any of
several cancers to be eligible for compensation, the Ranch Hand study has
had almost no impact on these decisions because of the small size of the
Ranch Hand population and the relative rarity of many cancers. In
addition, scientific studies such as the Ranch Hand study are involved
only in determining for which diseases veterans may be eligible for
compensation, not in any other part of the compensation process. There are
other requirements, such as obtaining medical diagnosis of a disease and
determination of disability, that Vietnam veterans must satisfy in order
to receive compensation. Moreover, reports and articles by the Ranch Hand
study comprise only a small fraction of the information the National
Academy of Sciences reviews and the Department of Veterans Affairs then
considers when weighing scientific evidence. 

A finding of a possible association between herbicide exposure and
diabetes mellitus was first suggested in a Ranch Hand study report in
1991. This and subsequent findings by the study have led to increased
research on the subject and more reviews by the National Academy of
Sciences. Although in its February 1999 report the Academy described the
evidence of an association between herbicide exposure and diabetes
mellitus as "equivocal," the Academy still concluded, as it did in its
earlier reports, that there was "inadequate/insufficient" evidence for
such an association. However, a Task Force within the Department of
Veterans Affairs recommended in April 1999 to allow Vietnam veterans with
diabetes mellitus to become eligible for compensation. The Task Force
wrote that because the weights of positive and negative evidence were
approximately equal, the requirement to give veterans the benefit of the
doubt (as stated by law) necessitated providing compensation for diabetes
mellitus. The Department of Veterans Affairs has asked the Academy to
conduct an additional evaluation of the scientific literature on the
subject. The evaluation is scheduled to be completed by the end of 1999. 

Conclusions

The Air Force has conducted many aspects of the Ranch Hand study in a
rigorous manner. However, some past and present problems in the conduct of
the study have led some veterans to view the study with suspicion and to
express concerns about the study's conduct. In particular, several
veterans' groups and scientists are critical of the study's methods and
results and want greater access to study data. Until all study data is
publicly available, people critical of the study cannot fully verify the
Air Force's analyses. Making all study data publicly available would
increase the credibility of Ranch Hand study results. In addition, without
full access to data, outside scientists cannot perform alternative or
additional analyses, which would facilitate open scientific debate on the
merits of the Air Force's methodologies and analyses. The Air Force
anticipates making all its study data available to the public by the end
of year 2000, but those with interest in the data say that unless the data
is available in a more easily accessible format (such as compact disc or
the Internet), they would have difficulty in using it.

In addition, while communication of study limitations to the public has
improved over the years, additional improvements are possible. Lack of
sufficient knowledge of the study's limitations can lead to misleading or
incorrect interpretations of the study's findings. Language used in early
reports that may have led to such interpretations is no longer being used,
but because of the study's sensitive and controversial nature, it is
important that publicly accessible documents such as executive summaries
and press releases provide accurate information on the study's limitations. 

Early problems in the Advisory Committee's composition and communications
have been remedied, but problems remain with the Committee's outreach to
veterans. The Committee's reliance on indirect means, such as the Federal
Register, to notify veterans' organizations of its activities perpetuates
the impression that the Committee does not seek veterans' input. Moreover,
better notification of Committee openings would help ensure that veterans'
groups perceive the Committee as fulfilling its role as an independent and
unbiased oversight body.

Recommendations

To facilitate public access to study data and more effective communication
of study limitations, we recommend that the Secretary of Defense direct
the Air Force scientists in charge of the Ranch Hand study, in
consultation with the Advisory Committee, to 

o establish and publicize a timetable for the release of all study data
  and provisions to release the data in a format (such as compact disc or
  the Internet) that is easily accessible to the general public and 

o include more information on the study's limitations in press releases
  and executive summaries, which should address the limited applicability
  of study results (especially negative results) to other Vietnam
  veterans and the limited ability of the study to detect small to
  moderate increases in risks of rare diseases.

In addition, to facilitate dissemination of information on the Advisory
Committee's activities to veterans, we recommend that the Secretary of
Health and Human Services direct the Committee's Executive Secretary at
the Food and Drug Administration to provide direct and timely notification
to veterans' organizations of scheduled Advisory Committee meetings and of
opportunities for veterans to nominate Committee members.

Agency Comments

In written comments on a draft of this report, the Department of Defense
concurred with our recommendations and indicated that it is taking steps
to address them. The Department also commented on our finding that the
publication of study findings in scientific journals was slow to begin,
stating that the time required to write and publish journal articles is
typically 3-5 years and that work on the papers published in 1990 actually
started in 1986. While we recognize that the publication process can be
lengthy, it should be noted that the journal articles published in 1990
only included analysis of data collected in 1987 and reported in 1990 in
an official Air Force report. No journal articles were published earlier
based on data collected in 1982 and 1985 and respectively released in Air
Force reports in 1984 and 1987. Other comments from the Department were
technical in nature, and we incorporated them where appropriate. Comments
from the Department of Defense are reproduced in their entirety in
appendix V.

The Food and Drug Administration also provided written comments on a draft
of this report. It stated that in general it found the report well done
and accurate. With regard to our recommendation that the Secretary of
Health and Human Services direct the Advisory Committee's Executive
Secretary to facilitate dissemination of information to veterans, it
indicated that they are working with the Office of Veterans Affairs and
Military Liaison of the Department of Health and Human Services to ensure
that veterans' organizations are notified in a timely manner of Advisory
Committee meetings and vacancies. The Food and Drug Administration also
provided technical comments that we incorporated where appropriate.
Comments from the Food and Drug Administration are reproduced in their
entirety in appendix VI.

As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from
its issue date. At that time, we will send copies of this report to other
appropriate congressional committees. We will also send copies to the
Honorable William S. Cohen, Secretary of Defense; the Honorable 
F. Whitten Peters, Secretary of the Air Force; and the Honorable 
Donna E. Shalala, Secretary of Health and Human Services. In addition, we
will make copies available to others upon request. 

If you have any questions about this report, please call me at
(202) 512-3652 or John Oppenheim at (202) 512-3111. Weihsueh Chiu was a
key contributor to this report.

Sincerely yours,

*****************

*****************

Kwai-Cheung Chan
Director
Special Studies and Evaluations

--------------------------------------
/Footnote1/-^The chemical 2, 3, 7, 8-tetrachlorodibenzo-p-dioxin.
/Footnote2/-^P.L. 98-542 (1984) and P.L. 102-4 (1991).
/Footnote3/-^Scientific studies of the incidence, distribution, or control
  of diseases in human populations.
/Footnote4/-^Spina bifida is a birth defect in which the vertebral column
  (backbone) fails to close and that may allow herniation of the spinal
  cord.
/Footnote5/-^The Agent Orange Working Group was replaced by the Agent
  Orange Task Force in 1990. The final meeting of the Agent Orange Task
  Force occurred in 1994.
/Footnote6/-^Nehmer v. U.S. Veterans' Administration, 712 F. Supp. 1404
  (N.D.Cal.1989).
/Footnote7/-^P.L. 102-4 (1991).
/Footnote8/-^Since 1991, the study's morbidity reports have concentrated
  on analyzing the association between dioxin levels and health rather
  than the health contrast between Ranch Hands and the comparison group.
  Almost half of the Ranch Hands have dioxin levels similar to those of
  the comparison group, further reducing the study size when applying the
  analysis of dioxin levels versus health.
/Footnote9/-^The Scientific Panel of the Interagency Work Group on Phenoxy
  Herbicides and Contaminants.
/Footnote10/-^P.L. 100-687 section 1205 (1988).
/Footnote11/-^P.L. 101-510 section 1468 (1990).
/Footnote12/-^ Currently, the Department of Veterans Affairs allows
  compensation for nine diseases (including six cancers) in veterans and
  one birth defect in children of veterans.

OBJECTIVES, SCOPE, AND METHODOLOGY
==================================

Our objectives were to (1) assess whether the study's interim findings and
data have been properly and promptly reported and disseminated, 
(2) examine the statistical limitations of the study and whether they have
been adequately reported and communicated, (3) examine the measures
established to monitor the study's conduct and to prevent improper
influence, and (4) assess the impact of the Ranch Hand study on National
Academy of Sciences reports and Department of Veterans Affairs
determinations regarding diseases for which Vietnam veterans are eligible
to receive compensation benefits. 

To assess whether the study's interim findings and data have been properly
and promptly reported and disseminated, we reviewed the study protocol and
memoranda and correspondence by the Air Force and the Advisory Committee
to determine the schedule for reporting study findings. In addition, we
reviewed these documents to examine the reasons for any delays. We also
obtained and reviewed published study reports and 
peer-reviewed articles to determine the actual date of these publications.
We conducted site visits to Brooks Air Force Base to interview scientists
in charge of the Ranch Hand study and to review data collection and
reporting procedures. Finally, we reviewed the current state of public
access to study data and queried several scientists and veterans'
organizations about their ability to access the data.

To examine the statistical limitations of the study and the reporting and
communication of these limitations, we reviewed the study protocol, the
study reports, and the memoranda and correspondence obtained from the Air
Force and the Advisory Committee. In addition, we reviewed literature on
epidemiology and communication of health risks. We reviewed study reports,
executive summaries, and Air Force press releases to assess how study
limitations have been communicated. We also queried scientists about
aspects of communicating study limitations. Using the National Cancer
Institute's Surveillance, Epidemiological, and End Results database, we
determined the expected incidence of several cancers in Ranch Hands based
on the rate in the U.S. population and performed a calculation of the
statistical power of the study (see app. II). 

To examine measures established to monitor the study's conduct and to
prevent improper influence, we reviewed the study protocol, memoranda, and
correspondence obtained from the Air Force and the Advisory Committee and
documentation provided by the Food and Drug Administration on veterans'
representation on the Committee. We documented the roles and activities of
key organizations involved in making scientific recommendations to Air
Force scientists and compared them with requirements outlined in the study
protocol. We also reviewed requirements on the selection and composition
of the Ranch Hand Advisory Committee and reporting requirements as stated
by law. 

To assess the impact of the Ranch Hand study on National Academy of
Sciences reports and Department of Veterans Affairs determinations of
diseases for which veterans are eligible to receive compensation benefits,
we reviewed laws outlining the process to determine for which diseases
Vietnam veterans are eligible for compensation. We then obtained and
reviewed reports by the Academy and the Department of Veterans Affairs. In
addition, we interviewed officials from these organizations to determine
how research results lead to compensation decisions. We also reviewed
those portions of the Federal Register that announce and codify
regulations governing compensation of Vietnam veterans.

In conducting our study, we contacted current and former members,
affiliates, or representatives from the following organizations: Advisory
Committee, Executive Secretary at National Center for Toxicological
Research, Food and Drug Administration; Air Force Research Laboratory,
Brooks Air Force Base, San Antonio, Texas; Air Force Surgeon General,
Bolling Air Force Base, Washington, D.C.; American Legion, Washington,
D.C.; Department of Veterans Affairs, Washington, D.C.; National Academy
of Sciences, Washington, D.C.; Veterans of Foreign Wars, Washington, D.C.;
and Vietnam Veterans of America. In addition, we spoke with several
scientists involved in research on the health effects associated with
exposure to herbicides and dioxin. 

We performed our work from January through October 1999 in accordance with
generally accepted government auditing standards.

STATISTICAL POWER OF THE RANCH HAND STUDY
=========================================

This appendix describes in more detail the limited statistical power of
the study caused by the small size of the Ranch Hand population. There
were a total of about 1,200 Ranch Hands in 1982, and there are currently
fewer than 1,150 still alive. An estimate of the ideal statistical power
of the Ranch Hand study from 1982 to the last physical examination, to be
performed in 2002, is summarized for several cancers in tables 1-3. The
last column of each table shows the expected number of cases among Ranch
Hands necessary for the study to be likely to detect an increased risk of
cancer with 95-percent significance. Statistical power increases with time
because cancers, like most health problems, are more common among older
people. This calculation is "ideal" because it assumes that all Ranch
Hands and comparisons participate and that none of them die. It also does
not account for a possible "healthy-soldier effect," in which military
populations are healthier than nonmilitary ones because entrance into
military service requires passing a medical examination that screens out
applicants with chronic illnesses and because remaining in the service
requires maintaining good enough health to perform relatively rigorous
duties. Because these numbers represent what the study could detect under
the most ideal conditions, the actual power of the study is probably less
than indicated here. 

As shown in these tables, the study cannot detect a doubling of cancer
incidence (relative risk of at least 2) until at least 10 years into the
study for many types of site-specific cancers. For all sites combined, the
study can detect a doubling in risk. 

Table****Helvetica:x11****1:    Minimum Detectable Effect in 1982

-------------------------------------------------------------------------
| Cancer site   : Cumulativ :  Number of :    Minimum :          Cases  |
|               :         e :      cases :   relative :       expected  |
|               :  expected :   expected :      riska :    among Ranch  |
|               : percentag :      among :  likely to :          Hands  |
|               :    e with : comparison : be detected:  corresponding  |
|               :    cancer :      group :            :     to minimum  |
|               :           :            :            :  relative risk  |
|               :           :            :            :   likely to be  |
|               :           :            :            :       detected  |
|-----------------------------------------------------------------------|
| Liver         :     0.012 :      0.14  :         42 :           6.0   |
|-----------------------------------------------------------------------|
| Prostate      :     0.088 :       1.0  :        7.5 :           7.9   |
|-----------------------------------------------------------------------|
| Non-Hodgkins  :      0.10 :       1.2  :        6.7 :           8.3   |
| Lymphoma      :           :            :            :                 |
|-----------------------------------------------------------------------|
| Melanoma      :      0.17 :       2.0  :        4.9 :           9.7   |
|-----------------------------------------------------------------------|
| All           :       2.2 :        26  :        1.7 :            43   |
| anatomical    :           :            :            :                 |
| sites         :           :            :            :                 |
-------------------------------------------------------------------------

Table****Helvetica:x11****2:    Minimum Detectable Effect in 1992

-------------------------------------------------------------------------
| Cancer site    : Cumulativ :  Number of :   Minimum :          Cases  |
|                :         e :      cases :  relative :       expected  |
|                :  expected :   expected :     riska :    among Ranch  |
|                : percentag :      among :    likely :          Hands  |
|                :    e with : comparison :     to be :  corresponding  |
|                :    cancer :      group :  detected :     to minimum  |
|                :           :            :           :  relative risk  |
|                :           :            :           :   likely to be  |
|                :           :            :           :       detected  |
|-----------------------------------------------------------------------|
| Liver          :     0.053 :      0.63  :        11 :           7.1   |
|-----------------------------------------------------------------------|
| Prostate       :       1.1 :        13  :       2.0 :            27   |
|-----------------------------------------------------------------------|
| Non-Hodgkins   :      0.36 :       4.3  :       3.1 :            14   |
| Lymphoma       :           :            :           :                 |
|-----------------------------------------------------------------------|
| Melanoma       :      0.49 :       5.9  :       2.7 :            16   |
|-----------------------------------------------------------------------|
| All            :       7.8 :        94  :       1.3 :           122   |
| anatomical     :           :            :           :                 |
| sites          :           :            :           :                 |
-------------------------------------------------------------------------

Table****Helvetica:x11****3:    Minimum Detectable Effect in 2002

--------------------------------------------------------------------------
| Cancer site    : Cumulativ :  Number of :    Minimum :          Cases  |
|                :         e :      cases :   relative :       expected  |
|                :  expected :   expected :      riska :    among Ranch  |
|                : percentag :      among :  likely to :          Hands  |
|                :    e with : comparison : be detected:  corresponding  |
|                :    cancer :      group :            :     to minimum  |
|                :           :            :            :  relative risk  |
|                :           :            :            :   likely to be  |
|                :           :            :            :       detected  |
|------------------------------------------------------------------------|
| Liver          :      0.18 :       2.1  :        4.6 :           9.9   |
|------------------------------------------------------------------------|
| Prostate       :       5.5 :        65  :        1.4 :            90   |
|------------------------------------------------------------------------|
| Non-Hodgkins   :      0.86 :        10  :        2.2 :            22   |
| Lymphoma       :           :            :            :                 |
|------------------------------------------------------------------------|
| Melanoma       :       1.2 :        15  :        1.9 :            28   |
|------------------------------------------------------------------------|
| All            :        22 :       258  :        1.2 :           299   |
| anatomical     :           :            :            :                 |
| sites          :           :            :            :                 |
--------------------------------------------------------------------------

a The minimum relative risk is the smallest relative risk that must exist
in order for the likelihood that the study can detect the effect with 95-
percent significance to be greater than 50 percent. The last column is the
corresponding minimum number of cases expected among Ranch Hands in order
for the study to detect an adverse effect. Numbers are rounded to two
digits unless otherwise shown.

Note: The Ranch Hands used in this analysis were the 1,197 who were
identified and alive as of December 31, 1982. An identical number of
comparisons were assumed. They were stratified by birth in 5-year bins,
with blacks and non-blacks considered separately. The age-specific cancer
incidence rates for 1973-96 (from the Surveillance, Epidemiological, and
End Results database) were used. Years before 1973 used the 1973 rates and
years after 1996 used the 1996 rates. The cumulative incidence was
calculated assuming all Ranch Hands left Vietnam in 1967. 

Sources: GAO analysis of National Cancer Institute and Air Force data.

DUTIES OF THE RANCH HAND ADVISORY COMMITTEE
===========================================

This appendix discusses the duties of the Advisory Committee in overseeing
the Ranch Hand study. In general, the Committee conducts scientific
reviews of the study and provides the Air Force with comments and advice.
However, this advice, which may be in the form of either formal or
informal recommendations, is nonbinding.

The Committee's primary purpose, as described in the study protocol, is to
review and assess the study's conduct. The Committee performs this role
through both site visits and by reviewing study reports before their
release. Site visits include visits to Brooks Air Force Base, where the
Air Force scientists are located, and to locations where physical
examinations are performed. The Committee also reviews all official Air
Force reports on the study prior to release, including executive
summaries. The Committee does not review peer-reviewed journal articles
but has consistently encouraged the Air Force to publish its results in
these journals. The Committee convenes meetings when the Air Force has
drafted reports to be reviewed and has not met on a regular basis.

According to the study protocol, the Committee should also review
suggested changes in the data collected and methods of analysis used by
the study. To this end, the Committee reviews the content of physical
examinations, along with the statements of work for those examinations. In
addition, the Air Force has contracted out the overall management,
logistics, statistical analyses, and writing of Air Force morbidity
reports subsequent to the first report in 1984. The Committee reviews
plans for statistical analyses.

ROLES OF AIR FORCE, NATIONAL ACADEMY OF SCIENCES, AND DEPARTMENT OF
VETERANS AFFAIRS IN EVALUATING HEALTH EFFECTS OF HERBICIDES
===========================================================================

In evaluating the health effects of herbicides, the Air Force, the
National Academy of Sciences, and the Department of Veterans Affairs have
different goals, use different criteria, and produce different types of
products. These differences are illustrated in table 4. 

The goal of the Ranch Hand study is to determine whether health effects
are attributable to herbicide exposure, while the goals of the Academy and
the Department of Veterans Affairs emphasize establishing only an
association. Although statistical methods are the primary tools the Air
Force uses in the study, the "strength of association" of a result is only
one of the many criteria Air Force scientists use in evaluating evidence
of a causal relationship between adverse health and exposure to
herbicides. The National Academy of Sciences and the Department of
Veterans Affairs, on the other hand, both look primarily at evidence of a
"statistical association" between diseases in humans and exposure to
herbicides. The Academy considers evidence of a causal relationship
separately.

The Ranch Hand study and the reviews by the National Academy of Sciences
apply a relatively high standard in evaluating evidence. The Department of
Veterans Affairs, on the other hand, is charged with using a specified
standard of evidence intended to give veterans the "benefit of the doubt."
Specifically, a "positive association" for the Department of Veterans
Affairs is defined by law as when "the credible evidence for the
association is equal to or outweighs the credible evidence against the
association." Although both the Air Force and the National Academy of
Sciences report findings of lesser significance, their emphasis is on
establishing relationships in which they are reasonably confident. Air
Force scientists noted, for instance, that though not all their criteria
need to be satisfied for a finding to be reported, the more indicators
that point to a causal relationship, the stronger the finding. The Academy
defines "sufficient" evidence as findings in which "a positive association
has been observed ****Symbol:xbc**** in studies in which chance, bias, and
confounding could be ruled out with reasonable confidence.(c) A second
Academy category, one of (r)limited/suggestive(c) evidence, includes
situations in which (r)evidence is suggestive of an association
****Symbol:xbc**** but is limited because chance, bias, and confounding
could not be ruled out with confidence.(c) 

Table****Helvetica:x11****4:    Summary of Roles and Criteria in
                                Evaluating the Health Effects of Herbicides

-------------------------------------------------------------------------
|       : Goal(s)    : Evidence   : Standards/cri : Product : Uses of   |
|       :            : evaluated  : teria used    : (s)     : products  |
|-----------------------------------------------------------------------|
| Air   : Existence  : Data       : Plausibility. : Ranch   : National  |
| Force : of health  : collected  :               : Hand    :           |
|       : effects    : by Air     : Internal and  : study   : Academy   |
| Ranch : in         : Force on   : external      : officia : of        |
|       : Vietnam    : Ranch      : consistency.  : l       : Sciences  |
| Hand  : veterans   : Hands and  :               : reports.:           |
| study : attributab : comparison : Strength of   :         : reviews.  |
|       : le to      :  group.    : association.  : Peer-   :           |
|       : exposure   :            :               : reviewe :           |
|       : to         :            : Statistical   : d       : Departme  |
|       : herbicides.:            : power.        : journal : nt of     |
|       :            :            :               :         : Veterans  |
|       :            :            : Biological    : article :           |
|       :            :            : mechanism.    : s.      : Affairs   |
|       :            :            :               :         : compensa  |
|       :            :            : Validity or   :         : tion      |
|       :            :            : bias.         :         : determin  |
|       :            :            :               :         : ations.   |
|       :            :            : Statistical   :         :           |
|       :            :            : significance. :         : Scientif  |
|       :            :            :               :         : ic base   |
|       :            :            : Statistical   :         : of        |
|       :            :            : assumptions.  :         : knowledg  |
|       :            :            :               :         : e.        |
|-----------------------------------------------------------------------|
| Natio : Existence  : Scientific : aExtent to    : Biennia : Departme  |
| nal   : of         :            : which         : l       : nt of     |
| Acade : statistica : literature : scientific    : reports : Veterans  |
| my    : l          :  on        : data permit   : :       :           |
| of    : associatio : effects    : determination : Veteran : Affairs   |
| Scien : n between  : of         : s.            : s and   : compensa  |
| ces   : disease    : herbicides :               : Agent   : tion      |
|       : in         :  in humans.: aStrength of  : Orange. : determin  |
|       : Vietnam    :            : evidence.     :         : ations.   |
|       : veterans   :            :               :         :           |
|       : and        :            : aAppropriaten :         : Recommen  |
|       : herbicides.:            : ess of        :         : dations   |
|       :            :            : methods.      :         : for       |
|       :            :            :               :         : future    |
|       :            :            :               :         : studies.  |
|-----------------------------------------------------------------------|
|       : Existence  : Scientific : aExtent to    :         :           |
|       : of an      :            : which         :         :           |
|       : increased  : literature : scientific    :         :           |
|       : risk       :  on        : data permit   :         :           |
|       : among      : Vietnam    : determination :         :           |
|       : Vietnam    : veterans   : s.            :         :           |
|       : veterans.  : and other  :               :         :           |

|       :            : human      :               :         :           |
|       :            : population :               :         :           |
|       :            : s.         :               :         :           |
|-----------------------------------------------------------------------|
|       : Existence  : Scientific : aExtent to    :         :           |
|       : of a       :            : which         :         :           |
|       : plausible  : literature : scientific    :         :           |
|       : biological :  on        : data permit   :         :           |
|       :            : effects    : determination :         :           |
|       : mechanism  : of         : s.            :         :           |
|       : or other   : herbicides :               :         :           |
|       : causal     :  in        : Validity of   :         :           |
|       : relationsh : humans     : animal models.:         :           |
|       : ip.        : and        :               :         :           |
|       :            : animals.   : Biochemical   :         :           |
|       :            :            : origin of     :         :           |
|       :            :            : dose          :         :           |
|       :            :            : response.     :         :           |
|-----------------------------------------------------------------------|
| Depar : Existence  : National   : aWhether      : Departm : Compensa  |
| tment : of a       : Academy    : evidence for  : ent     : tion      |
|  of   : positive   : of         : an            : Task    : determin  |
| Veter : associatio : Sciences   : association   : Force   : ations.   |
| ans   : n between  : review of  : equals or     : Reports.:           |
| Affai : disease    : scientific : outweighs     :         : Recommen  |
| rs    : in humans  :            : evidence      : Regulat : dations   |
|       : and        : literature : against an    : ions    : for       |
|       : herbicide  : .          : association,  : of      : future    |
|       : exposure.  :            : considering   : disease : studies.  |
|       :            : Other      : (a) academy   : s with  :           |
|       :            : available  : reviews,      : presump :           |
|       :            : scientific : (b)           : tion    :           |
|       :            :  and       : statistical   : of      :           |
|       :            : medical    : significance, : service :           |
|       :            : informatio : (c)           :         :           |
|       :            : n and      : capability    : connect :           |
|       :            : analyses.  : of            : ion.    :           |
|       :            :            : replication,  :         :           |
|       :            :            : and           :         :           |
|       :            :            : (d)           :         :           |
|       :            :            : capability    :         :           |
|       :            :            : to withstand  :         :           |
|       :            :            : peer review.  :         :           |
-------------------------------------------------------------------------

aCriterion required by law.

Sources: Air Force, Public Law, National Academy of Sciences, Department
of Veterans Affairs, and Federal Register.

COMMENTS FROM THE DEPARTMENT OF DEFENSE
=======================================

*****************

*****************

*****************

*****************

*****************

*****************

COMMENTS FROM THE FOOD AND DRUG ADMINISTRATION
==============================================

*****************

*****************

*****************

*****************
p

Table 1:  Minimum Detectable Effect in 1982     29

Table 2:  Minimum Detectable Effect in 1992     29

Table 3:  Minimum Detectable Effect in 2002     30

Table 4:  Summary of Roles and Criteria in Evaluating
the Health Effects of Herbicides                33

Figure 1:  Ranch Hand Study Milestones and Published
Articles and Reports                            10

Figure 2:  Communication Flow From the Advisory
Committee to Air Force Scientists Conducting the
Ranch Hand Study                                18

*** End of document. ***