Pacific Groundfish: Continued Efforts Needed to Improve 	 
Reliability of Stock Assessments (04-JUN-04, GAO-04-606).	 
                                                                 
Because of concerns raised about the accuracy of National Marine 
Fisheries Service (NMFS) stock assessments, GAO reviewed the	 
assessments for five species of Pacific groundfish: Pacific hake 
and four types of rockfish--bocaccio, canary, darkblotched, and  
yelloweye. Specifically, for these five species GAO (1) assessed 
the reliability of NMFS' stock assessments, (2) identified which 
relevant recommendations from NMFS' stock assessment improvement 
plan have been implemented and which have not, and (3) identified
the costs associated with planned and ongoing improvements to	 
groundfish stock assessments.					 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-04-606 					        
    ACCNO:   A10344						        
  TITLE:     Pacific Groundfish: Continued Efforts Needed to Improve  
Reliability of Stock Assessments				 
     DATE:   06/04/2004 
  SUBJECT:   Data collection					 
	     Data integrity					 
	     Evaluation methods 				 
	     Fishes						 
	     Population statistics				 
	     Standards and standardization			 
	     Statistical methods				 
	     Wildlife conservation				 
	     Cost analysis					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-04-606

United States General Accounting Office

                     GAO Report to Congressional Requesters

June 2004

PACIFIC GROUNDFISH

      Continued Efforts Needed to Improve Reliability of Stock Assessments

                                       a

GAO-04-606

Highlights of GAO-04-606, a report to congressional requesters

Because of concerns raised about the accuracy of National Marine Fisheries
Service (NMFS) stock assessments, GAO reviewed the assessments for five
species of Pacific groundfish: Pacific hake and four types of rockfish-
bocaccio, canary, darkblotched, and yelloweye. Specifically, for these
five species GAO (1) assessed the reliability of NMFS' stock assessments,
(2) identified which relevant recommendations from NMFS' stock assessment
improvement plan have been implemented and which have not, and (3)
identified the costs associated with planned and ongoing improvements to
groundfish stock assessments.

GAO recommends that the Secretary of Commerce require the Director of NMFS
to take actions to improve the reliability of stock assessments, such as
continuing efforts to improve the quality and types of data used in
groundfish assessments, establishing a standard approach that requires
that data used in stock assessments be evaluated for reliability, and
requiring stock assessment reports clearly present the uncertainties in
the assessments.

NOAA generally agreed with the report's accuracy and agreed with the
report's recommendations, but expressed concern that the report's
conclusion could be misconstrued to infer that the assessments are
unreliable for use in managing the west coast groundfish fishery.

www.gao.gov/cgi-bin/getrpt?GAO-04-606.

To view the full product, including the scope and methodology, click on
the link above. For more information, contact Anu K. Mittal at (202)
512-3841 or [email protected].

June 2004

PACIFIC GROUNDFISH

Continued Efforts Needed to Improve Reliability of Stock Assessments

The reliability of the NMFS assessments is questionable for the five
species GAO reviewed, although the assessments were based on the best
information available at the time they were conducted. According to NMFS
officials and a National Research Council report, to obtain reliable
results each stock assessment should include at least one NMFS data source
of sufficient scope and accuracy because such data are derived from
unbiased, statistical designs. However, in the yelloweye assessment, no
NMFS data were used, and in the darkblotched, canary, and bocaccio
assessments, the NMFS data were limited because the NMFS' surveys were
conducted in trawlable waters only. A 2003 NMFS report concluded that
darkblotched groundfish are less abundant and bocaccio and canary are more
abundant in untrawlable waters. Also for all five species, NMFS lacks a
standard approach for ensuring the reliability of non-NMFS data used in
stock assessments. Some assessors reviewed the quality of non-NMFS data;
others did not. The assessors who reviewed the quality of the non-NMFS
data found errors that made some of the data unusable or that could have
impaired the reliability of certain stock assessments. Finally, for four
species, the stock assessment reports were questionable because they did
not present the uncertainty associated with the population estimates. For
example, the canary stock assessment review panel recommended that
standard estimates of uncertainty be included in the assessment report
because without them it is difficult to determine their reliability.

NMFS has taken steps to implement some of the recommendations contained in
the NMFS stock assessment improvement plan, but much remains to be done.
NMFS has concentrated its efforts mostly on improving data quantity. For
example, NMFS increased the frequency of groundfish stock assessments and
extended the geographic ranges of the shelf and slope surveys to cover
over 300 more miles along the southern California coast. However, because
of staffing and funding limitations, NMFS has not yet implemented many of
the recommendations aimed at obtaining more types of data and improving
data quality. For example, NMFS has not collected enough ecosystem data,
and the frequency and range of recruitment surveys (estimated production
of new members of a fish population) are limited. Finally, because of
other program priorities, NMFS has not implemented the recommendation to
create a comprehensive plan that combines the improvement plan and its
complementary plans.

NMFS records indicate at least $8.9 million is needed to complete ongoing
and planned stock assessment improvements-$2.6 million that NMFS'
Northwest Fisheries Science Center requested but did not receive in fiscal
years 2001 to 2003, and $6.3 million requested for fiscal years 2004 and
2005. It will cost about (1) $7.7 million to improve the types of data
used, such as more untrawlable water and recruitment surveys and (2) $1.2
million to improve the quality of data used in stock assessments, such as
enhanced calibration of vessel equipment and standardized trawl survey
procedures. The actual cost of the remaining improvements may be even
higher than the $8.9 million estimated because the estimates primarily
reflect the amount of money that agency officials believed could be
realistically obtained, rather than what the improvements might cost.

Contents

  Letter

Results in Brief
Background
Reliability of NMFS' Pacific Groundfish Stock Assessments Is

Questionable Some Recommended Stock Improvements Have Been Implemented,
but Much Remains to be Done Remaining Improvements Estimated to Cost at
Least $8.9

Million Conclusions Recommendations for Executive Action Agency Comments
and Our Evaluation

1 3 5

13

18

23 26 27 28

Appendixes                                                            
                Appendix I:      Objectives, Scope, and Methodology        31 
               Appendix II:   Comments From the Department of Commerce     33 
                                            GAO Comments                   42 
              Appendix III:    GAO Contact and Staff Acknowledgments       47 
                                            GAO Contact                    47 
                                       Staff Acknowledgments               47 
                            Table 1: Stock Assessment Reliability Issues 
     Tables                               for Five Pacific               
                                         Groundfish Species                14 
                            Table 2: Pacific Groundfish Stock Assessment 
                                         Improvement Funds               
                                 Requested and Received by Data Type and 
                                                         Quality, Fiscal 
                                          Years 2001-2005                  24 
    Figures                    Figure 1: Photograph of a Pacific Hake         
                            Figure 2: Photograph of a Bocaccio Rockfish  9 10
                             Figure 3: Photograph of a Canary Rockfish     11 
                               Figure 4: Photograph of a Darkblotched      12 
                                              Rockfish                   
                            Figure 5: Photograph of a Yelloweye Rockfish   13 

Contents

Abbreviations

NMFS National Marine Fisheries Service
NOAA National Oceanic and Atmospheric Administration

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

A

United States General Accounting Office Washington, D.C. 20548

June 4, 2004

The Honorable Gordon H. Smith United States Senate

The Honorable Ron Wyden United States Senate

Pacific groundfish, 82 species of bottom-dwelling fish, such as several
species of rockfish and sole, have contributed to the economies of fishing
communities in California, Oregon, and Washington, generating over $220
million in income in 2001. However, some populations of Pacific groundfish
have declined sharply because of natural conditions, such as climate, as
well as man-made conditions, such as overfishing. In an effort to achieve
sustainable populations, the National Marine Fisheries Service (NMFS)
significantly reduced the allowable harvest for nine species of Pacific
groundfish. NMFS made this decision based on its stock assessments-which
include estimates of species' total biomass (weight of the total
population)-and recommendations from the Pacific Fishery Management
Council, one of eight councils responsible for managing fisheries in
federal waters.

NMFS generally uses internal scientists or contracts with outside experts
to conduct stock assessments. These assessors use multiple types of data,
including data from external sources, such as state-collected data on the
poundage and species of fish brought ashore by commercial and recreational
fishermen. NMFS' Northwest Fisheries Science Center (Northwest Center),
which coordinates Pacific groundfish stock assessments, also collects its
own data through fish population surveys, particularly two key surveys:
(1) the shelf and slope bottom trawl survey, in which a trawl net is
dragged along the ocean floor, and the resulting catch is sorted, counted,
and biologically examined (e.g., age and sex) and (2) the acoustic survey,
in which sound waves are used to measure density of schools of groundfish
in mid-level waters; bottom trawls are not feasible in these waters. Fish
survey and other data are entered into computerized mathematical models
(stock assessment models), which estimate the total biomass for the fish
species being assessed. However, concerns have been raised about the
accuracy of stock assessments and therefore whether they can be relied
upon to specify the amount of fish that can be harvested to ensure a
sustainable population.

In 1999, NMFS commissioned a national task force to review the stock
assessment process across the agency. Task force members were from NMFS
headquarters and five of the six NMFS science centers, including the
Northwest Center. In the October 2001 Marine Fisheries Stock Assessment
Improvement Plan,1 the task force identified a lack of adequate data as
the greatest impediment to producing reliable stock assessments and
recommended actions to improve the quantity, quality, and types of data
used in the models that estimate biomass. The task force also identified a
need to better communicate and quantify uncertainties in these models.

In this context, you requested that we review NMFS' stock assessments for
five species of Pacific groundfish: Pacific hake as well as four types of
rockfish-bocaccio, canary, darkblotched, and yelloweye. Specifically, for
these five species you asked us to (1) assess the reliability of NMFS'
stock assessments, (2) identify which relevant recommendations from the
stock assessment improvement plan have been implemented and which have
not, and (3) identify the estimated costs associated with planned and
ongoing improvements to groundfish stock assessments.

To address these issues, we reviewed key laws and stock assessment studies
(2002 and 2004 studies for Pacific hake, 2002 and 2003 studies for
bocaccio, 2002 study for canary, 2000 and 2003 studies for darkblotched,
and 2001 and 2002 studies for yelloweye), along with the types of fish
population surveys used and the controls over stock assessment data. We
spoke with officials from NMFS, the Oregon Department of Fish and
Wildlife, the Washington Department of Fish and Wildlife, the California
Department of Fish and Game, the Pacific Fishery Management Council, the
Pacific States Marine Fisheries Commission, environmental groups, and
industry associations, as well as fishermen and academics. We also
reviewed NMFS' budget documents to develop cost estimates. We did not
simulate NMFS' stock assessment models or evaluate the mathematical and
statistical methodologies used in the models. We conducted our review from
May 2003 through April 2004 in accordance with generally accepted
government auditing standards. See appendix I for additional details on
our scope and methodology.

1Marine Fisheries Stock Assessment Improvement Plan, U.S. Department of
Commerce, NOAA Tech. Memo. NMFS-F/SPO-56, 69, 2001.

Results in Brief 	For a variety of reasons, the reliability of the NMFS
assessments is questionable for the five species we reviewed, although the
assessments were based on the best information available at the time they
were conducted. First, according to NMFS officials and a 1998 National
Research Council report, to obtain reliable results, each stock assessment
should include at least one source of NMFS-collected data of sufficient
scope and accuracy because such data are derived from unbiased,
statistical, and scientific designs. However, the four rockfish
assessments did not meet this standard, although the assessment for
Pacific hake did. NMFS did not use any of its own data in the yelloweye
assessment, and the NMFS data used in the bocaccio, canary, and
darkblotched assessments were limited because NMFS conducted its surveys
in trawlable waters only; these fish also inhabit both trawlable and
untrawlable waters. In the absence of data from rocky, untrawlable waters,
the assessors estimated overall biomass using the NMFS data collected from
the trawlable areas. However, in 2003, NMFS found that darkblotched are
less abundant and bocaccio and canary are more abundant in untrawlable
waters. As a result, some rockfish populations may be overstated, while
others may be understated. Second, for all the species, NMFS lacks a
standard approach for ensuring the reliability of non-NMFS data used in
the stock assessments. As a result, some assessors reviewed the quality of
these data while others did not. The assessors who reviewed the quality of
the non-NMFS data found errors that they believed made the data unusable
or impaired the reliability of the stock assessment. Finally, the
reliability of four of the stock assessment reports-bocaccio, canary,
darkblotched, and yelloweye-is questionable because they did not present
the uncertainty associated with the population estimates. The National
Research Council, NMFS, and GAO agree that estimates of uncertainty should
be included in stock assessment reports because without them it is
difficult to determine their reliability.

NMFS has taken steps to implement the NMFS stock assessment improvement
plan recommendations to improve stock assessments, but much remains to be
done to enhance their reliability. While the task force considered the
lack of adequate data-in quantity, quality, and type-as the greatest
impediment to accurate stock assessments, the Northwest Center has
concentrated most of its efforts on improving data quantity. For example,
starting in 2003, the Northwest Center extended the geographic ranges of
the shelf and slope surveys, expanding them to cover over 300 additional
miles along the southern California coast. It also increased the frequency
of the groundfish shelf and slope survey from triennially to annually and
the frequency of the Pacific hake acoustic survey from

triennially to biennially. However, NMFS has not yet fully implemented
many of the recommendations aimed at obtaining more types of data and
improving data quality. For example, although Northwest Center officials
recognize that recruitment surveys (estimated production of new members of
a fish population) should be conducted biannually along the entire coast,
these surveys are currently conducted only annually and only in selected
regions. While Northwest Center officials also recognize that actions,
such as calibrating vessel equipment to ensure comparable survey data, are
needed to improve data quality, these actions have not yet been fully
implemented. Finally, NMFS has not implemented the recommendation to
create a comprehensive plan that combines the improvement plan and related
plans to develop integrated program initiatives. Although NMFS officials
recognize a need for such a plan, other priorities have precluded them
from developing it.

According to NMFS funding and budget documents, NMFS needs at least $8.9
million to complete ongoing and planned improvements to its stock
assessments of Pacific groundfish-$2.6 million that the Northwest Center
had requested but did not receive in fiscal years 2001 through 2003, and
$6.3 million requested for fiscal years 2004 and 2005. Specifically,
according to these records, the Northwest Center has the following funding
needs:

o 	$7.7 million to improve the types of data used, including $2.4 million
for surveys of untrawlable waters, $2.1 million to expand acoustic and
recruitment surveys, and $3.2 million to collect ecosystem data and

o 	$1.2 million to improve the quality of data used in stock assessments,
including $600,000 to enhance the calibration of vessel equipment,
$525,000 to develop and implement methods to more accurately distinguish
among groundfish species, and $75,000 to standardize trawl survey
procedures.

The actual cost of implementing improvements to Pacific groundfish stock
assessments may be even higher than the $8.9 million estimated because the
Northwest Center's budget requests primarily reflect the amount of money
the Center believed it could realistically obtain, rather than what the
improvements would cost, according to NMFS officials.

Background	The Magnuson-Stevens Fisheries Conservation and Management Act
granted responsibility for managing marine resources to the Secretary of
Commerce.2 The Secretary delegated this responsibility to NMFS, which is
part of Commerce's National Oceanic and Atmospheric Administration (NOAA).
The act also established eight regional fishery management councils, each
responsible for making recommendations to the Secretary of Commerce about
managing fisheries in federal waters. The eight fishery management
councils-consisting of fishing industry participants, state and federal
fishery managers, and other interested parties-and their areas of
responsibility include the following:

o 	Caribbean Council, covering waters off the U.S. Virgin Islands and the
Commonwealth of Puerto Rico;

o 	Gulf of Mexico Council, covering waters off Texas, Louisiana,
Mississippi, Alabama, and the west coast of Florida;

o 	Mid-Atlantic Council, covering waters off New York, New Jersey,
Delaware, Maryland, Virginia, and North Carolina;

o 	New England Council, covering waters off Maine, New Hampshire,
Massachusetts, Rhode Island, and Connecticut;

o  North Pacific Council, covering waters off Alaska;

o  Pacific Council, covering waters off California, Oregon, and
Washington;

o 	South Atlantic Council, covering waters off North Carolina, South
Carolina, Georgia, and the east coast of Florida; and

o 	Western Pacific Council, covering waters off Hawaii, American Samoa,
Guam, the Commonwealth of the Northern Mariana Islands, and uninhabited
U.S. territories in the Western Pacific.

In addition to these eight councils, NMFS has six regional science
centers, which are responsible for generating the scientific information
necessary for the conservation, management, and use of each region's
marine

2Pub. L. No. 94-265, 90 Stat. 331 (codified as amended at 16 U.S.C. S:S:
1801-1883).

resources. The six fishery science centers and their areas of
responsibility are as follows:

o 	Alaska Center, covering the coastal oceans off Alaska and parts of the
west coast of the United States;

o 	Northeast Center, covering waters along the Northeast Continental Shelf
from the Gulf of Maine to Cape Hatteras, North Carolina;

o 	Northwest Center, covering the northeast Pacific Ocean, primarily
waters off the coasts of California, Oregon, Washington, and British
Columbia;3

o  Pacific Islands Center, covering the central and western Pacific Ocean;

o 	Southwest Center, primarily covering waters off the coast of California
and areas throughout the Pacific and Antarctic Oceans;4 and

o 	Southeast Center, covering waters along the continental southeastern
United States as well as Puerto Rico and the U.S. Virgin Islands.

The Magnuson-Stevens Act, as amended by the Sustainable Fisheries Act,5
also established national standards for fishery conservation and
management. These standards deal with preventing overfishing, using
scientific information, using fishery resources efficiently, minimizing
bycatch,6 and minimizing administrative costs. The regional councils use
these standards to develop appropriate plans for conserving and managing
fisheries under their jurisdiction, including measures to prevent
overfishing and to rebuild overfished stocks as well as measures to
protect, restore, and promote the long-term health and stability of the
fishery.

3In addition, the Northwest Center and the Southwest Center share some
responsibilities in freshwater rivers and streams in Idaho, Oregon,
Montana, and Washington.

4Ibid.

5Pub. L. No. 104-297, 110 Stat. 3559 (1996).

6Under the Magnuson-Stevens Act, "bycatch" means fish that are harvested
in a fishery, but which are not sold or kept for personal use. Bycatch
includes fish discarded for regulatory or economic reasons.

In 1982, the Pacific Fishery Management Council (Pacific Council) released
its initial Fishery Management Plan for groundfish. The Pacific Council's
goal is to have long-range plans for managing groundfish fisheries that
will promote a stable planning environment for the seafood industry,
including marine recreation interests, and will maintain the health of the
resource and environment. To help achieve these goals, stock assessments
are conducted on groundfish species to estimate fish populations. Since
1995, the Northwest Center has had lead responsibility for conducting
stock assessments on Pacific groundfish. The Northwest Center receives
assistance from other NMFS science centers, such as the Southwest Center,
which conducted the bocaccio assessment.

Stock assessments are the biological evaluation of the status of fish
stocks. Stock assessments provide official estimates in key areas, such as
the size of the stock population, the size of the spawning population, the
amount of fish that have died (fish mortality), and the estimated number
of fish at a particular young age (recruitment). Stock assessments form
the scientific basis used by regional councils to determine biologically
sustainable harvests and guide the monitoring and rebuilding of overfished
and threatened stocks. For example, regional councils use stock
assessments and other indicators of biological productivity to recommend
to NMFS a maximum, or total allowable catch, in a particular
fishery-typically for a year. Stock assessments are a key tool for
managing fisheries. Without stock assessments, fishery managers would have
limited information about the status of fisheries in making decisions
about setting harvest levels and developing plans to rebuild overfished
stocks.

For each species, the assessor reviews previous stock assessments, gathers
available data about the species being assessed, runs the data through
computer-generated models, and estimates the species' total biomass. Stock
assessors use NMFS-collected data, such as stock surveys conducted on NOAA
vessels or contracted commercial fishing vessels, as well as data
collected by non-NMFS sources, such as commercial and recreational catch
data collected by state agencies. The following six key types of data are
collected:

o 	Stock abundance-surveys of how many fish constitute a stock's total
size or weight.

o 	Commercial and recreational fisheries data-the amount and composition
of fish caught from a particular stock, whether caught intentionally by
commercial and recreational fishermen or

unintentionally caught and discarded. Data sources include fishing
logbooks, dockside samples, and onboard observations, among others.

o 	Life history-biological data, such as the age and sex composition of
the stock, age at first maturity, fertility, average lifespan, and natural
mortality.

o 	Ecosystem relationships-data on the relationship between a fish stock
and its physical environment, as well the relationship of a fish stock to
other species.

o 	Recruitment research-data on the abundance of juvenile and larval fish
(fish at their earliest stage), which helps scientists forecast the size
of a particular stock in the future.

o 	Synoptic oceanographic sampling-data on the ocean ecosystem, such as
water temperature or salinity, plankton composition, or ocean currents.

For each stock assessment, a review panel, consisting of NMFS scientists
and outside experts, independently reviews the methodology of the
assessment and works with the assessor to ensure their comments are
adequately addressed.

Through 2003, 24 of the 82 species of Pacific groundfish have had a full
quantitative stock assessment. Relying on these assessments, NMFS has
declared that nine species of Pacific groundfish are overfished, including
the five species of Pacific groundfish we reviewed in this report-Pacific
hake as well as bocaccio rockfish, canary rockfish, darkblotched rockfish,
and yelloweye rockfish. Rockfish are long-lived, late-maturing and
slow-growing species, making them particularly susceptible to overfishing.
More specifically:

Pacific hake, also called Pacific whiting, is generally found off the west
coast of North America. It is one of many species of hake distributed in
the Atlantic and Pacific oceans. Fishing of Pacific hake primarily takes
place off the coasts of northern California, Oregon, Washington, and
British Columbia. Fishermen use mid-water trawls and generally fish over
the ocean bottom at depths of 100 to 500 meters. Pacific hake was declared
overfished in 2002 because the 2002 stock assessment estimated Pacific
hake biomass at 700,000 metric tons. By 2004, the biomass was estimated at
between 2.7 and 4.2 million metric tons, and Pacific hake is no longer
considered overfished. Figure 1 shows a picture of a Pacific hake.

Figure 1: Photograph of a Pacific Hake

        Source: Guy Fleischer, Northwest Fisheries Science Center, NMFS.

Bocaccio rockfish generally inhabit waters off the coast of northern Baja,
Mexico to Alaska. Bocaccio, commonly sold at market as "red snapper," are
commercially fished using trawls, hook-and-line and gillnets. Adult
bocaccio commonly live over rocky areas or open areas of the ocean's floor
to about 320 meters. Bocaccio were formally declared to be "overfished" in
1999. The 2003 stock assessment estimated the bocaccio biomass at 7,133
metric tons. Figure 2 shows a picture of a bocaccio rockfish.

                  Figure 2: Photograph of a Bocaccio Rockfish

Source: Robert Lauth, Alaska Fisheries Science Center, NMFS.

Canary rockfish inhabit the northeastern Pacific Ocean, from northern
Baja, Mexico to the western Gulf of Alaska. Adult canary rockfish are
primarily found along the continental shelf, from 46 to 457 meters deep.
Canary rockfish are harvested commercially using trawl nets and
hook-and-line and are also considered an important species for
recreational fishermen. NMFS declared canary rockfish as overfished in
2000. The 2002 stock assessment estimated the canary rockfish biomass at
6,197 metric tons. Figure 3 shows a picture of a canary rockfish.

                   Figure 3: Photograph of a Canary Rockfish

Source: Robert Lauth, Alaska Fisheries Science Center, NMFS.

Darkblotched rockfish are found in the waters from Santa Catalina Island,
California to the Bering Sea on soft bottom areas at about 29 to 549
meters deep. Commercial fishery concentrations are located off the coasts
of California and Oregon. Darkblotched rockfish are caught primarily by
commercial trawls and contribute to both commercial and recreational
fishing. NMFS determined that darkblotched rockfish was overfished in
2000, when the last full stock assessment was conducted; it was updated in
2003. This update estimated the darkblotched rockfish biomass at 7,266
metric tons in 2001.7 Figure 4 shows a picture of a darkblotched rockfish.

Figure 4: Photograph of a Darkblotched Rockfish

Source: Robert Lauth, Alaska Fisheries Science Center, NMFS.

7The 2003 darkblotched update was not a full stock assessment, but rather
updated the 2000 darkblotched stock assessment by adding some additional
data in order to estimate new biomass figures. The most recent estimate
from this update is for 2001.

Yelloweye rockfish almost exclusively inhabit rocky areas from northern
Baja, Mexico to the Aleutian Islands, Alaska. Yelloweye rockfish, found in
depths ranging from 15 to 550 meters, are caught using both trawl nets and
line gear and are highly prized by both commercial and recreational
fishermen. Stock assessments for yelloweye rockfish were first conducted
in 2001, and NMFS determined that the species was overfished in 2002. The
2002 stock assessment estimated the yelloweye biomass at 2,325 metric tons
in 2001.8 Figure 5 shows a picture of a yelloweye rockfish.

                  Figure 5: Photograph of a Yelloweye Rockfish

The reliability of NMFS' stock assessments is questionable for the Pacific
hake and four rockfish species we reviewed, although they were based on
the best information available at the time the assessments were conducted.
The reliability of the stock assessments we reviewed is questionable
because (1) four of the assessments did not have at least one
NMFS-collected data source of sufficient scope and accuracy; (2) NMFS
lacked a standard process for assessing the reliability of non-NMFS data
used in all five assessments; and (3) for four of the assessments, the
stock

8The most recent estimate of biomass for yelloweye from this stock
assessment is for 2001.

          Source: Robert Lauth, Alaska Fisheries Science Center, NMFS.

  Reliability of NMFS' Pacific Groundfish Stock Assessments Is Questionable

assessment reports did not adequately identify the uncertainty of the
biomass estimates. (See table 1.) To address these limitations, the
Northwest Center plans to increase the scope and accuracy of its collected
data, as additional funds become available; is implementing changes that
will help ensure the reliability of non-NMFS data; and plans to update the
stock assessment model to provide uncertainty ranges for the 2005 stock
assessments.

Table 1: Stock Assessment Reliability Issues for Five Pacific Groundfish
Species

                     Species NMFS-collected data available?

Standard data reliability testing conducted by NMFS?

Uncertainty ranges for biomass estimates provided?

                Pacific hake          Yes          No          Yes 
                  Bocaccio          Limiteda       No           No 
                   Canary           Limiteda       No           No 
                Darkblotched        Limiteda       No           No 
                  Yelloweye            No          No           No 

Source: GAO analysis of information provided by NMFS.

aNMFS-collected data are available for only the trawlable portion of the
species' habitat.

    Absence of NMFS Survey Data Adversely Affects the Assessments' Reliability

Stock assessors use a variety of data, including NMFS data and non-NMFS
data, in developing their assessments.9 Two key pieces of NMFS survey data
are the shelf and slope bottom trawl survey and the acoustic survey. Other
NMFS data that assessors sometimes use include larval surveys (data for
fish in their earliest stage) and recruitment surveys. The non-NMFS data
assessors use include commercial catch data and recreational catch data.

A 2002 National Research Council report found that the inclusion of NMFS
survey data was the best option for a reliable estimate of abundance
because such surveys use an unbiased statistical design, control sampling
locations, and provide for quality assurance.10 According to Northwest

9NMFS generally refers to its data as fishery-independent data and to
non-NMFS data as fishery-dependent data.

10Science and Its Role in the National Marine Fisheries Service, National
Research Council, National Academy Press, July 2002.

Center officials, to obtain reliable results, each stock assessment should
include at least one source of NMFS-collected data of sufficient scope and
accuracy because such surveys are unbiased and scientifically designed.

Northwest Center officials raised concerns about basing assessments solely
on non-NMFS data such as commercial and recreational catch data. Catch
data do not provide the species' relative or absolute biomass, according
to NMFS officials. For example, catch data alone is insufficient because

o 	fishermen are not randomly sampling the ocean but are fishing areas
that they are allowed to fish and they believe have the most fish;

o 	fishing restrictions, such as a total allowable catch, can limit the
amount of fish being caught; and

o 	catch data have often been inaccurate for a variety of reasons, such as
imprecise accounting for dead fish tossed back into the ocean.

Although the assessors used several different data sources, four of the
five assessments did not use NMFS survey data or the NMFS data used
covered only a portion of the species' habitats. In the yelloweye
assessment, no NMFS survey data were available because yelloweye live
almost exclusively in the rocky habitat that NMFS trawl surveys cannot
cover. As a result, the yelloweye assessment was based solely on non-NMFS
data. Similarly, the NMFS survey data used in the bocaccio, canary, and
darkblotched assessments were limited in scope because the surveys were
conducted only in trawlable waters. Bocaccio, canary, and darkblotched
live in both the trawlable and untrawlable habitats. Using data from trawl
surveys conducted from 1977 through 1998, NMFS reported in 2003 that 77
percent of the survey area was trawlable and 23 percent was untrawlable.11
Lacking data on species in the 23 percent of the ocean floor that is
untrawlable, the assessors estimated the overall biomass using the NMFS
data collected from the trawlable areas. However, the abundance in the
trawlable area is not representative of the abundance in the untrawlable
area. The 2003 NMFS report also found that darkblotched groundfish are
less abundant in untrawlable waters, while canary and bocaccio species are
more abundant in untrawlable waters. As a result, some rockfish

11Zimmermann, Calculations of Untrawlable Areas within the Boundaries of a
Bottom Trawl Survey, NRC Research Press, July 2003.

populations may be understated while others may be overstated. According
to stock assessors, relying solely on survey data from trawlable waters
increases the uncertainty of stock assessments.

In contrast, the fifth groundfish species, Pacific hake, lives primarily
in mid-water habitat; and so the concerns about the lack of NMFS data in
rocky, untrawlable habitats are not applicable.

    NMFS Lacks a Standard Process for Ensuring the Reliability of Non-NMFS Data
    Used in the Stock Assessments

NMFS does not have a standard process for evaluating whether the non-NMFS
data used in its stock assessments are reliable. We believe that certain
internal control activities, such as a standard process for ensuring data
reliability, can help ensure that information used to make management
decisions is complete and accurate.12 Lacking a standard process, some
assessors reviewed the quality of the raw non-NMFS data, while others did
not. Assessors who reviewed for data quality found mistakes that they
believed made some of the data unusable or that could have impaired the
accuracy of the stock assessments. For example, the assessor for the 2002
yelloweye stock assessment found numerous errors in the recreational catch
data, such as attributing the catch from an entire fishing vessel to a
single fisherman, and thus did not use the data because doing so could
have resulted in overestimating the biomass. According to another stock
assessor, commercial catch data frequently have inconsistencies.
Specifically, the assessor said California, Oregon, and Washington require
fishermen to enter catch and location information into logbooks, but
logbooks are often incomplete and inaccurate. While the stock assessment
review panels evaluated the assessments, the panels did not evaluate the
quality of the raw data used in the assessments.

According to a Northwest Center official, several assessors have raised
concerns about data quality and accessibility in feedback meetings. In
response to these concerns, the Northwest Center official has recently
begun assigning data stewards to each data set used in its assessments.
Data stewards are responsible for helping assessors compile relevant data
and for conducting quality assurance and quality control checks on the
data. The Northwest Center plans to conduct a data quality workshop in
July 2004 to formally establish the roles and responsibilities of the data
stewards, with the intent of standardizing the data evaluation process.

12U.S. General Accounting Office, Standards for Internal Control in the
Federal Government, GAO/AIMD-00-21.3.1, (Washington, D.C.: November 1999).

    Stock Assessments for Four Species Did Not Provide Uncertainty Estimates

In 1998, the National Research Council recommended that NMFS include
realistic measures of uncertainty in its stock assessments.13 NMFS' 2001
stock assessment improvement plan also recognized the need to better
quantify and communicate the uncertainty in assessments. In a review of
the 2002 canary assessment, the stock assessment review panel recommended
that standard estimates of uncertainty be included in future assessments
because it is difficult to determine the reliability of the stock
assessment without them.14 Similarly, we believe that estimates based on
samples should have a range of uncertainty to show the amount of
variability in the estimates.15 However, the bocaccio, canary,
darkblotched, and yelloweye assessments did not present a measure of
uncertainty associated with the biomass estimates.

Without uncertainty ranges, it is difficult for regional councils and NMFS
to know how much confidence they can have in relying on the estimates for
determining stock abundance and hence for setting allowable harvests of
the fish. For example, lacking uncertainty ranges, the 2002 bocaccio stock
assessment estimated a bocaccio biomass of 2,914 metric tons in 2002. The
2003 assessment of bocaccio biomass, however, estimated 6,506 metric tons
in 2002-more than doubling the previous estimate because of additional and
updated data. With such wide variations, it is important to provide
uncertainty ranges, otherwise management may make inappropriate decisions.

While assessors told us that their stock assessments included some
information about differences in estimated biomass when using different
data sources (sensitivity analyses), the mathematical model that NMFS uses
to estimate biomass (Stock Synthesis model) does not calculate uncertainty
ranges. NMFS officials told us that NMFS is updating the model so that it
can compute uncertainty ranges; NMFS expects to use the updated model for
all 2005 stock assessments. The Pacific hake assessor used a mathematical
model (AD Model Builder) that could compute

13Improving Fish Stock Assessments, National Research Council, National
Academy Press, 1998.

14Canary Rockfish STAR Panel Meeting Report, Northwest Fisheries Science
Center, May 7, 2002.

15U.S. General Accounting Office, Quantitative Data Analysis: An
Introduction, GAO/PEMD-10.1.11, (Washington, D.C.: May 1992).

uncertainty ranges and included these ranges in the Pacific hake
assessment report.

  Some Recommended Stock Improvements Have Been Implemented, but Much Remains to
  be Done

NMFS has taken some steps recommended in the Marine Fisheries Stock
Assessment Improvement Plan to improve the quantity, quality, and type of
data used in Pacific groundfish stock assessments, but much remains to be
done to make the assessments more reliable. The Northwest Center has
concentrated most of its efforts on implementing recommendations aimed at
obtaining more data. Recommendations aimed at increasing the types of data
and improving their quality have not yet been fully implemented for a
variety of reasons, such as staffing and funding limitations. In addition,
other program priorities have precluded NMFS from implementing the
recommendation to create a comprehensive plan that incorporates the
improvement plan and related plans so that it can develop integrated
program initiatives to improve stock assessments.

    The Northwest Center Has Taken Some Steps Recommended for Improving Stock
    Assessments

The October 2001 stock assessment improvement plan identified three
scenarios (tiers) to consider when analyzing the resources needed to
improve stock assessments. The three tiers of assessment improvements are
as follows:

o 	Tier 1-improve stock assessments using existing data without initiating
new data collection programs.

o 	Tier 2-conduct baseline monitoring of species, which in most cases
requires sampling the species at least every 1 to 3 years, and preferably
at least every 1 to 2 years.

o 	Tier 3-implement "next generation" stock assessments by explicitly
incorporating ecosystem considerations, such as multispecies interactions
and environmental effects in assessments.

The improvement plan also made a number of recommendations to improve
stock assessments. The recommendations fall into the following four
categories:

o 	Data collection-pursue new initiatives to expand data collection
efforts that at a minimum bring stock assessment science to Tier 2. In
addition, continue to develop partnerships and cooperative research

programs with other entities, such as state agencies, commercial and
recreational fishing organizations, and individuals to improve the
quantity, quality, and type of data collected.

o 	Communication-educate constituents about NMFS' strategies for improving
stock assessments.

o 	Training-implement comprehensive training and staff development
programs for NMFS' analytical and quantitative staff, and augment existing
programs that support graduate students interested in stock assessment
science.

o 	Planning-develop integrated program initiatives by preparing a
comprehensive plan that combines the improvement plan with its
complementary plans.

Improvement in the quantity of data collected for use in stocks
assessments is a key component to achieving Tier 2 status. The improvement
plan states that as the quantity of the data increases, the assessments
become more reliable because the data cover a longer period of time,
producing better population trend information. Northwest Center officials
said that the quality of the data improves with more frequent surveys and
more randomly selected survey locations that, over the long term, provide
a better understanding of the variability inherent in the population
distribution and abundance. A better sense of trends and variability allow
for improved short-term predictions of the status of the species.

The Northwest Center has concentrated most of its efforts on implementing
improvements in data quantity, such as more frequent acoustic, and shelf
and slope bottom trawl surveys. The following illustrate some of the
actions the Northwest Center took in 2003 to improve data quantity:16

o 	Increased the frequency of the Pacific hake acoustic survey from
triennially to biennially. Beginning in 2003, the survey was restructured
into a single, integrated survey with Northwest Center and Canadian
officials jointly planning all survey elements. Officials from the

16The Southwest Center also took some independent and joint actions with
the Northwest Center to improve data quantity.

Northwest Center and Canada now jointly conduct all of the acoustic
surveys.

o 	Increased the frequency of the groundfish shelf and slope bottom trawl
survey from triennially to annually, leveraging available resources by
cooperatively working with the fishing industry. Specifically, contracting
with private commercial fishing vessels to conduct the surveys. According
to Northwest Center officials, working collaboratively with the fishing
industry has afforded fishermen the opportunity to become stakeholders in
the data collection process.

o 	Extended the geographic range of the groundfish shelf and slope bottom
trawl survey. The surveys are now coastwide from Cape Flattery, Washington
to the Mexican border, adding over 300 more miles along the southern
California coast. Previous surveys ended at Morro Bay, California.

Efforts continue to communicate the strategies needed to improve stock
assessments and to augment existing programs aimed at developing future
stock assessment scientists. For example, through the Pacific Fishery
Management Council, Northwest Center staff meet with their constituents,
such as representatives from state agencies and the fishing industry, to
discuss groundfish management issues. In addition, the Northwest Center
organized a series of public meetings to discuss new initiatives that
affect the stock assessment program. For example, the Northwest Center
held public meetings in several communities along the Pacific coast to
discuss implementation of the Observer Program-a program designed to
collect information about discarded fish for the non-hake west coast
groundfish fleets.17

Finally, the Northwest Center now participates in NMFS' National Sea Grant
program to augment a Northwest Center-supported graduate study program at
the University of Washington that trains stock assessment scientists. The
Sea Grant program provides fellowships for students interested in marine
research, such as stock assessment methodology and marine resource
economics. The Northwest Center plans to employ two Sea Grant students
during the summer of 2004.

17Pacific hake has a separate observer program. Discards are the amount of
fish unintentionally caught and not retained on the fishing vessel.
Discard information is collected for use in assessing the mortality for a
number of groundfish species.

    Further Actions Are Needed to Improve Stock Assessments

While the Northwest Center has implemented some recommendations aimed at
improving stock assessments, it has not yet fully implemented many others.
These recommendations include collecting additional types of data, such as
ecosystem and recruitment information; improving data quality, such as
calibrating survey vessel equipment; and increasing training opportunities
for Northwest Center staff. Also, NMFS has not acted on the task force
recommendation to combine the improvement plan and its complimentary plans
into a comprehensive plan that provides integrated program initiatives.

According to the improvement plan, additional types of data will allow
NMFS to further test and validate model assumptions, thereby increasing
the reliability of the stock assessments. The improvement plan further
states that information derived from ecosystem research and recruitment
surveys is essential if assessments are to meet the national standards of
"next generation" assessments or Tier 3 status. According to Northwest
Center officials, ecosystem information and coastwide recruitment surveys
are two of the most critical data sets needed to ensure continuous
improvement of groundfish stock assessments. The Northwest Center conducts
ecosystem research as part of its Science for Ecosystem-based Management
Initiative. Understanding the complex ecological relationships between
fish and the environment within which they exist provides a better
understanding of the effects of the ecosystem on the groundfish fisheries
and the scientific knowledge needed to make informed ecosystem-based
management decisions. Although research is ongoing to develop ecosystem
information, only a limited amount of the data is collected and used in
stock assessments. For example, ecosystem data are collected during shelf
and slope bottom trawl surveys as time and resources allow. However, this
information is not widely incorporated into stock assessments. For the
five species we reviewed, only the boccacio assessment used ecosystem
data-information on the temperature of the ocean's surface.

According to Northwest Center officials, the collection of ecosystem data
is limited because the relatively small size of the commercial vessels
used in the shelf and slope bottom trawl surveys cannot support the number
of researchers needed to effectively conduct comprehensive ecosystem
research and collection activities. Furthermore, the implementation of
comprehensive ecosystem research and data collection programs is
contingent upon the funding of a dedicated research vessel for west coast
surveys. Northwest Center officials said they are to receive a dedicated

research vessel sometime during calendar years 2008 or 2009, at the
earliest.

Better recruitment information for Pacific groundfish is also needed
because such information provides an early predictor of fish abundance,
especially for species such as hake, where there is a great variation in
recruitment. Northwest Center officials said that current recruitment
surveys are limited because existing funds support only yearly surveys in
selected areas. To achieve the best early predictions of stock status,
these officials said, recruitment surveys should be coastwide and
conducted twice a year. According to Northwest Center officials, 13
full-time staff are needed to expand these and other high-priority data
collection efforts, such as surveys in untrawlable habitat and expanded
acoustic surveys.

The lack of quality data was identified in the improvement plan as an
impediment to producing reliable stock assessments. For example, when
equipment on different survey vessels are not calibrated, the data are not
comparable, and trends may not be accurately determined. The Northwest
Center is continuing its efforts to calibrate survey vessel equipment.

The improvement plan also recommended that NMFS provide additional
training to ensure that qualified NMFS staff are available now and in the
future to conduct stock assessments and related activities. For example,
the plan recommended the development of a comprehensive training program
and more professional developmental opportunities for NMFS' scientific
staff. Northwest Center officials said they try to meet the training and
professional development needs of their scientific staff. However, to date
they have focused on developing external training programs, such as the
University of Washington graduate program, to develop stock assessment
scientists for the future and have yet to develop a comprehensive training
program for in-house stock assessment scientists.

Finally, the improvement plan recommended that NMFS prepare a
comprehensive plan that combines the improvement plan with other
complementary plans, such as the NOAA Fisheries Data Acquisition Plan and
the NMFS Social Sciences Plan. A comprehensive plan would allow NMFS to
better integrate and coordinate program initiatives for improving stock
assessments. For example, the acquisition plan-the key complementary plan
to the improvement plan-identifies the need for fishery research vessels
to satisfy NMFS' data collection needs. Although the improvement plan
includes the number of staff that would participate in data collection
surveys, it does not contain the capital and operating

costs of the research vessels. Similarly, the staffing requirements for
augmenting the social sciences capabilities of NMFS to conduct economic
analyses is represented in the sciences plan and not in the stock
improvement plan. A NMFS official said that other program priorities, such
as conducting more stock assessments and improving data collection
activities, have precluded them from developing a comprehensive plan.

Remaining According to NMFS funding and budget requests, the Northwest
Center

needs at least $8.9 million to complete ongoing and planned
improvementsImprovements to the stock assessments for Pacific groundfish.
However, the actual cost of Estimated to Cost at implementing remaining
improvements to Pacific groundfish stock Least $8.9 Million assessments
may be even higher because the Northwest Center's budget

requests primarily reflect the amount of money the Center believed it
could

realistically obtain, rather than the actual cost of the improvements.

    Remaining Improvements Are Estimated to Cost at Least $8.9 Million, but
    Estimate Is Likely Understated

According to NMFS, the Northwest Center needs at least $8.9 million to
complete ongoing and planned improvements for Pacific groundfish stock
assessments: $2.6 million that NMFS' Northwest Center requested but did
not receive between fiscal years 2001 to 2003 and $6.3 million the Center
requested for fiscal years 2004 and 2005. Specifically, as shown in table
2, the Northwest Center records have identified the following funding
needs

o 	$7.7 million to improve the types of data used, including $2.4 million
for surveys of untrawlable waters, $2.1 million to expand acoustic and
recruitment surveys, and $3.2 million to collect ecosystem data; and

o 	$1.2 million to improve the quality of data used in stock assessments,
including $600,000 to enhance the calibration of vessel equipment;
$525,000 to develop and implement methods to collect information on stock
identification, structure, and movement; and $75,000 to standardize trawl
survey procedures.

Table 2: Pacific Groundfish Stock Assessment Improvement Funds Requested
and Received by Data Type and Quality, Fiscal Years 2001-2005

Dollars in millions

Unfunded Category Requested Funded requests

               Types of data used Surveys for untrawlable waters

                Nearshore surveys (2003)                 $1.10  $.50     $.60 
          Untrawlable habitat surveysa (2003)              .30   .20   
    Longline groundfish fishing tag surveysa (2003)        .40    0    
        Augment trawl survey in nearshore (2004)           .51   N/A   
                Fixed-gear survey (2004)                   .78   N/A   

Subtotal

                    Expand acoustic and recruitment surveys

                      Recruitment surveys (2003)                      .40 .20 
    Acoustic studies of Southern California bight rockfisha (2003)    .20  0  
       Coastwide augmentation of recruitment survey by Southwest      .45     
                               Fisheries                                  N/A
                         Science Center (2004)                            
            Acoustic surveys in inaccessible habitat (2005)          1.25 N/A 

Subtotal

                                 Ecosystem data

               Page 24 GAO-04-606 Pacific Groundfish Assessments
                                                                         Ecosystem                                                                                                                                                                                                                                                                           
Coastwide            West                Habitat-specific                studies of                                                        Fishing                                              Stock                                          Stock                    Stock                   Stock                Stock                                 
observing            coast               resource surveys                  ocean                                              Quality      survey                   Trawl                  identification,                                   assessment              assessment               assessment           assessment                              
 system   .80 .60  observing .46 .20 .26 and fishing gear 1.45 N/A 1.45 productivity 1.28 N/A 1.28 Subtotal   3.2 Total   7.7 of data      vessel    .60 0 .60 standardization .45 .38 .08 structure, and  .53 N/A .53 Total   1.2 Other    training and .35 .15 .20 traininga,b .30 .50 -.20  training  .29 .29 0  training  .29 .29 0   Stock    .29 .29 0
 (2003)            systema,b              impacts (2004)                and climate                                            used      calibration               (2003)                  movement (2004)                                  coordination               (2003)                   (2003)               (2002)             assessment         
                    (2003)                                                impacts                                                          (2003)                                                                                              (2003)                                                                                    training          
                                                                           (2004)                                                                                                                                                                                                                                                         (2001)           

(Continued From Previous Page)

Dollars in millions

Unfunded Category Requested Funded requests

Total

Total unfunded requests

Source: GAO analysis of NMFS data.

Notes: These figures exclude the cost of constructing a new fishing
research vessel, which is currently designated to be home ported at and
used primarily by the Northwest Center for surveys of fish species, such
as Pacific groundfish. This vessel has not yet been funded and
construction has not yet begun.

The amounts requested by the Northwest Center in fiscal years 2004 and
2005 are preliminary budget requests.

aIncludes funding for the Southwest Fisheries Science Center assistance to
the Northwest Center's Pacific groundfish stock assessment improvement
program.

bRepresents requests and funding for all marine species-not Pacific
groundfish alone. NMFS was unable to identify what portion of these funds
went solely to groundfish.

The Northwest Center did not receive its full funding request, in part,
because NMFS did not receive all the funding it had requested. Between
fiscal years 2001 and 2003, NMFS received $20.6 million (80 percent of its
request) in additional funding to implement improvements for all marine
stock assessments. NMFS allocated $3.6 million (58 percent of funds the
Northwest Center requested) to the Northwest Center for improving Pacific
groundfish stock assessments, resulting in a $2.6 million shortfall in the
Center's request. This shortfall occurred in part because of NMFS' need to
balance the requests of its six science centers against its program
priorities and the available funds.18 According to NMFS officials, their
goal is to achieve parity among the science centers in terms of their
capability to conduct scientific work, such as stock assessments.

    Estimated Costs May Understate Actual Cost of Remaining Improvements

The $8.9 million needed to implement remaining recommended improvements is
probably understated because the Northwest Center's budget requests
primarily reflect the amount of money the Center officials believed they
could realistically obtain, rather than the amount the improvements would
actually cost, according to NMFS officials. The Northwest Center's budget
requests for fiscal years 2004 and 2005 are preliminary requests submitted
before the Northwest Center received its

18The sixth science center, the Pacific Islands Fisheries Science Center,
opened in 2003.

fiscal year 2003 funding.19 Consequently, the Northwest Center will likely
submit revised budget requests for fiscal years 2004 and 2005 that account
for both its unfunded needs from fiscal years 2001 through 2003 and items
that were unexpectedly funded in fiscal year 2003. Moreover, the fiscal
year 2004 and 2005 preliminary budget requests do not incorporate any
unanticipated problems or data gaps that have developed since the
Northwest Center submitted its preliminary requests.

According to NMFS officials, NMFS' science centers, including the
Northwest Center, primarily make and justify their funding requests in
response to how much money Congress appropriates. After Congress passes
NMFS' budget, NMFS asks its science centers to reassess and detail how
much new money each needs to implement science center programs, such as
marine stock assessment improvements. According to NMFS officials, it is
unrealistic for a science center to request more funds than are available
in its appropriation, even if it needs more. While NMFS' Northwest Center
requested $6.2 million to implement improvements to Pacific groundfish
stock assessments between fiscal years 2001 and 2003, NMFS' 2001 West
Coast Groundfish Research Plan estimated that almost twice as much money
would be needed-approximately $11.7 million in new funding-to implement
top-priority improvements to Pacific groundfish stock assessments.

NMFS is now updating its plan and cost estimates for improving Pacific
groundfish stock assessments. Using key findings from its December 2003
review of the groundfish program, the Northwest Center plans to update its
groundfish research plan, last published in 2001. According to NMFS, the
updated groundfish research plan should be completed in late 2004 and is
designed to (1) provide a comprehensive framework for Pacific groundfish,
(2) identify some of the greatest information gaps, and (3) provide
guidance for setting priorities on work to fill these gaps. In addition,
the updated plan will estimate how much such improvements will cost.

Conclusions	Stock assessments are the key to effectively managing
fisheries. They provide estimates of the species population, which NMFS
uses to set

19In light of the delayed congressional appropriation of fiscal year 2004
funds, NMFS has not yet determined the precise amount of money available
to its science centers. As a result, as of April 2004, NMFS had not yet
asked the Northwest Center to update its preliminary requests for fiscal
year 2004 funds.

harvest limits that allow for sustainability and/or recovery of the
species. While stock assessment results often change from assessment
report to assessment report, the more types of information used in the
assessments, such as recruitment surveys and ecosystem studies, and the
greater the accuracy and quality of the data, such as scientifically
designed and collected data, the more reliable the assessment results.
However, the Pacific groundfish assessments we reviewed did not (1) use
scientifically designed and collected NMFS data of sufficient scope and
accuracy, such as survey data on the abundance of groundfish residing in
rocky, untrawlable habitats; (2) subject the non-NMFS data used to a
standard process for assessing its reliability; and/or (3) identify the
uncertainties of the assessments total biomass estimates. As a result, the
reliability of the five assessments is questionable. Without reliable
assessments, fishery managers may reach erroneous conclusions and take
actions that could adversely affect the fishing industry economically or
adversely affect the recovery and sustainability of the fishery resources.
Moreover, without a comprehensive, integrated improvement plan, funding
requests and planned actions to improve the stock assessments may not be
coordinated, jeopardizing successful and timely implementation of
assessment improvements.

  Recommendations for Executive Action

To improve the reliability of Pacific groundfish stock assessments, we
recommend that the Secretary of Commerce require the Director of National
Marine Fisheries Service to take the following four actions:

o 	Continue efforts to collect more types of data, such as data obtained
from surveys in rocky, untrawlable waters, recruitment surveys, and
ecosystem studies, for groundfish assessments where reliable data are now
lacking.

o 	Establish a standard approach that requires that non-NMFS data used in
stock assessments be evaluated for its reliability, and continue efforts
to implement the task force's recommendations to improve data quality.

o 	Require that stock assessment reports clearly present the uncertainties
in the assessments, such as the margin of error associated with species
biomass estimates.

o 	Develop a comprehensive plan that integrates the NMFS stock assessment
improvement plan with other NMFS plans to ensure that

stock assessment improvement actions and budget requests are coordinated.

  Agency Comments and Our Evaluation

We provided the Department of Commerce with a draft of this report for
review and comment. We received a written response from the Under
Secretary of Commerce for Oceans and Atmosphere that included comments
from the National Oceanic and Atmospheric Administration (NOAA). NOAA
generally agreed with the report's accuracy and concurred with the reports
recommendations. However, NOAA said it was concerned about the report's
conclusion-that the reliability of stock assessments is questionable for
the five species reviewed-because it could be misconstrued to infer that
the assessments are unreliable for use in managing the west coast
groundfish fishery. In this regard, NOAA provided additional comments to
show the usefulness of the assessments, even if some of the input data
used in the assessments contained errors. We stand by our conclusions that
the five stock assessments we reviewed were questionable because the input
data were insufficient and/or potentially inaccurate and that four of the
assessment reports did not present the uncertainties associated with the
biomass estimates. Nonetheless, we added language to the report to address
NOAA's concern. Specifically, we expanded upon the fact that NMFS used the
best information available at the time the stock assessments were
conducted by adding information on the importance of the assessments to
effectively manage the fisheries. Without these stock assessments, NMFS
and fishery managers would have very limited information on which to base
fishery management decisions.

NOAA agreed with the report recommendation to continue collecting more
types of data for groundfish assessments where reliable data are now
lacking. NOAA said that the reliability of stock assessments will be
improved if NMFS survey efforts are expanded and additional NMFS fishery
data are collected. NOAA said NMFS places a priority on these improvements
and will continue efforts to address this and other recommendations to
improve the collection of fishery data as funding becomes available.

NOAA also agreed with the report recommendation to establish a standard
approach to evaluate the reliability of non-NMFS data used in stock
assessments and continue efforts to improve data quality. NMFS said,
through its west coast fishery science centers, it participates on
interagency data committees, to develop quality assurance protocols and to
assess the quality of non-NMFS data. NOAA agreed that it is important to

ensure that these interagency data committees continue to highlight the
need for standardized quality control procedures for the collection of
data.

NOAA agreed with the report recommendation to clearly present the
uncertainties in the stock assessments. NOAA said that quantifying
uncertainty of stock assessments is important to sound decision-making by
providing more information about the assessment, although this
quantification does not reduce the uncertainty in the assessment itself.
While the methods used and the completeness of the uncertainty
characterization varied from assessment to assessment, NOAA said it is
desirable to have both a quantitative analysis of model uncertainty and an
evaluation of the consequences of alternative model scenario.

Finally, NOAA agreed with the report recommendation to develop a
comprehensive plan that integrates the stock assessment plan with other
NMFS plans to ensure that improvement actions and budget initiatives are
coordinated. NOAA said that while much remains to be done, long-term
planning efforts and coordination among field and headquarters are
ongoing, and NOAA is committed to these actions.

NOAA's comments and our detailed responses are presented in appendix II of
this report. NOAA also provided technical comments that we incorporated in
this report as appropriate.

As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 7 days from the
report date. At that time, we will send copies of this report to the
Secretary of Commerce and the Director of the National Marine Fisheries
Service. We will also provide copies to others upon request. In addition,
the report will be available at no charge on the GAO Web site at
http://www.gao.gov.

If you or your staff have any questions about this report, please call me
at (202) 512-3841 or Keith Oleson at (415) 904-2218. Key contributions to
this report are listed in appendix III.

Anu K. Mittal Director, Natural Resources and Environment

Appendix I

                       Objectives, Scope, and Methodology

We reviewed National Marine Fisheries Service (NMFS) stock assessments for
five species of Pacific groundfish: Pacific hake (Pacific whiting) as well
as four types of rockfish-bocaccio, canary, darkblotched, and yelloweye.
Specifically, for these five species you asked us to (1) assess the
reliability of NMFS' stock assessments, (2) identify which relevant
recommendations from the stock assessment improvement plan have been
implemented and which have not, and (3) identify the estimated costs
associated with planned and ongoing improvements to groundfish stock
assessments. We did not review the stock assessments of any of the other
west coast Pacific groundfish species, thus the information contained in
this report pertains to the five species we reviewed unless stated
otherwise.

For all three objectives, we reviewed key laws and agency reports and
interviewed officials from NMFS, including officials from the Northwest
Fisheries Science Center, which has lead responsibility for conducting
Pacific groundfish stock assessments with assistance from other west coast
science centers.

To assess the reliability of the Pacific hake, bocaccio, canary,
darkblotched, and yelloweye stock assessments, we examined methodological
and administrative documents developed by NMFS and others to support the
groundfish data collection, maintenance, and assessment process. We
reviewed the controls over stock assessment data, the types of fish
population surveys used, and recent Pacific groundfish stock assessment
studies (2002 and 2004 studies for Pacific hake, 2002 and 2003 studies for
bocaccio, 2002 study for canary, 2000 and 2003 studies for darkblotched,
and 2001 and 2002 studies for yelloweye). We examined whether and to what
extent NMFS has processes and procedures in place to ensure the
reliability of data used in the Pacific groundfish stock assessments. We
reviewed the stock assessment reports and determined whether they
articulated the level of uncertainty in the assessment model estimates. We
interviewed an array of government officials and fisheries experts,
including the Oregon Department of Fish and Wildlife, the Washington
Department of Fish and Wildlife, the California Department of Fish and
Game, the Pacific Fishery Management Council, the Pacific States Marine
Fisheries Commission, environmental groups, and industry associations, as
well as fishermen and academics. We did not simulate NMFS' stock
assessment models nor evaluate the mathematical and statistical
methodologies used in the models for Pacific hake, bocaccio, canary,
darkblotched, and yelloweye.

Appendix I
Objectives, Scope, and Methodology

To identify the relevant recommendations to improve stock assessments that
NMFS has implemented and has not implemented, we reviewed agency reports
on marine fisheries stock assessments, strategic planning, and data
collection. We also interviewed officials from the Oregon Department of
Fish and Wildlife, the Washington Department of Fish and Wildlife, the
California Department of Fish and Game, the Pacific Fishery Management
Council, and the Pacific States Marine Fisheries Commission as well as
environmental groups, industry associations, fishermen, and academics.

To determine the estimated costs associated with NMFS' planned and ongoing
improvements to Pacific groundfish stock assessments, we reviewed relevant
budget requests and funding documents for fiscal years 2001 through 2005
and interviewed National Oceanic and Atmospheric Administration officials.
We did not evaluate the accuracy of NMFS' budget requests for specific
project items but rather used the amounts NMFS requested for these project
items to estimate the total additional costs of implementing the planned
and ongoing improvements to Pacific groundfish stock assessments.

We conducted our review from May 2003 through April 2004 in accordance
with generally accepted government auditing standards.

Appendix II

Comments From the Department of Commerce

Note: GAO comments supplementing those in the report text appear at the
end of this appendix.

Appendix II
Comments From the Department of
Commerce

Page numbers in the draft report may differ from those in this report.

See comment 1.

See comment 2. See comment 3.

See comment 4.

See comment 5.

See comment 6.

See comment 7. See comment 8.

See comment 9.

Appendix II
Comments From the Department of
Commerce

See comment 10.

See comment 2. See comment 11.

See comment 12.

See comment 2. See comment 6. See comment 13.

See comment 14. See comment 15.

See comment 16.

See comment 17. See comment 18.

See comment 19.

Appendix II
Comments From the Department of
Commerce

See comment 19.�

See comment 20.�See comment 7.�See comment 21.�See
comment 2.�See comment 22.�

See comment 23.�

See comment 24.�

See comment 25.�

See comment 26.�See comment 27.�See comments 7 and
21.�

See comment 28.�

Appendix II
Comments From the Department of
Commerce

                                See comment 29.

                        See comment 30. See comment 31.

See comment 32.

See comment 6.

See comment 33. See comment 19. See comment 34.

See comment 35.

Appendix II
Comments From the Department of
Commerce

                                See comment 36.

                                See comment 37.

                                See comment 38.

Appendix II
Comments From the Department of
Commerce

                                See comment 39.

                                See comment 40.

Appendix II
Comments From the Department of
Commerce

Appendix II
Comments From the Department of
Commerce

                                  Appendix II
                        Comments From the Department of
                                    Commerce

      The following are GAO comments on NOAA's letter dated May 13, 2004.

  GAO Comments 1.

2.

3.

4.

5.

6.

7.

8.

9.

We added clarifying language to the scope and methodology section of the
report to clearly identify the species and activities covered by the
review.

We revised the report to show the publication date.

We revised the report accordingly.

We revised the report to clearly show that NMFS has not collected enough
ecosystem data and that the frequency and range of recruitment surveys are
limited. The statement does not address untrawlable habitat.

We revised the report to clarify that NMFS "generally uses" NMFS's staff
or contracts with outside experts.

We revised the report to more clearly differentiate between NMFS as a
whole and NMFS' Northwest Center in particular. We made similar revisions,
as appropriate, throughout the report.

We revised the report to specify "bottom" trawl survey. We made similar
changes, as appropriate, throughout the report.

We revised the report to include the year and scope of the task force
review.

We revised the report to indicate that the Northwest Center is responsible
for coordinating groundfish stock assessments.

10. We revised the report to include the citation.

11. We revised the report to include the date and citation of the National
Research Council report.

12. The NMFS data used in the bocaccio, canary, and darkblotched
assessments were limited because NMFS conducted its surveys in trawlable
waters only. NMFS data were not available for untrawlable waters, which
these species also inhabit. For this reason, we did not revise the report.

Appendix II
Comments From the Department of
Commerce

13. We revised the report to clarify the shared responsibilities of the
Northwest and Southwest Centers.

14. We revised the report to include NOAA's recommended definition of
stock abundance.

15. We revised the report to include larval fish.

16. We revised the report to clarify the role of the review panel.

17. We revised the report to more explicitly distinguish the five species
related to our report from other overfished Pacific groundfish.

18. We revised the report to more clearly describe the distribution of
Pacific hake.

19. Bocaccio survey data for untrawlable habitats, as stated in comment
12, was unavailable. For this reason, we did not change the report.

20. NOAA commented that highly standardized protocols are used for
collecting non-NMFS data (fishery dependent data) for rockfish. We found
that although NMFS does have collection and quality assurance procedures
for state-collected non-NMFS data, NMFS does not check or have a standard
process to verify that these data have been reviewed for reliability. As
discussed in our report, some assessors chose to review the raw data,
while others did not. Assessors who voluntarily reviewed raw non-NMFS data
found mistakes that either made some of the data unusable or could have
impaired the accuracy of the stock assessments. For these reasons, we did
not change the report.

21. We revised the punctuation accordingly.

22. We revised the report to clarify that Pacific hake live in mid-water
habitat.

23. The footnote placement and citation are in accordance with GAO
guidelines. For this reason, we did not change the report.

24. We believe that our report has addressed this issue. By referring to
the West Coast Groundfish Research Plan by its complete title, we
adequately distinguish between the two reports. For this reason, we did
not change the report.

Appendix II
Comments From the Department of
Commerce

25. We changed "health" to "stock abundance."

26. The footnote placement is repositioned in report.

27. We revised the report to include assessment "improvements."

28. As indicated in our report, we illustrate some of the actions that the
Northwest Center took to improve data quantity and did not intend to
provide a comprehensive list of all actions conducted to improve data
quantity coast wide. However, we added footnote 16 to clarify the actions
taken by the Southwest Center.

29. The example we provided is not intended to be a comprehensive list of
all ecosystem research conducted on the west coast. Instead, it
illustrates the type of work the Northwest Center is conducting and the
opportunities for improving ecosystem research. For this reason, we did
not change the report.

30. After reviewing the report we believe no change is required because of
subject-verb agreement.

31. We added clarifying language.

32. We believe that table 2 notes "a" and "b" in our report already
adequately address this issue. Annotations for projects that do not
separate out groundfish funds occur only in items that are annotated as
Southwest Center projects. For this reason, we did not change the report.

33. We changed "survivability" to "sustainability."

34. We changed "survivability" to "sustainability" and added fishery
"resources" for clarification.

35. NOAA commented that GAO does not adequately convey the different
degrees of precision associated with the stock assessments and GAO's
conclusion that the reliability of the five assessments we reviewed is
questionable and could easily be misconstrued to mean all these
assessments are an unreliable basis for management of the west coast
groundfish fishery. NOAA also commented that the five assessments GAO
reviewed all passed scientific review and are serving as the basis for
formal status determination and fishery management. Our report

Appendix II
Comments From the Department of
Commerce

acknowledges that stock assessments are scientifically reviewed and are a
key tool for managing fisheries. However, we found the reliability of the
five assessments questionable for the three reasons we highlighted in our
report, and we recommended actions on how to improve the reliability of
the stock assessments. We added clarification to the report to show that
stock assessments are a key tool for managing fisheries and are important
in making decisions about setting harvest levels and developing plans to
rebuild overfished stocks.

NOAA also commented that quality assurance for non-NMFS data is not
absent. As stated in our response number 20, we found that although NMFS
does have collection and quality assurance procedures for statecollected
non-NMFS data, NMFS does not check or have a standard process to verify
that these data have been reviewed for reliability. As discussed in our
report, some assessors chose to review the raw data, while others did not.
Assessors who voluntarily reviewed raw non-NMFS data found mistakes that
either made some of the data unusable or could have impaired the accuracy
of the stock assessments. For these reason, we did not change the report.

36. NOAA commented that it is more pertinent to focus on the degree of
standardization of the survey data than the source. By categorizing data
as NMFS data and non-NMFS data, we were not implying that non-NMFS
organizations could not conduct useful fishery-independent surveys. We
categorized the data in this manner because NMFS currently conducts nearly
all of the fishery-independent surveys and non-NMFS organizations collect
most of the fishery-dependent data. Footnote 7 in the report states that
NMFS generally refers to its data as fishery-independent data and to
non-NMFS data as fishery-dependent data. For these reasons, we did not
change the report.

37. We believe the Pacific hake biomass estimates are questionable because
the assessment used non-NMFS data that NMFS did not check or subject to
standard data reliability testing. Assessors who reviewed raw non-NMFS
data for other stock assessments found mistakes that either made some of
the data unusable or could have impaired the accuracy of the stock
assessments. For this reason, we did not change the report.

38. NOAA commented that bocaccio, canary, and darkblotched assessments all
obtain adequate abundance trend information from the NMFS bottom trawl
surveys. NOAA also commented that although

Appendix II
Comments From the Department of
Commerce

bottom trawl survey cannot access the roughest habitat, it is useful as an
index of relative changes in the overall abundance. As stated in our
report, we found that the NMFS survey data used in these assessments were
limited in scope because the surveys were conducted only in trawlable
areas. Assessors estimated overall biomass using the NMFS data collected
from the trawlable area, which has a different abundance rate than the
untrawlable area. Stock assessors commented that relying on survey data
from trawlable waters only increases the uncertainty of stock assessments.
For these reasons, we did not change the report.

39. As noted in our report, the National Research Council found that the
inclusion of NMFS survey data was the best option for a reliable estimate
of abundance because such surveys use an unbiased statistical design,
control sampling locations, and provide for quality assurance. Northwest
Center officials said that to obtain reliable results, each stock
assessment should include at least one source of NMFS-collected data of
sufficient scope and accuracy because such surveys are unbiased and
scientifically designed. NMFS data were unavailable for the yelloweye
assessment. Northwest Center officials also raised concerns about basing
assessments solely on non-NMFS data such as commercial and recreational
catch data. Catch data do not provide the species' relative or absolute
biomass, according to NMFS officials. Catch data alone are insufficient
because fishermen are not randomly sampling the ocean, but are fishing
areas that they are allowed to fish and they believe have the most fish;
fishing restrictions, such as a total allowable catch, can limit the
amount of fish being caught; and catch data have often been inaccurate for
a variety of reasons, such as imprecise accounting for dead fish tossed
back into the ocean. For these reasons, we did not change the report.

40. NOAA commented that the doubling of estimated bocaccio biomass in 2003
was due to factors that would not be addressed in a standard statistical
analysis. Although a standard statistical analysis may not fully address
the doubling of an estimate, an assessment without an uncertainty range
does not quantify and communicate any of the uncertainty. For this reason,
we did not change the report.

Appendix III

                     GAO Contact and Staff Acknowledgments

GAO Contact Keith W. Oleson, (415) 904-2218

Staff 	In addition to the person named above, Leo G. Acosta, Kristine N.
Braaten, Allen T. Chan, David Dornisch, Alan Kasdan, Robert Marek, Cynthia
C.

Acknowledgments	Norris, Carol Herrnstadt Shulman, and Tama R. Weinberg
made key contributions to this report.

GAO's Mission	The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

  Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.

Order by Mail or Phone	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548

To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

  To Report Fraud, Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htmWaste, and Abuse in E-mail:
[email protected] Federal Programs Automated answering system: (800)
424-5454 or (202) 512-7470

Public Affairs	Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

                               Presorted Standard
                              Postage & Fees Paid
                                      GAO
                                Permit No. GI00

United States
General Accounting Office
Washington, D.C. 20548-0001

Official Business
Penalty for Private Use $300

Address Service Requested
*** End of document. ***