Food Safety: Weaknesses in Meat and Poultry Inspection Pilot
Should Be Addressed Before Implementation (17-DEC-01, GAO-02-59).
In 1997, the Department of Agriculture (USDA) announced that it
would modify its meat and poultry slaughter inspection program to
make industry more responsible for identifying carcass defects.
Before making the change permanent, USDA developed a model to
test whether a prevention-oriented inspection system that uses
plant personnel to examine each carcass and USDA inspectors to
verify that quality standards are met would continue to ensure
the safety of meat and poultry products. USDA's pilot project for
chickens has several design and methodology limitations that
compromise the overall validity and reliability of its results.
First, the chicken pilot that USDA designed lacks a control
group--a critical design flaw that precluded a comparison between
the performance of the inspection systems at those plants that
volunteered to participate in the pilot and that of plants that
did not participate. Second, the chicken plants that volunteered
to participate in the baseline measurement phase of the pilot
were not randomly selected, and they did not include plants from
all chicken-producing ares or plants of all sizes. Third, the
pilot project's methodology did not take into account such
variables as seasonal changes and plant modifications that could
affect project results. Finally, USDA's pilot project did not
include features of the modified inspection systems in Australia
and Canada that would be important considerations in ensuring the
successful implementation of a modified inspection system
nationwide. Notwithstanding the project's design problems, the
data themselves do not conclusively demonstrate that modified
inspections are at least equal to traditional inspections.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-59
ACCNO: A02437
TITLE: Food Safety: Weaknesses in Meat and Poultry Inspection
Pilot Should Be Addressed Before Implementation
DATE: 12/17/2001
SUBJECT: Food and drug law
Meat inspection
Poultry inspection
Contaminated foods
Consumer protection
Safety regulation
Food facilities
Listeria monocytogenes Bacteria
Salmonella Enteritidis Bacteria
USDA Meat and Poultry Inspection Program
Australia Meat Safety Enhancement
Program
Canada Modernized Poultry Inspection
Program
E.coli Bacteria
FSIS Hazard Analysis and Critical
Control Point System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-59
United States General Accounting Office GAO Report to the Committee on
Agriculture, Nutrition, and Forestry, U. S. Senate
December 2001 FOOD SAFETY Weaknesses in Meat and Poultry Inspection Pilot
Should Be Addressed Before Implementation
GAO- 02- 59
Page i GAO- 02- 59 Meat and Poultry Pilot Inspection Project Letter 1
Results in Brief 2 Background 5 Pilot Project Makes Industry More
Responsible and Accountable
for Product Safety and Quality, While Continuing Government Monitoring With
Fewer Resources 9 Design and Methodology Limitations Compromise Overall
Validity
and Reliability of Pilot Project Results 15 Pilot Project Results Are
Inconclusive, but FSIS Inspection
Personnel Generally Support Modified Inspections 25 Conclusions 33
Recommendations for Executive Action 35 Agency Comments and Our Evaluation
35
Appendix I Scope and Methodology 37
Appendix II Chronology of Pilot Project Litigation 40
Appendix III Performance Standards for Hog Plants 43
Appendix IV Survey Results 44
Appendix V Australia?s Meat Safety Enhancement Program 59 Background and
Trends That Led to the Program 59 How the MSEP System Works 60 Prerequisites
for Plants to Participate in MSEP 61 AQIS? Evaluation of Its Program 62
Appendix VI Canada?s Modernized Poultry Inspection Program 63 Background 63
How the MPIP Pilot Works 63 Prerequisites for Plants to Participate in the
MPIP Pilot 64 Performance Standards 65 Contents
Page ii GAO- 02- 59 Meat and Poultry Pilot Inspection Project
MPIP Pilot Success Attributed to Extensive Training of Inspectors and
Industry 66 Under MPIP Pilot, CFIA Officials Monitor Plant Performance
Trends 67 Current Status of MPIP 68
Appendix VII Comments From the U. S. Department of Agriculture 69 GAO?s
Comments 76
Appendix VIII GAO Contact and Staff Acknowledgments 81 GAO Contact 81
Acknowledgments 81
Tables
Table 1: Number and Types of Pilot Plants That Participated in Traditional
and Modified Inspections 11 Table 2: Traditional Inspection System?s Results
(Baseline Data)
From 16 Chicken Pilot Plants: Defects per Shift in Percents 13 Table 3:
Performance Standards for Inspection Systems at Chicken
Plants in the Pilot Project 26 Table 4: Eleven Chicken Plants? Inspection
Systems Compliance
With Performance Standards (RTI?s Data) 27 Table 5: Eleven Chicken Plants?
Inspection Systems Performance
After Shifting to Modified Inspections (RTI?s Data) 28 Table 6: Ten Chicken
Plants? Inspection Systems Compliance With
Performance Standards (FSIS? Data) 29 Table 7: Changes in Chicken Plants?
Inspection Systems
Performance Over Two Time Periods After Shifting to Modified Inspections
(FSIS? Data) 30 Table 8: Summary of Responses to GAO?s Survey of USDA
Inspectors and Veterinarians 31 Table 9: Performance Standards for
Inspection Systems at Hog
Plants in the Pilot Project 43 Table 10: Division of Responsibility Under
Australia?s Meat Safety
Enhancement Program 60
Page iii GAO- 02- 59 Meat and Poultry Pilot Inspection Project Figures
Figure 1: Traditional and Modified Inspection Systems on a Chicken Slaughter
Line 8 Figure 2: Location of Chicken Pilot Plants 18 Figure 3: Fecal
Noncompliance Records Under Traditional and
Modified Inspections 23
Abbreviations
CCP Critical Control Point CDC Centers for Disease Control and Prevention
FSIS Food Safety and Inspection Service HACCP Hazard Analysis and Critical
Control Point HIMP HACCP- Based Inspection Models Project RTI Research
Triangle Institute USDA U. S. Department of Agriculture
Page 1 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
December 17, 2001 The Honorable Tom Harkin Chairman The Honorable Richard G.
Lugar Ranking Minority Member
Committee on Agriculture, Nutrition, and Forestry United States Senate
While the food supply in the United States is generally safe, foodborne
bacteria, viruses, and other pathogens can cause a significant number of
illnesses, a small fraction of which are life- threatening. According to
estimates from the Centers for Disease Control and Prevention (CDC),
foodborne diseases cause 76 million illnesses in the United States each
year, including 325,000 hospitalizations and 5,000 deaths. Meat and poultry
products contaminated with bacteria such as Salmonella, Campylobacter,
Escherichia coli (E. coli), and Listeria cause most foodborne illnesses and
deaths. To control the spread of foodborne illnesses through meat and
poultry products, approximately 3,400 U. S. Department of Agriculture (USDA)
inspectors at 1,300 slaughter plants are stationed along slaughter lines to
provide continuous inspection and conduct organoleptic examinations- using
sight, touch, and smell- of each and every carcass.
In 1997, USDA announced the need to modify its meat and poultry slaughter
inspection program to make industry more responsible for identifying carcass
defects. This approach is consistent with the agency?s previous adoption of
the Pathogen Reduction: Hazard Analysis and Critical Control Point (HACCP)
regulations. The HACCP approach is risk- based and makes industry, rather
than federal inspectors, responsible for identifying steps in food
production where food safety hazards are most likely to occur and for
establishing controls that prevent or reduce it. USDA had not extended the
HACCP principles to slaughter inspections because the agency has
traditionally provided continuous inspection of each and every carcass.
However, USDA believes that changing its traditional inspection system would
also reduce inspectors? reliance on organoleptic inspections, allow for a
shift to prevention- oriented inspection systems based on risk, and permit
redeployment of its resources to better protect the public from foodborne
diseases.
Before making a permanent change to its slaughter inspections system, USDA
developed a model to test whether such a change would continue to ensure the
safety of meat and poultry products. This model is being tested
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
at several volunteer chicken and hog plants. At plants participating in the
project, plant personnel, instead of USDA inspectors, examine each carcass
to make an initial determination whether it is unacceptable and should be
removed from the slaughter line. At the participating plants, USDA first
measures the performance of traditional inspection systems and then compares
it with the performance of modified inspections. USDA will propose
regulations to modify its inspection system nationwide if the project shows
that the modified inspection system performs as well as or better than
traditional inspections. A reduced number of USDA inspectors are still at
each plant to verify that safety and quality standards are met. In addition,
at least one USDA inspector is positioned near the end of each slaughter
line in order to comply with a federal court ruling that USDA cannot
delegate the inspection of carcasses to plant employees. As of July 2001,
the federal cost associated with this project was approximately $5.7
million, and USDA estimates that it will spend another $1.2 million.
Australia and Canada have tested similar modified inspection programs for
meat and poultry and approved their use on a nationwide basis.
Concerned about the design, methodology, and reliability of the pilot
project, you asked us to (1) describe the objectives, design, and scope of
the project; (2) identify limitations, if any, in the project?s design and
methodology; and (3) any design and methodology limitations notwithstanding,
determine if the data generated by the project will allow USDA to reach
valid conclusions on the relative effectiveness of modified and traditional
inspection methods in ensuring food safety and quality. In addressing these
objectives, you also asked us to obtain information from similar projects in
Australia and Canada.
In response to your request, we reviewed USDA?s pilot project as implemented
at chicken and hog plants. However, we were able only to analyze data from
chicken plants because complete data from hog plants were not yet available.
We also visited Australia and Canada to learn about similar pilot projects.
Appendix I describes our methodology in detail.
USDA?s original objective in implementing the pilot project was to test
whether a prevention- oriented inspection system that uses plant personnel
to examine each carcass for safety and quality and USDA inspectors to verify
that safety and quality standards are met can provide a level of product
safety and quality equal to or better than traditional inspections. At 11
chicken and 3 hog plants that are voluntarily participating in the before
and after phases of the project, plant personnel, instead of USDA
inspectors, initially determine which carcasses and parts are unacceptable
Results in Brief
Page 3 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
and should be removed from the slaughter line because they are diseased or
unwholesome. USDA is, therefore, able to use fewer inspection personnel at
these plants. Using safety and quality performance standards developed for
the project, an independent contractor measured how well pilot plants?
inspection systems performed against these standards. The contractor
measured how well USDA inspectors identified carcass defects under
traditional inspections and then conducted similar measurements once plant
personnel assumed those duties. Also, as a part of the pilot project, USDA
inspectors at the project plants have examined carcasses to evaluate how
well plant employees detect carcass defects. USDA will analyze these data to
determine if at least the same level of safety and quality is maintained
under the modified inspections. USDA has announced that preliminary results
from the chicken pilot project show that plants? inspection systems perform
better under the modified inspections, and that it expects to propose
regulations to modify its slaughter inspection system for all chicken plants
early in 2002. Both Australia and Canada have recently tested and adopted
modified inspection programs that resemble USDA?s pilot project in that they
include the replacement of some government inspectors with plant personnel.
USDA?s pilot project for chickens has several design and methodology
limitations that compromise the overall validity and reliability of its
results. Hence, it is questionable whether the data generated by the project
are indicative of how all of the chicken plants? inspection systems would
perform if modified inspections were adopted nationwide. First, the chicken
pilot that USDA designed lacks a control group- a critical design flaw that
precludes a comparison between the performance of the inspection systems at
those plants that volunteered to participate in the pilot and that of plants
that did not participate. Without a control group, USDA cannot determine
whether changes in inspections systems are due to personnel changes or other
possible explanations, such as the addition of chlorine rinses. Second, the
chicken plants that volunteered to participate in the baseline measurement
phase of the pilot were not randomly selected, and they did not include
plants from all chickenproducing areas or plants of all sizes. Thus, the
results cannot be generalized to the entire population of chicken slaughter
plants in the United States. Third, the pilot project?s methodology did not
take into account variables such as seasonal changes and plant modifications
that could affect project results. For example, after the project began,
many plants added antimicrobial rinses and washers, which usually reduce the
levels of microbial contamination. USDA acknowledges some of the project?s
limitations but maintains that the design is consistent with that of
Page 4 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
other USDA studies and that it is difficult to conduct this type of
experiment without relying on volunteer participants. Finally, USDA?s pilot
project did not include features of the modified inspection systems in
Australia and Canada that would be important considerations in ensuring the
successful implementation of a modified inspection system nationwide. For
example, during the pilot project, USDA did not require the training of
plant employees. Such training would be important in ensuring that plant
personnel are as competent as federal inspectors in identifying carcass
defects that can affect product safety and quality. Similarly, the pilot
project did not require the use of statistical process controls as required
in Australia and Canada. Statistical process controls provide a means to
determine whether the plant?s production processes are performing within
established performance standards. This technique would enable USDA and the
plants to better measure and control their performance daily and over time
to ensure continuous improvement. Additionally, USDA allowed plants with
repeated noncompliance records to continue participating in the pilot
project. If USDA decides to permanently modify its inspection system, a
phased- in approach such as Canada?s would help to ensure that plants
continually improve their ability to ensure product safety and quality and
that those that have difficulty return to traditional inspections.
Notwithstanding the project?s design problems, which we believe make the
results unreliable, we found that, so far, the data themselves do not
conclusively demonstrate that modified inspections are at least equal to
traditional inspections. Part of the difficulty is that the two data sets
collected for this pilot show somewhat different results. These data show
whether the pilot plants met the seven organoleptic performance standards-
two food safety standards and five other consumer protection (quality)
standards- that USDA developed to compare, first the inspection systems?
performance at plants under traditional inspections and then under modified
inspections. The modified inspection systems at plants in the pilot project
were expected to perform at least as well as they did under traditional
inspections. However, the contractor?s data show that while the majority of
the chicken plants? inspection systems met or exceeded four or more of the
seven organoleptic standards adopted by USDA, none met all seven.
Specifically, the contractor?s data show that most of the inspection systems
at these plants did not meet the safety standard for the presence of fecal
material, which could contain harmful bacteria such as E. coli. However, the
traditional inspection system at these plants was also unable to meet this
standard. Most inspection systems also did not meet quality standards for
defects such as feathers and oil glands, but USDA officials pointed out that
such defects are of a
Page 5 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
lesser concern because they are not associated with foodborne illness and
they can generally be removed from the carcass after it leaves the slaughter
line and moves to the processing area. USDA?s contractor also measured
whether the modified inspection systems at these plants met two other
standards that existed before the pilot project- the pathogen reduction
performance standard for Salmonella and generic E. coli. The data show that
9 of the 11 inspection systems met the Salmonella standard and that 80
percent of the samples collected for generic E. coli were within the
acceptable ranges. On the other hand, the data collected by USDA inspectors
at these chicken plants show better results than those shown by the
contractor?s data. According to USDA?s data, 7 of 10 plants? inspection
systems met 6 of the 7 standards. While the data generated by the pilot
project yielded inconclusive results, the pilot project?s inspectors and
veterinarians that we surveyed for the most part believe that a modified
inspection system results in safer products. More than half of those
surveyed said that modified inspections are equal to or somewhat better than
traditional inspections in ensuring product safety and quality. Modified
inspections allow for additional time to inspect slaughter line operations
more thoroughly because inspectors, except for the carcass inspector, are no
longer tied to a fixed location.
This report reiterates our previous recommendation for legislative revisions
aimed at reducing the potential for further legal challenges by providing
USDA with clear authority to modify its inspection system. In addition, this
report makes several recommendations to the Secretary of Agriculture aimed
at ensuring that, if USDA decides to implement a modified inspection system,
such a system will effectively ensure product safety and quality. In
commenting on a draft of this report, USDA said that the report?s
recommendations for executive action are appropriate and that the issues
noted in the report can be satisfactorily addressed. USDA also provided
technical comments which we incorporated as appropriate.
The Federal Meat Inspection Act and the Poultry Products Inspection Act, 21
U. S. C. sections 604 and 455, respectively, give USDA overall
responsibility for ensuring the safety and wholesomeness of meat and poultry
products that enter interstate commerce. Acting under these legislative
authorities, USDA has engaged in continuous government inspection of each
and every carcass at slaughter plants throughout the United States. Within
USDA, the Food Safety and Inspection Service (FSIS) is responsible for
inspections at all meat and poultry slaughter and processing plants and for
ensuring plants? compliance with regulatory requirements. Background
Page 6 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
At slaughter plants, FSIS inspectors are located at fixed stations and
examine carcasses to identify and remove defects that impair product safety
and quality. Defects that are likely to present a public health risk, such
as the presence of potentially pathogenic fecal material, are considered
food safety hazards. Product quality defects, such as feathers or bruises on
the carcasses, are aesthetic flaws that rarely present a direct safety risk.
Inspectors perform different tasks, depending on the animal species being
slaughtered. For example, in chicken plants, the inspectors are located at
fixed stations immediately after the place on the slaughter line where
carcasses are opened and the organs removed. From these positions, the
inspectors examine the exterior, interior cavity, and organs of each
carcass. The inspectors then determine which carcasses must be condemned,
which have conditions that may be trimmed, and which may proceed to the next
stage in the slaughter line. Carcasses are then trimmed, rinsed, and, as the
final step of the slaughter line, placed into a chiller. From here,
carcasses move to the processing area of the plant, where they are cut up
and packaged. In contrast, in hog plants, the inspectors are located at
three fixed stations on the slaughter line. At the head station, the
inspectors excise and evaluate the head lymph nodes; at the viscera station,
they evaluate most organs; and at the final carcass station, they examine
each carcass for possible contamination, disease conditions, and other
adulterants before it goes into the chiller or cooler.
For a variety of reasons, including responding to recommendations from the
National Academy of Sciences and GAO that FSIS shift its resources to a
prevention- oriented, risk- based inspection system, in 1996 USDA embarked
on regulatory reform efforts to reduce foodborne hazards in meat and poultry
products. The agency adopted a new science- based process control system,
the Pathogen Reduction: Hazard Analysis and Critical Control Point system.
HACCP is a risk- based approach that makes industry, not federal inspectors,
responsible for identifying steps in food production where contamination is
most likely to occur and for establishing controls that prevent or reduce
contamination.
USDA anticipated that the HACCP framework would also require changes in the
roles and responsibilities of its inspection workforce, as industry, not
government, becomes increasingly responsible for the safety of its products.
For slaughter line operations, however, making industry more directly
responsible for the safety and quality of their products was problematic
because of a legal requirement for continuous carcass- bycarcass inspection.
USDA believed that, to more fully integrate the HACCP
Page 7 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
concept into meat and poultry slaughter operations, its traditional
inspection system would have to be modified to transfer some of the carcass
defect detection responsibilities to industry personnel. Before implementing
such an approach, FSIS decided to conduct a pilot to test and evaluate
whether product safety and quality could be maintained at a comparable level
if plant employees perform the carcass defect detection duties and a reduced
number of federal inspectors remain to verify product safety and quality
through increased testing and observation.
The pilot project is known as the HACCP- Based Inspection Models Project
(HIMP) and its original design was intended to measure the effect of
completely removing FSIS inspectors from their fixed locations on the
slaughter line while maintaining one oversight and one verification
inspector. In April 1998, the inspectors? union filed suit in the United
States District Court for the District of Columbia to enjoin the USDA from
proceeding with the pilot project, on the grounds that it violated the
Federal Meat Inspection Act, 21 U. S. C. section 604, and the Poultry
Products Inspection Act, 21 U. S. C. section 455, by not requiring federal
government officials to perform carcass- by- carcass postmortem inspections.
In June 2000, the United States Court of Appeals for the District of
Columbia Circuit ruled that delegating the task of inspecting carcasses to
plant employees violated the acts because both statutes require that federal
inspectors, rather than private employees, determine whether a product is
adulterated. As a result of this ruling, in September 2000, FSIS redesigned
the pilot project for chickens and hogs and placed at least one FSIS
inspector back at a fixed location on each slaughter line to inspect each
carcass.
Following a district court determination in January 2001 that the redesigned
pilot project does not violate the acts, the inspectors? union appealed in
February 2001. That appeal is still pending. A detailed description of the
litigation surrounding the pilot project is contained in appendix II.
Figure 1 depicts inspection responsibilities under FSIS? traditional and
modified inspections on a chicken slaughter line.
Page 8 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Figure 1: Traditional and Modified Inspection Systems on a Chicken Slaughter
Line
Note: Under the traditional inspection system, the slaughter line?s speed at
most chicken plants is 91 birds per minute. Each inspector is responsible
for about 35 birds per minute. Therefore, three FSIS inspectors are required
to inspect carcasses on a slaughter line that runs at a speed of 91 birds
per minute. The figure above shows only one FSIS inspector?s station in
detail. Under the pilot project, a plant employee (sorter) replaces the FSIS
inspector who examined the carcasses to determine their condition.
As figure 1 shows, under traditional inspection systems, three FSIS
inspectors are present at fixed locations on the slaughter line. In
contrast, at chicken plants that participate in the pilot project, one FSIS
inspector is present at a fixed location on each slaughter line, while a
verification inspector monitors the entire line. The verification inspector
is free to move along the slaughter line to continuously observe and
evaluate the plant?s implementation of its HACCP system and process
controls. This inspector is also responsible for randomly selecting and
examining 80 carcasses per line per shift to verify that the plant is
complying with the performance standards for food safety and quality. A
slaughter line
Page 9 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
averaging 91 chickens per minute would process over 43,000 chicken carcasses
during each shift. The carcass inspector is located at the end of the
slaughter line, immediately before carcasses enter the chiller. This
inspector visually examines each carcass to comply with a court mandate that
each carcass should receive FSIS inspection. At participating hog plants,
carcass inspectors are positioned at the head, viscera, and final carcass
inspection stations. Most of the inspectors and some veterinarians at the
pilot project plants received promotions.
The pilot project?s hypothesis was that a prevention- oriented inspection
system that uses plant personnel to examine each carcass can provide a level
of product safety and quality equal to or better than traditional
inspections. At selected chicken and hog plants that voluntarily agreed to
participate in the project, plant personnel, instead of FSIS inspectors,
initially determine which carcasses and parts are unacceptable and should be
removed from the slaughter line. Under this project, FSIS is able to use
fewer inspection personnel at each plant. FSIS used an independent
contractor to collect and analyze organoleptic and microbial data under the
traditional inspection system first and then under the modified inspection
system. FSIS developed organoleptic performance standards to measure how
well the inspection system at participating plants performed after shifting
from traditional to modified inspections. In addition, as part of their
duties, FSIS inspectors continued to collect organoleptic performance data
at these plants. Recently, both Australia and Canada tested and adopted
modified inspection programs that resemble USDA?s pilot project. Pilot
Project Makes
Industry More Responsible and Accountable for Product Safety and Quality,
While Continuing Government Monitoring With Fewer Resources
Page 10 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
To initiate the pilot project, FSIS invited meat and poultry plants that
slaughter young, healthy animals- market hogs, fed cattle, and young poultry
(including young turkeys)- to volunteer to participate in the project. 1
Sixteen chicken, 5 hog, and 5 turkey plants initially volunteered to
participate. 2 No cattle plants volunteered. In general, plant managers that
volunteered to participate in the pilot said that they elected to do so
because they believe they can enhance product safety and quality if they
have better control of slaughter line operations and, at the same time,
increase productivity.
FSIS contracted with the Research Triangle Institute 3 (RTI) to collect and
analyze organoleptic and microbial data at plants that volunteered to
participate in the pilot project. The purpose of this data collection is to
evaluate whether the inspection systems at plants operating under modified
inspections perform at least as well as the traditional inspection systems.
In August 2000, RTI completed its collection of organoleptic and microbial
data at 16 chicken, 5 hog, and 5 turkey plants under the traditional
inspection. These baseline data documented the accomplishments of the
traditional inspections system, and FSIS decided to adopt the results as
performance standards for measuring the accomplishments of the same plants?
inspection systems after they shifted to modified inspections. After a
transition period during which plant personnel practiced their new roles,
RTI again collected organoleptic and microbial data under the modified
inspection system to provide a beforeand- after comparison. 4
After RTI collected data on the performance of the traditional inspection
system at the volunteer plants (baseline data), 5 of the 16 chicken plants
1 FSIS designed the pilot project for meat and poultry plants that slaughter
young animals because this group comprises approximately 90 percent of the
animals slaughtered and those carcasses do not have complex pathology or
other problems associated with older animals.
2 Originally, 17 chicken plants volunteered to participate, but 1 of the
chicken plants elected to drop out of the project shortly thereafter. 3 The
Research Triangle Institute is an independent nonprofit organization that
conducts multidisciplinary research on issues including health and
pharmaceuticals, environment, and education and training. RTI?s contract
with USDA for this pilot project contains modifications for increased future
work.
4 After FSIS placed at least one carcass inspector back on the slaughter
line, RTI had to return to collect a new set of food safety and quality
data. Under Pilot, Industry Is
More Responsible and Accountable for Product Safety and Quality, While FSIS
Provides Inspection Using Fewer Government Resources
Page 11 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
and 2 of the 5 hog plants dropped out of the project. 5 As a result, RTI has
collected modified inspections data at 11 chicken and 3 hog plants. These
data were collected over a 5- to 6- week period. Some managers from plants
that dropped out cited uncertainty about the future of the pilot project
because of the court?s action, which resulted in modifications to the
pilot?s original design.
Ten additional chicken plants have since volunteered to participate in the
pilot, bringing the total number of participating plants to 21, but RTI will
collect data at only 5 of the additional 10 plants. According to FSIS, no
traditional inspection performance data (baseline data) will be collected at
these additional plants. Table 1 shows the number of slaughter plants that
participated both in the baseline data collection phase (traditional
inspections) and in data collection after they switched to modified
inspections.
Table 1: Number and Types of Pilot Plants That Participated in Traditional
and Modified Inspections
Species Plants that completed baseline data
collection under traditional inspections
Plants that completed data collection under modified
inspections
Chickens 16 11 (10) a Hogs 5 3 Turkeys 5 0 a One plant dropped out after it
completed data collection under the modified inspection system.
Source: USDA, FSIS.
Because turkey plants have not participated in modified inspections and
because data from hog plants under modified inspections are not yet
available, this report discusses pilot project results from chicken plants
only. (See app. III for information on the development of performance
standards for hogs.)
At the pilot project plants, FSIS has been able to use fewer inspection
resources than would otherwise be used under traditional inspection systems,
resulting in an overall 22 percent reduction (from 259 to 202 inspectors) in
the number of FSIS inspectors on the slaughter floor. At most of the 14
chicken and hog plants, the reduction of inspectors ranged
5 All five turkey plants dropped out after the baseline data were collected.
Page 12 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
from 1 and 10 inspectors, depending on the number of slaughter lines and the
speed of those lines. Under traditional inspections, there were three
inspectors per line on average, compared with two inspectors per line under
modified inspections. At three of the pilot project plants, the change in
the number of inspectors was not entirely due to the modified inspection
system.
To collect and analyze organoleptic data on the defects that the FSIS
inspectors had overlooked under traditional inspections at chicken plants,
RTI veterinarians at each volunteer plant examined approximately 2,000
randomly selected carcasses that had passed FSIS? slaughter inspection. RTI
veterinarians examined carcasses to record the same type of defects that
FSIS inspectors identify when they examine each carcass. These included
safety defects, such as visible fecal material and evidence of septicemia
and/ or toxemia, and other consumer protection defects (quality defects)
such as feathers, hairs, or bruises. FSIS grouped this information into two
food safety and five other consumer protection (quality) categories:
Food Safety 1: Infectious conditions such as septicemia and/ or toxemia.
Food Safety 2: Contamination with fecal material. Other Consumer
Protection 1: Animal diseases such as airsaculitis. Other Consumer
Protection 2: Defects such as bruises and sores. Other Consumer Protection
3: Contamination with digestive tract contents
(ingesta). Other Consumer Protection 4: Dressing defects such as feathers
and oil
glands. Other Consumer Protection 5: Dressing defects of the digestive
tract tissue
such as bursa and cloaca. FSIS then developed performance standards for each
of these seven categories. The quality performance standards were set at
position 12 (out of 16) on the basis of measurement of the traditional
inspection system at the 16 chicken plants. That is, 25 percent of the
plants? inspection systems (4 different plants) would have to improve on
their baseline results in each of the 7 performance categories. FSIS
officials explained that they decided to adopt the 12th position of the
baseline results as the standard to ensure that most participants would be
able to meet them. The officials stated that this represents a reasonable
tightening of the traditional system?s accomplishments. The Pilot?s Scope
Included
the Development of Safety and Quality Standards to Measure Performance of
Traditional Inspections as Compared With Modified Inspections
Page 13 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
To better protect consumers from defects that may be harmful, FSIS set the
two food safety standards at zero rather than adopting the results of the
12th position. Thus, a performance standard of zero for fecal contamination
means that FSIS expects all carcasses slaughtered during a shift to be free
of fecal material. Conversely, an 80- percent performance standard for
dressing defects such as feathers and oil glands means that if 80 of 100
carcasses have these defects, the performance standard is met. The
performance standards represent the percentage of carcasses per shift with
each defect that FSIS considers acceptable. Table 2 shows the traditional
inspection system?s baseline results for the 16 chicken plants.
Table 2: Traditional Inspection System?s Results (Baseline Data) From 16
Chicken Pilot Plants: Defects per Shift in Percents
Note: Ranking numbers represent different plants for different food safety
and OCP standards. Although the 12th position?s performance was 0.05 percent
for Food Safety 1 and 1.5 percent for Food Safety 2, FSIS adopted a zero
tolerance policy for these two standards. For Food Safety 2, the zero
tolerance standard was used because it is already a regulatory requirement.
Source: RTI.
RTI also measured the prevalence of generic E. coli and Salmonella under
traditional inspections and modified inspections at the 11 chicken plants to
determine if changes in the inspection system would have an effect on the
microbial profile of carcasses. RTI collected and analyzed 300 samples
Page 14 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
for both microorganisms over a 6- week period. RTI analyzed the
Salmonella samples to determine if they met FSIS? already existing
regulatory standard for Salmonella and the generic E. coli samples to
determine if the results were in the acceptable, marginal, and unacceptable
ranges established by FSIS.
FSIS is also collecting data on product safety and quality defects at
chicken plants in the pilot project and is analyzing these data to compare
how well traditional inspection systems perform versus modified inspection
systems. FSIS inspectors collect the data as they conduct scheduled and
unscheduled verification tests at each chicken plant. These tests evaluate,
among other things, how well plant employees detect carcass defects. When
the pilot project is concluded, FSIS plans to analyze these data to
determine if at least the same level of safety and quality is maintained
under the pilot program. FSIS has announced that preliminary results from
the chicken pilot project show that inspection systems perform better after
the plants shifted to modified inspections, and that it expects to propose
regulations to modify its traditional slaughter inspection system for all
chicken plants in 2002.
The Australian Quarantine and Inspection Service and the Canadian Food
Inspection Agency have tested and adopted alternative inspection systems
similar to that tested in the FSIS pilot project. (See apps. V and VI for
additional details.)
Australia has tested and adopted a modified inspection system for all meat
plants that produce products for domestic consumption. In addition,
Australia is now in the process of implementing the Meat Safety Enhancement
Program at plants that export meat. Under the program, plants are
responsible for plant process controls, including sanitation, the microbial
monitoring of pathogens, and detecting product defects. The program
incorporates the use of government- licensed company employees, acting under
the supervision and oversight of government veterinarians, to replace
government inspectors. In order to be licensed by the government, the
company?s employees are required to undertake about 600 hours of training
and pass an examination. The Australian Quarantine and Inspection Service
sets public health product standards and audits plant operations to ensure
that food safety standards are met. A key feature of the program is the
mandatory use of statistical process controls to track plants? performance
over time. Pilot Projects in Australia
and Canada Used a Similar Strategy
Page 15 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Canada has adopted a new poultry inspection system, known as the Modernized
Poultry Inspection Program that allows plant personnel to assume defect
detection duties and remove defective carcasses from the slaughter line
under the supervision of government inspectors. The program is being
implemented in a phased approach. Performance standards for Canadian poultry
plants participating in this program are similar but not identical to FSIS?
standards for chicken plants in the U. S. pilot project. Like the Australian
program, the Canadian program has stringent training requirements for
industry personnel replacing inspectors. Industry personnel are required to
demonstrate competency through testing and certification.
The pilot project has several design and methodology limitations that
compromise the overall validity and reliability of its results. For example,
the lack of a control group prevents valid comparisons between the
inspection system of participating plants and that of nonparticipating
plants. In addition, the participating plants were not randomly selected;
therefore, results from these plants cannot be generalized to the entire
population. Finally, the pilot project does not appropriately explain how
variables, such as seasonal changes and plant modifications, could affect
the project?s results. In addition, the pilot project does not include
important features of similar programs in Australia and Canada.
FSIS officials characterized the design of the pilot project as a classic
before- and- after experiment that is typical of other FSIS experiments. A
classic before- and- after experiment involves both a control and an
experimental group and thus provides a basis for comparing the results of
the two groups. Ideally, the groups are as similar as possible, except for
the variable that the experiment is attempting to measure. However, because
this level of rigor is rarely achievable in applied settings,
quasiexperimental designs are often used in which comparison groups are
established to compare the impact of a new program with the status quo.
According to our discussion with FSIS officials, the volunteer plants were
not separated into a control group and an experimental group because the
agency felt that it would have been unfair to exclude volunteer plants from
taking part in the modified inspection aspect of the experiment.
Nevertheless, under a quasi- experimental design, FSIS could have used as a
control group the five chicken plants that decided to discontinue
participation in the pilot project after their baseline performance under
the traditional system was measured. The agency could have then compared the
results from these plants with the results obtained from the Design and
Methodology Limitations Compromise Overall Validity and Reliability of Pilot
Project Results
Pilot Project Lacks a Control Group
Page 16 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
plants that participated in the modified inspections. Lacking such a
comparison, it is difficult to determine how the inspection systems at the
plants that did not participate in the pilot project would have performed
under the performance standards established for the pilot project.
The managers of two large chicken plants that dropped out of the pilot said
that they proposed another approach to FSIS. Namely, these managers wanted
to conduct what could have been a controlled experiment within their own
plant. Both of these plants have multiple slaughter lines, and the managers
wanted to operate half of their slaughter lines under traditional
inspections and the other half under modified inspections. However, FSIS
would not authorize such an arrangement because the agency wanted the entire
plant to be under one inspection system. Officials said that it would have
been too difficult to maintain two separate inspection systems in the same
plant. As a result, both of these plants withdrew from the pilot project.
The Australian pilot project compared three plants that volunteered to
participate in the pilot project with three plants operating under
traditional inspections, so the pilot had both an experimental group and a
control group. The plants in the experimental group removed the government
inspectors from the slaughter line and replaced them with plant personnel,
while the control group plants made no changes to their inspection system.
The pilot project measured both microbial and organoleptic data at plants in
both groups to determine how well they complied with standards. The results
were then compared in order to determine how the modified inspections
affected food safety and quality. The Canadian Food Inspection Agency did
not use a control group.
Because the volunteer chicken plants were not randomly selected, the results
cannot be generalized to the population of plants that slaughter young
chickens. FSIS is not authorized to compel plants to participate in any
pilot project; therefore, it had to rely on volunteer plants instead of
randomly selecting participants. However, FSIS did have the option to
randomly select a subgroup to serve as a control group from the volunteer
plants, but decided not to pursue that option. Australia and Canada also did
not select plants on a random basis.
FSIS officials acknowledge that the number of plants in the pilot was not
statistically derived. The officials believe, however, that the 16 chicken
plants that originally volunteered constitute a sufficient number to
estimate results for the population. However, only 11 chicken plants have
Plants Participating Are
Not Randomly Selected and Do Not Represent the Population
Page 17 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
participated in all phases of the pilot project. These plants represent
about 6 percent of the population of young chicken plants nationwide, or 11
of 186 plants in 1999. We estimate that 125 randomly selected plants would
need to participate in the project in order for FSIS to obtain results that
can be generalized to the population of chicken plants. 6
As a result of the pilot project?s reliance on volunteer plants, the results
of the pilot do not provide sufficient geographic representation. That is,
the 11 chicken plants that have participated in all phases of the pilot
project are located in 7 of 35 states that produce young chickens, or about
20 percent of the chicken- producing states. These plants are concentrated
in the southern region of the United States and account for about 57 percent
of U. S. young chicken production. (See fig. 2.) Unlike FSIS, the Canadian
Food Inspection Agency designed its pilot to include chicken plants from the
western, eastern, and central provinces. As a result, the Canadian pilot
project has better geographic representation.
6 This sample size is calculated to provide for results of plus or minus 5
percent at the 95- percent confidence interval.
Page 18 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Figure 2: Location of Chicken Pilot Plants
Note: Numbers indicate how many chicken plants participated in the baseline
data collection phase of the pilot project.
Source: USDA, FSIS.
In addition, there are eight large plants, three small plants, and no very
small plants in the pilot project. According to FSIS, there are 112 large,
61 small, and 13 very small plants that slaughter young chickens in the
United States. We discussed with officials at all participating plants in
the pilot project whether the plants represent the population of chicken
plants in the United States. Most of them said that, in their opinion, the
plants are representative of the industry nationwide. FSIS officials are
also of the opinion that all plants that volunteered to participate are
similar to nonparticipants in all respects except for the fact that they
volunteered. However, since only volunteer plants participated in the pilot,
potential selection bias is introduced because such plants may be
predisposed to better performance.
Page 19 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
The pilot project does not adequately control for the effect that seasonal
variations and plant modifications can have on the its results. Because
seasonal changes can cause animals to develop diseases and/ or poor health,
two poultry science experts that we consulted said that it is imperative to
collect samples over sufficiently long periods of time to cover all seasons.
Seasonal variations significantly affect microbial results, especially in
the case of bacteria such as Salmonella, which was measured in the pilot
project. However, the pilot project?s design called for 6 consecutive weeks
of microbial data collection. Accordingly, the experts we consulted believe
that microbial data collected during the pilot project are of limited value
because the data cover less than 2 months at each plant.
The pilot project?s results may also be affected by another set of
variables. Most of the plant managers at pilot project plants told us that
they introduced changes to their slaughter lines to improve product safety
and quality after the baseline performance of the traditional system was
measured. These changes included the addition of new brushes and scrubbers,
plain water washes, and antimicrobial washes. Some managers also modified
their evisceration equipment and developed computerized systems to track
carcass safety and quality defects. All the inspectors and veterinarians at
pilot project plants that we surveyed indicated that at least one change had
been made after modified inspections were implemented at the plant. In
effect, it is not possible to discern whether the before- andafter results
measured by the two food safety and five quality standards are to be solely
attributed to the shift from traditional to modified inspections at these
plants.
According to FSIS officials, the pilot project was designed to measure the
accomplishments of the traditional inspection system and the accomplishments
of the models inspection system allowing for measuring multiple process
changes- not isolated comparisons between FSIS inspector performance and
plant employee performance. If this is the case, then the pilot project
should have included the measurement of all the process changes introduced,
not just the only change introduced by the agency; namely, the transfer of
carcass defect detection from FSIS inspectors to plant personnel. We
understand that, at their initiative, plants decided to make changes to
their process during this pilot. However, FSIS did not collect data on what
changes were made or on what the effect of those changes may be on the
overall pilot project results.
USDA officials disagree that the pilot project does not take into account
variables that could affect its results. They stated that, if the results
from Pilot Project?s
Methodology Does Not Take Into Account Variables That Could Affect Results
Page 20 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
all plants are viewed collectively, their sample extended over a sufficient
period of time to consider the effect of seasonal variations, evisceration
line effects, and other non- quantifiable variables. Even if all plants are
considered collectively, FSIS is still missing microbial data for the entire
year. According to officials, from January through April, no data were
collected at any of the pilot plants. Moreover, according to the experts we
consulted, FSIS should have a year?s worth of microbial data from each plant
in the pilot to be able to account for seasonal variations.
In discussing these limitations with USDA officials, they explained that the
focus of the project is studying an entire system, including all changes
made by the plants, and not simply comparing the work of company employees
with that of government inspectors. However, we note that the project?s
stated objective was to determine whether a modified inspection system is at
least as good as the traditional system. In order for such a comparison to
be valid, USDA would have to determine that other variables introduced into
the slaughter line, such as the addition of rinses or brushes, are not the
cause of changes noted after the shift in inspection systems took place.
The pilot project did not address three important features of similar
programs in Australia and Canada. Those features would help to ensure that
plant personnel are able to discharge their carcass defect detection duties
as well as federal inspectors and that plants are capable of continuing to
ensure the safety and quality of their products under a modified inspection
system. First, the pilot project did not require that, prior to assuming
their new duties, plant personnel receive any training and pass competency
tests. Second, the pilot project did not require the use of statistical
process controls as a means to ensure that plants continually improve their
processes under the modified inspection system. Third, the plants continued
in the pilot project irrespective of repetitive noncompliance with
regulatory requirements.
FSIS did not require that plant employees complete training before assuming
the carcass defect detection duties that FSIS inspectors? performed.
Furthermore, FSIS did not establish a way to measure plant employee?s
knowledge and competence. FSIS officials told us that training for plant
personnel will not be required if the system is adopted nationwide. Plant
personnel and FSIS inspectors at each pilot project plant did participate,
however, in a transition phase during which they practiced their new roles.
FSIS officials told us that the transition phase was also Pilot Project
Lacks Key
Elements Project Does Not Require Formal Employee Training and Certification
Page 21 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
used to explain how and when data were to be collected during the project.
If FSIS implements modified inspections nationwide, plant personnel will be
assuming carcass defect detection duties previously performed by FSIS
inspectors who have received formal training on how to identify carcass
defects and determine whether they are suitable for human consumption.
However, unless FSIS institutes training requirements, plant personnel will
not have received comparable training. We asked plant managers participating
in the pilot project about the training that they have provided for plant
employees. They replied that they provided various levels of training.
Individual plants have sole discretion for determining the type of training
provided for their employees as FSIS does not approve or monitor plants?
training programs. According to plant officials, FSIS is supportive but does
not endorse a training program that the International HACCP Alliance
developed for plant personnel participating in this pilot project. 7 In some
instances, the Alliance, not FSIS, has accredited plants? training programs
but it does not certify trainees who complete these courses.
Seven of the 14 pilot project managers whom we interviewed support a
training program for plant personnel. About two- thirds of the managers also
were in favor of developing a joint training program between industry and
FSIS. At nine pilot project plants, plant employees were required to pass a
test before being assigned to their new jobs, and seven plants adopted some
kind of continuous on- the- job training. The managers told us that
classroom training for plant employees ranged from 4 to 80 hours. The
duration of on- the- job training ranged from 8 to 160 hours.
In contrast, the Australian modified inspection program requires plant
personnel to complete about 600 hours of classroom training as a
prerequisite to participation in the modified inspection program. Plant
personnel must also pass a competency test and be certified. Upon completion
of the on- the- job training, state authorities license the plant personnel.
The Australian government?s veterinarian- in- charge at each plant is
authorized to withdraw or deny license renewal of plant personnel on the
basis of performance.
7 The International HACCP Alliance membership includes over 120 members from
nine countries representing industry associations, professional
associations, educational foundations, universities, third party/ private
companies, and government cooperators. Their mission is to promote
international public health and safety by facilitating uniform development
and implementation of HACCP programs from farm to table.
Page 22 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Similarly, under Canada?s modified inspection program, government inspectors
train plant personnel. The required training includes standardized classroom
training, in- plant training, a final examination, and a demonstration of
competency on verification tests and finished product standards. At the
completion of the training, the Canadian Food Inspection Agency accredits
plant employees. Plant employees must be reaccredited every 6 months, and
the agency?s inspectors reserve the right to conduct additional
reaccreditation of plant employees at any time.
Sixty- seven percent of the FSIS inspectors and veterinarians whom we
surveyed believe FSIS should develop a training program that includes
mandatory testing and certification of plant employees.
Although FSIS has encouraged plants in the pilot project to use statistical
process controls, it is not a requirement of the pilot. However,
threequarters of the inspectors and veterinarians responding to our survey
indicated that their plants use statistical process controls. About a third
of these respondents believe that the use of statistical process controls
have the effect of making food safety and quality better, while a quarter of
them believe that it does not make a difference. At the pilot project
plants, FSIS inspectors do not review or analyze plant data using
statistical process controls. However, by applying process controls, FSIS
could obtain a more comprehensive assessment of the effectiveness of the
modified inspection system because it could review the performance of an
individual plant?s data to determine whether the plant?s systems are
continually improving over time.
In contrast, the Australian and Canadian food inspection agencies use
statistical process control methods to audit a plant?s performance over
time. This method entails verification by the plant?s management that its
production process is stable, that it is capable of producing products that
meet performance standards, that it takes appropriate actions if changes
occur in the process before it results in unacceptable products, and that it
takes actions to continuously improve its process. Project Does Not Require
Use
of Statistical Process Controls
Page 23 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
FSIS allowed all plants to continue participating in the pilot project even
though several had multiple instances of noncompliance with regulatory
requirements for fecal contamination. 8 Our analysis of noncompliance
records issued by FSIS inspectors at 11 chicken pilot plants showed an
increased number of such records after the plants shifted to modified
inspections for fecal noncompliance. In two instances, however, FSIS took
additional action and notified plants of problems with repeated
noncompliance. FSIS officials told us that they are now considering how many
repeated instances of noncompliance will result in further regulatory
action.
Figure 3: Fecal Noncompliance Records Under Traditional and Modified
Inspections
Note: Data are provided for a 12- month period before and after the pilot
project?s implementation. However, at some plants, data reported after the
implementation of the pilot project covered periods of less than 12 months,
data for plant 7 cover 8 months, data for plants 1 and 10 cover 10 months,
and data for plant 9 cover 11 months. It is likely that a full twelve months
of data for these plants would result in an even higher number of
noncompliance records.
Source: USDA, FSIS.
As figure 3 shows, the number of noncompliance records resulting from fecal
material increased significantly at several plants after they shifted from
traditional inspections to modified inspections. At one plant, FSIS
8 FSIS inspectors document each instance of noncompliance with regulatory
requirements in a noncompliance record that plant managers must address
immediately. If this is done, the plant continues to operate without
interruption. When deficiencies occur repeatedly, FSIS may take further
action to withhold products or to suspend inspections. If FSIS suspends
inspections, the plant cannot operate. Project Does Not Fully Take
Into Account Plants? Repeated Instances of Noncompliance With Regulatory
Requirements
Page 24 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
inspectors issued 469 fecal material noncompliance records during a 10-
month period after the plant shifted to modified inspections. Many
noncompliance records reflect multiple violations recorded within the same
record. At one plant, FSIS inspectors reported that over 90 percent of the
fecal material noncompliance records reflected repetitive failures.
FSIS officials acknowledge the rise in the number of noncompliance records
and attribute it to the fact that at pilot project plants, FSIS inspectors
have more time to verify compliance with regulatory requirements and to
document instances of noncompliance when they occur. The officials also
point out that, as part of the their verification duties, FSIS inspectors
now examine 80 carcasses per line per shift compared with 20 carcasses under
the traditional inspection system. 9 In addition, unscheduled verification
tests of additional carcasses are conducted under each inspection system.
Officials also told us that staffing shortages existed under traditional
inspections and that in some instances, it was not possible to complete
verification activities on 20 carcasses for each line. Even accounting for
the fourfold increase in the number of verification checks taking place
under the modified inspections system, one plant experienced a more than
twentyfold increase in the number of noncompliance records for fecal
contamination. At other plants, however, the number of noncompliance records
did not even double.
Another factor may also account for the increases in noncompliance records
issued for fecal failures. Although inspectors and veterinarians working at
pilot project plants who responded to our survey were not comparing line
speeds to those of the traditional inspection system, about 60 percent of
them think that line speeds are too fast under modified inspections to
ensure product safety. For example, one inspector stated that fast line
speeds cause inexperienced employees to miss carcasses with visible fecal
contamination. FSIS does not regulate line speeds under this pilot. However,
under traditional inspections, each inspector is responsible for about 35
carcasses per inspector per minute. Canadian inspectors told us that line
speeds need to be regulated under modified inspection systems.
9 Each carcass is inspected under both traditional and modified inspections.
In addition, under both systems, inspectors also select a sample of
carcasses at the end of each line to verify that the inspection system is
working as intended.
Page 25 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
In contrast to FSIS, the Canadian Food Inspection Agency decided to proceed
with a phased approach to nationwide implementation of Canada?s modified
inspection program, the Modernized Poultry Inspection Program. Under this
approach, poultry plants must pass a preliminary assessment before being
accepted into the program and are required to pass additional audits as they
move from one phase of the program to the next. Producers that choose not to
participate in or do not qualify for a partially modified or fully modified
inspection system are inspected under the traditional system.
Notwithstanding the design and methodology limitations, we found that data
from the chicken pilot that RTI and FSIS inspectors collected show somewhat
different results and do not provide a conclusive basis for FSIS to make a
decision regarding the merits of one inspection system compared with
another. However, 71 percent of the FSIS inspectors and veterinarians that
we surveyed believe that product safety is equal to or somewhat better under
modified inspections. Some of them commented that under the modified system,
they are able to oversee the entire slaughter line and have more time to
collect carcass samples for detailed examination. However, they also raised
concerns about the modified inspection system.
RTI?s data do not conclusively show that inspection systems at chicken
plants perform as well as or better than they did after they switched to
modified inspections. Under modified inspections, none of the inspection
system at these plants met all of the performance standards that FSIS
developed to measure their performance. Table 3 shows the performance
standards that the inspection systems at these plants were expected to meet
after shifting to modified inspections. 10
10 These standards, set by data gathered at the 16 plants under the
traditional inspection system, reflect the performance level of the 12th
position. (See p. 13.) Pilot Project Results
Are Inconclusive, but FSIS Inspection Personnel Generally Support Modified
Inspections
Contractor?s Data Do Not Conclusively Show That Modified Inspection Systems
at Chicken Pilot Plants Performed ?As Well As or Better Than? Traditional
Inspection Systems
Page 26 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Table 3: Performance Standards for Inspection Systems at Chicken Plants in
the Pilot Project
Categories Performance standards
(percentage of defects allowed per shift)
Food Safety (FS) 1: Infectious condition such as septicemia and toxemia 0
FS2: Digestive content contamination (visible fecal material) 0 Other
Consumer Protection (OCP) 1: Animal diseases such as airsaculitis 1.7 OCP2:
Miscellaneous such as bruises, sores, etc. 52.5 OCP3: Digestive content
contamination (ingesta) 18.6 OCP4: Dressing defects such as feathers and oil
glands 80.0 OCP5: Dressing defects of digestive tract such as bursa and
cloaca 20.8
Source: USDA, FSIS.
Data from the inspection systems at the 11 chicken plants whose performance
RTI measured under both inspection systems show that, while the majority of
the systems met four or more of the performance standards for food safety
and quality under modified inspections, none met all seven of the standards.
11 The data show that 10 of the 11 plants? inspection systems met the food
safety standard for septicemia and/ or toxemia, but only 1 of the 11 systems
met the zero- tolerance food safety standard for visible fecal material,
which could contain harmful bacteria such as E. coli. Most plants?
inspection systems, however, were also unable to meet the fecal material
standard under the traditional inspection system. All 11 systems at these
plants met the quality standard for bruises and sores (OCP2), but only 3 met
the quality standard relating to the presence of feathers (OCP4). According
to FSIS officials, carcasses with product quality defects are of a lesser
concern than those with food safety defects because they are not associated
with foodborne illness and quality defects can generally be removed from the
carcass after it leaves the slaughter line and moves to the cut- up and
processing area.
11 We analyzed the statistical significance of inspection system changes at
each plant using the 95- percent confidence level.
Page 27 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Three of the 11 plants? inspection systems met six of the seven standards, 3
plants met five of the standards, 4 plants met four standards, and 1 plant
met two standards. Table 4 shows which systems met the performance standards
that FSIS set for the pilot project.
Table 4: Eleven Chicken Plants? Inspection Systems Compliance With
Performance Standards (RTI?s Data) Food safety standards Food quality
standards (in percent)
Plant FS1:
septicemia and/ or toxemia (zero allowed)
FS2: visible fecal material (zero allowed)
OCP1: animal disease, e. g., airsaculitis (1.7 allowed)
OCP2: bruises, sores, etc. (52.5 allowed) OCP3: ingesta
(18.6 allowed) OCP4:
feathers, oil glands, etc. (80.0 allowed)
OCP5: bursa, cloaca, etc. (20.8 allowed)
1 Yes No Yes Yes Yes No Yes 2 Yes No Yes Yes Yes No No 3 Yes No Yes Yes Yes
No No 4 Yes No Yes Yes Yes Yes Yes 5 Yes No Yes Yes Yes Yes Yes 6 Yes No Yes
Yes Yes No No 7 Yes No Yes Yes Yes No No 8 Yes Yes Yes Yes Yes No Yes
9NoNoYesYesNo No No 10 Yes No Yes Yes Yes No Yes 11 Yes No Yes Yes No Yes
Yes
Plants that met standard
10111 11 936
Source: RTI.
As discussed earlier, under its contract with FSIS, RTI collected microbial
data as well. Specifically, it analyzed samples for the presence of
Salmonella and generic E. coli. The results show that 9 of the 11 chicken
plants? inspection systems met the performance standard for Salmonella.
Also, about 80 percent of the samples collected at the11 plants were within
the acceptable ranges that FSIS specifies for generic E. coli.
Irrespective of whether plants? inspection systems met the performance
standards, we analyzed these data to determine if, under modified
inspections, the systems performed at least as well as they did under
traditional inspections. Table 5 summarizes the results of this analysis.
Page 28 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Table 5: Eleven Chicken Plants? Inspection Systems Performance After
Shifting to Modified Inspections (RTI?s Data) Microbial standards Food
safety standards Food quality standards
Performance Changes Generic
E. coli Salmonella
FS1: septicemia
and/ or toxemia
FS2: visible
fecal material
OCP1: animal disease, e. g.
airsaculitis OCP2:
bruises, sores, etc. OCP3:
ingesta OCP4:
feathers, oil glands,
etc. OCP5:
bursa, cloaca
, etc.
Better 2 2 1 4 5 5 4 3 4 Same 9 4 10 7 5 3 1 1 1 Worse 0 5 0 0 1 3 6 7 6
Total 11 11 11 11 11 11 11 11 11
Source: GAO?s analysis of RTI?s data.
The data show that some of the plants? inspection systems improved or
remained unchanged in certain categories such as the food safety standard
for septicemia and/ or toxemia. On the other hand, the systems? ability to
meet the quality performance standards deteriorated over time. In general,
most measures of performance showed deterioration in one or more plants. In
fact, every inspection system showed deterioration for at least one of the
nine standards measured. At 5 of the 11 plants, results for
Salmonella were worse under modified inspections than they were under the
traditional inspection system- only 2 improved their performance in this
category. FSIS has stated that any new inspection system should perform at
least as well as the current system of inspections.
FSIS? data for 10 chicken plants in the pilot project also show that none of
their inspection systems met all seven standards. 12 In contrast to the RTI
data, however, the FSIS data show that 7 of the 10 systems met six of seven
standards, and the remaining 3 plants met five of the seven standards. FSIS
officials stated that the differences in results might be caused, at least
in part, by differences in data collection. As part of FSIS inspectors?
verification duties at the chicken plants participating in the pilot
project, the inspectors sample and analyze carcasses to determine whether
they meet performance standards. Inspectors are not required to collect and
analyze microbial samples, as RTI was required to do for the pilot project.
The data collected cover a longer period than RTI?s and include much larger
samples per plant. The FSIS officials acknowledge,
12 We analyzed FSIS data from 10 of the 11 plants at which RTI collected
data for both traditional and modified inspections because one plant dropped
out of the pilot shortly after RTI completed data collection. FSIS? Data Do
Not
Conclusively Show That Plants? Inspection Systems Perform ?As Well As or
Better? Under Modified Inspections
Page 29 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
however, that disparities in the FSIS and RTI data for some performance
standards, such as dressing defects (e. g., feathers and oil glands),
suggest a need for further investigation.
The FSIS data show that 7 of the 10 chicken plants? inspection systems met
six of the seven performance standards, but none of the plants met the zero-
tolerance standard for fecal material over time. It should be noted,
however, that the actual performance of the 12th position for this standard
was 1.5 percent, not zero as required by FSIS? performance standard. So
noncompliance with this standard may not provide a basis for comparing
performance of the modified with traditional inspection systems. After
shifting to modified inspections, each of the 10 plants? systems had reached
a performance that was better than 1.5 percent for this standard. We
analyzed data on individual inspection systems? performance from the time
that each plant implemented modified inspections through June of 2001. Table
6 shows a summary of the inspection systems? performance at individual
plants.
Table 6: Ten Chicken Plants? Inspection Systems Compliance With Performance
Standards (FSIS? Data) Food safety standards Food quality standards
Plant FS1:
septicemia and/ or toxemia FS2: visible
fecal material OCP1: animal
disease, e. g., airsaculitis)
OCP2: bruises, sores, etc. OCP3:
ingesta OCP4:
feathers, oil glands, etc. OCP5: bursa,
cloaca, etc.
1 Yes No Yes Yes Yes Yes Yes 2 Yes No Yes Yes No Yes Yes 3 Yes No Yes Yes
Yes Yes Yes 4 Yes No Yes Yes Yes Yes Yes 5 Yes No Yes Yes Yes Yes Yes 6 No
No Yes Yes Yes Yes Yes 7 Yes No Yes Yes Yes Yes Yes 8 Yes No No Yes Yes Yes
Yes 9 Yes No Yes Yes Yes Yes Yes 10 Yes No Yes Yes Yes Yes Yes
Plants that met standard
9 0 9 10 9 10 10
Note: FSIS inspectors do not collect and analyze generic E. coli and
Salmonella as part of their verification duties in this pilot project.
Source: FSIS inspectors? verification data from November 2000 through June
2001.
Page 30 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Table 6 shows which plants? inspection systems did not meet specific
performance standards. That is, plant 6 did not meet the food safety
standard for infectious conditions (e. g., septicemia and/ or toxemia);
plant 8 did not meet the food quality standard for animal diseases such as
airsaculitis; and plant 2 did not meet the food quality standard for
digestive content contamination (ingesta). At the pilot plants, FSIS
inspectors are issuing noncompliance records when plants do not meet the
performance standards; however, FSIS officials have not yet decided how many
instances of noncompliance will be tolerated before the agency can decide to
take further action to ensure regulatory compliance.
We reviewed data collected by FSIS inspectors after the plants shifted to
modified inspections during two different time periods. In comparing these
two periods, we found that in several categories, the inspection system?s
performance at each plant deteriorated over time. Table 7 summarizes the
changes for these 10 plants.
Table 7: Changes in Chicken Plants? Inspection Systems Performance Over Two
Time Periods After Shifting to Modified Inspections (FSIS? Data)
Food safety standards Food quality standards Performance
FS1: septicemia and/ or toxemia FS2: visible
fecal material OCP1:
animal disease, e. g.,
airsaculitis OCP2:
bruises, sores, etc. OCP3:
ingesta OCP4:
feathers, oil glands, etc.
OCP5: bursa, cloaca, etc.
Better 0 1 1 5 2 05 Same 1 0 9 9 0 2 12 Worse 0 0 0 5 6 93
Total 10 10 10 10 10 10 10
Note: During the first period, the plants were implementing the original
pilot project design in which there were no FSIS inspectors stationed at
fixed positions on the slaughter lines. During the second, period, a carcass
inspector was reinstated at each slaughter line to comply with a court
ruling. The first period starts just after each plant shifted from
traditional to modified inspections and ends on October 31, 2000. The second
period starts November 1, 2000, and ends on June 30, 2001.
Source: GAO?s analysis of FSIS inspectors? verification data from
implementation of modified inspections until June 2001. We used a difference
of plus or minus 0.5 percent to analyze the changes for the seven categories
described in the table. We estimated that, on average, 43,000 chickens are
processed per line, per shift.
On the basis of this information, in addition to providing plant managers
with feedback on a daily basis, FSIS inspectors could identify plants whose
inspection systems need to improve. In addition, FSIS could aggregate data
from these plants and use this type of analysis to evaluate whether a
specific performance standard needs to be revised. For example, we made two
observations by analyzing the data as follows:
Page 31 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
As shown in table 7, 9 of the 10 plants? performance under the OCP4
standard deteriorated over time. While this standard permits 80 percent of
carcasses per shift to have this defect and still meet the standard, this
trend may suggest the need for FSIS to investigate why the inspection
system?s performance at an individual plant is deteriorating.
As shown in table 7, 9 of the 10 plants maintained their performance and
one improved under the FS2 standard.
Australian officials told us that under their modified inspection system,
the Australian Quarantine and Inspection Service requires plants to use
process controls to demonstrate continuous improvement in plant sanitation,
microbial pathogen reduction, and quality defect reduction. The agency uses
these data to monitor plant performance and enforce compliance with
standards.
About 70 percent of the respondents to our survey of FSIS inspectors and
veterinarians believe that modified inspections are equal to or somewhat
better than traditional inspections for product safety while more than half
believe that modified inspections are the same as or better than traditional
inspections for product quality. Table 8 summarizes inspectors? and
veterinarians? responses to major topics covered by our survey. (See app. IV
for further detail on the survey responses.)
Table 8: Summary of Responses to GAO?s Survey of USDA Inspectors and
Veterinarians
Topic Responses in percentages
Product safety 71: same as or better under modified inspections versus
traditional inspections 24: better under traditional inspections 5: unsure
Product quality 57: same as or better under modified inspections versus
traditional inspections 39: better under traditional inspections
5: unsure Effect of plants? adding anti- microbial rinses and washes a
62: food safety the same as or better than without rinses and washes
3: food safety worse than without rinses and washes 10: don?t know its
effect on food safety Line speeds: Concerning product safety 59: line speed
is too fast
39: line speed is about right 0: line speed is too slow 2: unable to
determine
Most FSIS Inspectors and Veterinarians Believe That Modified Inspections Are
Equal to or Better Than Traditional Inspections for Ensuring Product Safety
Page 32 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Topic Responses in percentages
Concerning product quality 64: line speed is too fast
33: line speed is about right 0: line speed is too slow 2: unable to
determine Original modified inspection system versus modified system after
court order: Concerning product safety
Concerning product quality
57: modified system after court order is better 27: original modified system
is better
8: they are about the same 8: no opinion on which is better
45: modified system after court order is better 25: original modified system
is better 19: they are about the same 11: no opinion on which is better.
Noncompliance records 76: increased under the modified inspections versus
traditional inspections b 11: decreased under the modified inspections
versus traditional inspections 10: the same under the modified inspections
and traditional inspections
3: don?t know Training of plant personnel: Concerning adequacy of plant
sorter training
Concerning USDA?s development of standardized training curriculum for plant
personnel
Concerning USDA?s development of a testing and certification program for
plant personnel
15: plant?s training helped to a great extent 51: plant?s training helped to
some extent 28: plant?s training helped to little or no extent
6: unable to determine 67: FSIS should develop curriculum 17: FSIS should
not develop curriculum 15: neutral
2: unable to determine 64: FSIS should develop this program 18: FSIS should
not develop this program 16: neutral
1: unable to determine. FSIS training for pilot project inspectors 40:
helped to a great extent
51: helped to some extent 8: helped to little or no extent 1: have not
received training for the pilot project 0: unable to determine. a This
change was experienced by 75 percent of respondents at their plants after
modified inspections
were implemented. b Some inspectors cited more time and freedom to find
defects and write noncompliance records and
more carcasses are sampled under modified inspections. Note: Because of
rounding, numbers may not add up to 100 percent.
Page 33 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Nearly all of the inspectors and veterinarians provided additional written
comments on several issues of concern and made suggestions for improvement.
The inspectors commented that under the modified system, they are able to
oversee the entire slaughter line and have more time to collect carcass
samples for detailed examination. In addition, 27 of the 210 inspectors and
veterinarians commented that they were concerned about regulatory
noncompliance and enforcement issues. Particularly noteworthy are comments
that indicate that stronger measures should be in place to reduce repetitive
instances of fecal material noncompliance and that inspectors should have
more authority to hold a plant accountable for multiple and/ or repetitive
problems. In addition, four inspectors would like each failure of the zero-
tolerance standard for fecal material to be documented in separate
noncompliance records rather than recording multiple instances in a single
record, which may obscure the frequency of the problem. Thirty- nine
inspectors and veterinarians also raised concerns regarding product quality.
In addition, seven inspectors stated that the quality performance standards
need to be tightened so that fewer defects are overlooked.
In addition, 30 inspectors and veterinarians commented that they are
concerned that the zero- tolerance standard for fecal material is not
adequately enforced, especially since the carcass inspector examines only
the back of the carcass and does not look inside the cavity. Seven suggested
that FSIS place a mirror or provide some other mechanism to facilitate
observation by the carcass inspector.
Finally, 62 inspectors commented that working conditions under the modified
inspection system are better because the inspectors no longer need to make
repetitive motions, as they did when they continuously examined each carcass
on the slaughter line by touch. Sixteen inspectors also said that they like
the ability to physically move from their location at least once every hour.
Twenty- five of them mentioned, however, that the carcass inspectors?
location might expose them to agents, such as chlorine or trisodium
phosphate that could adversely affect their health.
We believe that a risk- based inspection system- such as the one that USDA
is pilot- testing at chicken plants and is starting at hog plants- has merit
in concept and is consistent with the existing risk- based framework for
HACCP. However, while we support this approach, we believe that the design
of this pilot will not permit USDA to reach conclusions about whether the
new system of modified inspections performs as well as the traditional
system. Conclusions
Page 34 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
If, as planned, USDA undertakes to modify its current inspection system for
chicken plants, it will need to consider several factors that we discuss in
this report and that are also key features of the Australian and Canadian
modified inspection systems. First, if USDA issues regulations that require
all chicken plants to modify their inspection systems concurrently without
first determining whether individual plants are able to meet standards, it
may risk including those plants with repeated records of noncompliance. Such
plants may have difficulty meeting their new responsibilities. Continued
participation in a modified inspection system should depend on the plants?
ability to maintain good performance. Second, adding a requirement that
plants use statistical process control systems to identify variations in
performance will allow the plants to better manage and control their
production processes and will also allow USDA to appropriately monitor and
verify inspection systems? performance at each plant over time. Third,
without requiring that plant personnel receive adequate training to
undertake carcass defect detection responsibilities, USDA may jeopardize
product safety and quality. At the very least, USDA should ensure that
personnel conducting carcass defect detection duties have knowledge and
training comparable to that of USDA inspectors who were previously
responsible for these duties. Last, inspectors and veterinarians responding
to our survey provided several useful comments on how to enhance a modified
inspection system. For example, they believe that USDA needs to address
multiple instances of plant noncompliance with regulatory requirements. As
discussed in this report, many plants had repeated instances of
noncompliance with a critical food safety performance standard.
In addition to the current legal challenges, USDA faces future potential
legal challenges unless its statutory inspection authorities are revised.
These challenges will likely hinder the Department?s objective of modifying
its inspection system at meat and poultry slaughter plants. We continue to
believe that, as we have recommended before, the Congress should consider
revising the Meat and Poultry Acts, 21 U. S. C. sections 604 and 455, to
provide FSIS with the flexibility and discretion to target its inspection
resources for the most serious food safety risks. Such revisions would
eliminate the requirements that USDA has traditionally implemented through
continuous carcass- by- carcass government inspection and replace them with
a risk- based inspection system that includes government oversight and
verification.
Page 35 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
If USDA decides to implement modifications to its inspection system, we
recommend that the Secretary of Agriculture direct FSIS to
phase in the implementation of modified inspections so that only plants
with a good history of regulatory compliance are eligible to participate and
continue in the program;
require plants to adopt statistical process control systems to manage and
control their production and require FSIS personnel to monitor and verify
these systems;
in conjunction with industry, develop a training and certification program
for personnel involved in tasks previously performed by federal inspectors
and require that only trained and certified plant personnel perform these
duties; and
consider the merits of adopting suggestions from inspectors and
veterinarians at pilot project plants contained in this report, such as how
to address repetitive instances of noncompliance with regulatory
requirements.
We further recommend that, if in addition to the current pilot project for
chickens, USDA decides to conduct similar pilots for other species- hogs,
turkeys, or cattle- the Department take steps to ensure that the pilot?s
design and methodology are sufficiently rigorous to allow more valid
conclusions than those in this chicken pilot.
We provided USDA with a draft of this report for review and comment. In
written comments, USDA said that the report?s recommendations were
appropriate and that the agency will address them when it seeks public
comment prior to making any regulatory changes. USDA provided technical
comments, which we incorporated in the report as appropriate. USDA?s
comments and our responses are contained in appendix VII.
We performed our review from December 2000 through October 2001, in
accordance with generally accepted government auditing standards.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from
the date of this letter. At that time, we will send copies to congressional
committees with jurisdiction over food safety issues; the embassies of
Recommendations for
Executive Action Agency Comments and Our Evaluation
Page 36 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Australia and Canada; the Secretary of Agriculture; the Director, Office of
Management and Budget; and other interested parties. We will make copies
available to others on request.
If you have any questions about this report, please contact me at (202) 512-
3841. Major contributors to this report are listed in appendix VIII.
Lawrence J. Dyckman Director, Natural Resources
and Environment
Appendix I: Scope and Methodology Page 37 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
To describe the objectives, design, and scope of the pilot project, we
interviewed cognizant government and industry officials. Specifically, we
interviewed officials and/ or reviewed documents from the U. S. Department
of Agriculture?s (USDA) Food Safety and Inspection Service (FSIS); Office of
Policy, Program Development, International Programs, and Field Operations;
and Foreign Agriculture Service. We also consulted with the National Joint
Council of Food Inspection Locals, Research Triangle Institute (RTI), Center
for Science in the Public Interest, Government Accountability Project,
European Union Commission, Consumer Federation of America, National Chicken
Council, National Turkey Federation, National Veterinarian Association, and
American Meat Institute. We obtained information as well from Federal
Register notices and court filings. Additionally, we visited and interviewed
plant managers, USDA inspectors, and veterinarians at five chicken plants
and one hog plant participating in the pilot project. To provide information
on similar projects in Australia and Canada, we visited and toured their
meat and poultry plants. We also interviewed Australian and Canadian
government officials, inspectors, and veterinarians, consumer groups, and
plant managers to obtain their perspectives on their country?s modified
inspection systems. We did not evaluate the validity of the Australian and
Canadian projects? design and methodology.
To identify limitations, if any, in the project?s design and methodology, we
reviewed USDA?s contract with RTI and discussed data collection and data
analysis methods with statisticians from USDA, the Australian Quarantine and
Inspection Service, and the Canadian Food Inspection Agency. We obtained
input on the pilot project?s design from consumer groups, industry
associations, poultry science experts, and our design methodologists. We
also reviewed the sample sizes selected by USDA and determined whether the
results generated from the volunteer plants can be generalized to entire
population. We interviewed managers of pilot project plants to identify the
length of industry training provided for plant personnel who replaced USDA
inspectors on the slaughter line and any modifications made to the slaughter
line before and during the pilot project that could affect the project?s
results. Moreover, we participated in FSIS? three- day training course
covering pilot project inspections.
Additionally, we analyzed noncompliance records issued by USDA inspectors
for failures to meet the zero- tolerance standard for fecal material at
chicken plants. We asked USDA to provide a complete set of noncompliance
records for each of the 11 chicken plants that participated in the baseline
and modified inspection phases of the pilot project. That is, we asked for
noncompliance records 1 year before and 1 year after Appendix I: Scope and
Methodology
Appendix I: Scope and Methodology Page 38 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
implementation of the modified inspections at each plant. To the best of our
knowledge, we received complete data from all the plants. However, for some
plants, we did not receive 12 months? worth of data reflecting the modified
inspections because the plants had been operating under modified inspections
for less than a year.
We began our analysis by removing duplicate noncompliance records and
noncompliance records resulting from sanitation and other nonfecal problems.
We then counted the number of noncompliance records resulting from fecal
material violations by month for each plant to determine whether the total
number of noncompliance records was increasing or decreasing after
implementation of the modified inspections. Since each noncompliance record
can document multiple violations, we also analyzed, on a monthly basis,
whether the total number of carcasses contaminated with fecal material for
each plant was increasing or decreasing under the modified versus
traditional inspections. Furthermore, we counted and compared the number of
noncompliance records generated from the same type of system failure
(repetitive failures) before and after implementation of the modified
inspections. For some months, we did not have any noncompliance records
documenting problems with fecal material.
To evaluate whether the data generated by the pilot project will allow USDA
to reach valid conclusions on the relative effectiveness of the modified
inspection system tested in the pilot project, we reviewed, compared, and
contrasted pilot project data gathered by USDA inspectors and RTI. We
requested and received pilot project data gathered by RTI at the 11 chicken
plants that participated in both the baseline and modified inspection phases
of the pilot project. We compared the individual plants? results with the
two food safety and five food quality performance standards to determine if
each plant was able to meet the standards. We also determined whether plants
improved their performance over time under modified inspections by comparing
individual plants? results under modified inspections with their results
under traditional inspections. We also requested USDA inspectors?
verification data from the same 11 chicken plants. Because one plant dropped
out of the pilot project in early 2001, we did not evaluate the data from
this plant. USDA averaged the verification results from all of the plants
together to determine if the plants collectively met the performance
standards. We chose to average each plant?s results individually to
determine its ability to meet or exceed the performance standards over time.
Appendix I: Scope and Methodology Page 39 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
In addition, we surveyed all pilot project inspectors and veterinarians who
the USDA identified as working at pilot project plants to obtain their views
and concerns regarding the modified inspection system. USDA officials
provided the survey respondents with 1 hour of compensatory time for
completing our survey. In designing our survey, we interviewed pilot project
program officials, inspectors, and veterinarians to determine what issues
should be included in the survey. During the pretesting phase of our survey,
it became evident that respondents considered the survey questions to be
highly sensitive. To address these concerns, we used procedures to guarantee
the anonymity of all survey responses. However, the use of a separate return
postcard for follow- up purposes allowed us to track which respondents did
and did not mail back survey responses. After developing an initial draft of
the survey questionnaire, we visited and pretested the questionnaire with 10
respondents at two chicken plants and one hog plant to obtain comments from
pilot project inspectors and veterinarians and to ensure that the survey was
appropriate for both species of animals. We revised the questionnaire in
accordance with their comments. Then we incorporated input from union
officials and USDA officials before sending out the questionnaires. We
distributed them by mail according to a list provided by USDA officials.
However, we sent additional questionnaires to those pilot project inspectors
and veterinarians who were missing from the initial list and who called us
to say they did not receive one. We also sent three follow- up reminders in
the weeks following the initial distribution of the questionnaire. We
reviewed each survey response to identify internal data inconsistencies. We
sent out questionnaires to 225 persons and received 210 responses, for a 93-
percent- response rate.
We performed our review from December 2000 through October 2001, in
accordance with generally accepted government auditing standards.
Appendix II: Chronology of Pilot Project Litigation
Page 40 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
USDA?s Food Safety and Inspection Service sought public comment on the
development of new inspection models for the slaughter and processing of
meat and poultry in connection with the implementation of its previously
promulgated HACCP rule. 1 FSIS stated that the inspection models would not
include fixed FSIS inspection stations on slaughter lines, as under existing
procedures. Instead, FSIS inspectors would provide oversight at the
slaughter lines and verification that plants were properly implementing
HACCP, among other things. FSIS further stated that, upon completion of the
models project, it will initiate rulemaking, as appropriate, to change
existing inspection procedures.
American Federation of Government Employees (AFGE) filed a complaint in the
U. S. District Court for the District of Columbia seeking to enjoin the USDA
from proceeding with the pilot project, on the grounds that it violated the
Federal Meat Inspection Act, 21 U. S. C. section 604, and the Poultry
Products Inspection Act, 21 U. S. C. section 455, by not requiring federal
government officials to perform carcass- by- carcass postmortem inspection.
U. S. District Court (Lamberth, R.) granted USDA?s motion for summary
judgment, holding that neither the Federal Meat Inspection Act nor the
Poultry Products Inspection Act was violated by the HACCP- Based Inspection
Models Project. American Federation of Government Employees v. Glickman, No.
98- 0893 (D. D. C. Sept. 23, 1999). The court reasoned that even though the
statutes require postmortem inspections by USDA inspectors, the word
?inspection? in these statutes does not necessarily mandate direct, physical
?organoleptic?- involving sight, touch, and smell- examination by USDA
inspectors.
AFGE appealed the district court?s ruling to the U. S. Court of Appeals for
the District of Columbia Circuit, which reversed the district court?s
decision. The court of appeals held that delegating the task of inspecting
carcasses to plant employees violated the Federal Meat Inspection Act and
the Poultry Products Inspection Act because both statutes require that
federal inspectors, rather than private employees, determine whether a
product is adulterated. American Federation of Government Employees v.
Glickman, 215 F. 3d 7, 9 (2000). The court of appeals remanded the case to
the district court for further proceedings consistent with its opinion.
1 ?HACCP- Based Meat and Poultry Inspection Concepts,? 62 Fed. Reg. 31553
(June 10, 1997). Appendix II: Chronology of Pilot Project
Litigation 6/ 10/ 97
4/ 8/ 98 9/ 23/ 99 6/ 30/ 00
Appendix II: Chronology of Pilot Project Litigation
Page 41 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
AFGE moved the district court to enter an order declaring that the pilot
project violated the Federal Meat Inspection Act and the Poultry Products
Inspection Act and permanently enjoining USDA from delegating its statutory
duty to inspect each meat and poultry carcass and its parts to anyone other
than federal inspectors.
USDA responded to AFGE?s motion, stating that to comply with the court of
appeals? decision, FSIS was modifying the pilot project so that an FSIS
inspector would directly inspect every carcass. Specifically, USDA stated
that in the 11 chicken plants in the pilot project, one FSIS inspector would
be stationed toward the end of the slaughter line, between the final wash
and the chiller. This inspector would be responsible for examining each
carcass and determining whether it is adulterated. In the three hog plants,
FSIS inspectors would be placed at up to three fixed locations on the
slaughter line and would be responsible for examining the carcass, head, and
viscera, and determining whether they were adulterated.
All pilot project plants had implemented the redesigned inspection model. 2
U. S. District Court for the District of Columbia ruled that the redesigned
pilot project does not violate the court of appeals ruling in American
Federation of Government Employees v. Glickman, 215 F. 3d 7 (2000) and does
not violate the Federal Meat Inspection Act and the Poultry Products
Inspection Act. American Federation of Government Employees v. Glickman, No.
98- 893 (Jan. 17, 2001). According to the district court, the redesigned
pilot project is consistent with the court of appeals decision because the
thrust of that opinion was that to satisfy its inspection duties, FSIS must
do more than merely observe others performing inspections, and in the
redesigned project, federal inspectors will make the critical determination
as to whether a product is adulterated. Furthermore, the redesigned pilot
project satisfied the requirements of the federal meat and poultry
inspection statutes, according to the district court, because it reflects a
reasonable interpretation of the meaning of the term ?inspection? within
those statutes.
2 See FSIS Briefing Paper: The History of the HACCP- Based Inspection Models
Project (Jan. 19, 2001). 8/ 25/ 00
9/ 15/ 00 11/ 6/ 00 1/ 17/ 01
Appendix II: Chronology of Pilot Project Litigation
Page 42 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
AFGE filed a notice of appeal to the U. S. Court of Appeals for the District
of Columbia Circuit. The appeal is still pending. 2/ 5/ 01
Appendix III: Performance Standards for Hog Plants
Page 43 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
For hog plants, FSIS developed three food safety and three food quality
standards from traditional inspections data (baseline data) gathered at five
plants. (See table 9.) These standards were set slightly below the
performance of the fourth (out of five) plants. 1 At the time of our review,
FSIS did not have complete pilot project data available for the hog plants.
Table 9: Performance Standards for Inspection Systems at Hog Plants in the
Pilot Project
Categories Performance standards
(percentage of defects allowed per shift)
Food Safety 1: Infectious conditions such as septicemia and toxemia 0 Food
Safety 2: Digestive content such as fecal material, ingesta, milk
0 Food Safety 3: Antemortem symptoms such as neurologic conditions 0 Other
Consumer Protection 1: Carcass pathology such as arthritis, emaciation, etc.
4.1 Other Consumer Protection 2: Visceral pathology such as enteritis/
gastritis, fecal contamination of viscera, etc.
7.2 Other Consumer Protection 3: Miscellaneous such as anemia, bile,
bruises, scabs, etc. 20.5
Source: USDA, FSIS.
USDA is using the safety and quality performance standards developed for
this project to determine, among other things, if the plants can perform at
least as well as they did under traditional inspections.
1 USDA developed the hog performance standards from the following formula:
(0. 25 * 3rd plant?s performance) + (0. 75 * 4th plant?s performance). These
standards are comparable to the 25th percentile established for the chicken
pilot. Appendix III: Performance Standards for Hog
Plants
Appendix IV: Survey Results Page 44 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results
Appendix IV: Survey Results Page 45 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 46 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 47 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 48 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 49 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 50 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 51 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 52 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 53 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 54 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 55 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 56 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 57 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix IV: Survey Results Page 58 GAO- 02- 59 Meat and Poultry Pilot
Inspection Project
Appendix V: Australia?s Meat Safety Enhancement Program
Page 59 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Australia?s Meat Safety Enhancement Program (MSEP) is a pilot meat
inspection program developed by the Australian Quarantine and Inspection
Service (AQIS) for use in meat export plants. MSEP is patterned after the
Australian domestic meat inspection system, which is grounded in the theory
of Hazard Analysis Critical Control Point procedures and uses a quality
systems approach to control for hazards in food processing. Under the
Australian inspection approach, slaughter plants are clearly identified as
being responsible for plant process control, including sanitation, microbial
monitoring, and inspection of the product for quality defects. AQIS sets
public health product standards and audits plant operations to ensure that
food safety standards are being achieved.
Australia?s traditional system of meat inspection had an end- product
inspection focus and employed government staff to perform inspections.
However, Australian authorities were concerned that this system did not
adequately address all of the risks to food safety. For example, public
health officials, among others, thought that proper emphasis was not being
given to microbial contaminants, such as E. coli and Salmonella, or residues
of veterinary and agricultural chemicals and antibiotics, which had emerged
as threats to the safety of fresh meat and meat products.
Australia first moved in 1991 to address these emerging food safety issues
with a comprehensive review of domestic meat inspection. Upon the completion
of this review, the review task team outlined changes to existing meat
inspection procedures that it thought would result in a more cost- effective
system while maintaining adequate safeguards to public health. The
recommended changes included strategies for implementing quality assurance
programs and HACCP procedures.
Over an 18- month period, commencing in early 1992, three domestic slaughter
plants developed quality assurance systems that included replacing
regulatory agency inspectors with company employees. AQIS subsequently
approved this new inspection approach for trial. In the trial, each plant
was paired with a nearby similar plant as a control plant. The control
plants continued operations under the traditional meat inspection system.
According to AQIS, the trial results showed that all three plants operating
under their quality assurance systems and using companyemployed meat
inspectors were capable of maintaining standards of food safety at least
equal to those of the control plants. Appendix V: Australia?s Meat Safety
Enhancement Program Background and Trends That Led to the Program
Appendix V: Australia?s Meat Safety Enhancement Program
Page 60 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
These findings were examined and endorsed by a scientific panel composed of
officers from AQIS, universities, and meat inspector representatives of the
Public Sector Union and the Australian Meat Research Council. All Australian
domestic meat establishments in each state subsequently moved to a self-
regulated system of meat inspection, whereby companies assumed
responsibility for meat safety. MSEP, which was implemented in 1996 for meat
export facilities, was an outgrowth of the domestic movement to quality
assurance systems using HACCP procedures and company- employed inspectors.
The MSEP model incorporates the use of routine meat inspection performed by
qualified company employees working under a system approved and legislated
by the government. The government, acting through AQIS, maintains full- time
oversight, verification, and certification of the final product. As a part
of Australia?s MSEP system, a full- time AQIS veterinary officer, whose
salary is reimbursed by the company, is required to be on duty at all times
in each participating export plant. At establishments producing products for
export to the United States, an AQIS inspector assists the veterinary
officer in performing the zero tolerance and end- product inspection.
Company management is responsible for on- line inspection using company-
paid inspectors that have been trained in accordance with a standardized
curriculum and are under the general supervision of the full- time AQIS
veterinary officer. The officer provides continuous oversight, verification,
and daily audit of plant operations and production control systems. The
company- paid inspectors must also be licensed by their respective states
before they may act as inspectors. To be eligible for a license, they must
complete about 600 hours of formal classroom training and pass examinations.
The division of responsibilities between AQIS and companies under MSEP are
as shown in table 10.
Table 10: Division of Responsibility Under Australia?s Meat Safety
Enhancement Program Function Company?s responsibility AQIS? responsibility
Antemortem inspection Sort healthy animals for slaughter and separate
animals identified as potential suspects for inspection by AQIS.
Inspect suspect animals and decide on their disposition.
Postmortem inspection Develop procedures for carcass, head, and offal
inspection using all the inspection activities required by AQIS.
Set up monitoring checks for all inspection activities conducted by company-
employed inspectors.
Conduct regular oversight and verification checks of post- mortem inspection
activity at least daily.
Conduct independent checks of the inspection procedures during the monthly
audit review.
How the MSEP System Works
Appendix V: Australia?s Meat Safety Enhancement Program
Page 61 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Function Company?s responsibility AQIS? responsibility
E. coli and Salmonella monitoring Company Quality Assurance officers or
inspectors will take samples for testing.
Check the E. coli samples to ensure that processing is consistent with good
laboratory practice.
Supervise each Salmonella test. Review microbiological results in the
monthly review of the plant. End- of- line inspection Take appropriate
corrective action if the AQIS
veterinary officer detects defects. Perform regular ?check the checker?
audits of company inspection and quality assurance personnel. Monitoring
Monitor all procedures and all aspects of the
HACCP program. Continuously monitor and recheck
antemortem and postmortem inspections and
microbiological testing and the HACCP program.
Source: AQIS.
According to AQIS officials, the service has established several
prerequisites for companies that wish to participate in MSEP and sell
products for export. In addition, plants must demonstrate the following
before they can be a part of the program. Specifically, plants must
have a good operational/ coregulatory trade record; have demonstrated a
commitment to and appreciation of HACCP/ Quality
Assurance principles, to include an adequate company infrastructure and
training competencies; accept responsibility for performing the sorting
(inspection) function; develop and maintain a plant- specific, AQIS-
approved Meat Safety Quality
Assurance System; act as a promotional site for industry/ overseas
officials; and maintain a comprehensive plant quality assurance manual
that was
developed by and is ?owned? by relevant areas of plant operations. The
manual must be a working document and receive AQIS? approval. When approved,
the HACCP/ Quality Manual acts as a legal contract between the plant and
AQIS on the specific procedures, controls, and standards at the plant.
Participation in the program is currently voluntary. However, eventually,
all plants will be required to move to the new inspection system.
Prerequisites for
Plants to Participate in MSEP
Appendix V: Australia?s Meat Safety Enhancement Program
Page 62 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
According to AQIS officials, much empirical evidence on the effectiveness of
HACCP quality assurance systems has accumulated since MSEP was implemented
in Australia. AQIS officials told us they are fully confident that the MSEP
approach to regulatory inspection control achieves guarantees of food safety
that are at least equivalent to those of the traditional approach to meat
inspection. They said the system incorporates a reengineering of meat
inspection practices and results in various changes to the culture or
philosophy of food inspection, including changes that
prevent rather than correct, build quality in rather than rely on end-
product inspection, place more reliance on microscopic verification,
transfer responsibility from government to industry for performing certain
functions and maintaining standards, and incorporate government
verification and certification. AQIS? Evaluation of
Its Program
Appendix VI: Canada?s Modernized Poultry Inspection Program
Page 63 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
The Canadian Food Inspection Agency (CFIA) recently adopted a new poultry
inspection system that allows industry to progressively assume defect
detection duties and remove defective carcasses from the slaughter line. The
agency began pilot testing this modified inspection system in September
1997. The pilot project is similar in design to USDA?s pilot project for
poultry. The new system, known as the Modernized Poultry Inspection Program
(MPIP), has goals to reduce microbial pathogens in raw poultry products and
to enhance the use of science and risk- based management in Canada?s poultry
inspection system. MPIP builds on a precursor program- the Canadian Poultry
Inspection Program (CPIP)- which also allows industry operators to assume
some defect- detection duties. The significant difference between the two
programs is that under MPIP, industry personnel are also responsible for
detecting viscera defects. Under both programs, CFIA inspectors monitor
poultry plants? slaughter operations and verify industry?s compliance with
food safety and quality standards. For poultry plants that choose not to
participate in either of the programs, government inspectors continue to
perform all carcass defect detection and removal duties.
Before CFIA amended its meat inspection regulations to adopt the MPIP
program in May 2001, the agency pilot tested the modified inspection system
at four chicken plants that represented about 10 percent of the total
chicken production. CFIA officials told us that several factors contributed
to their decision to modify the poultry inspection system, including the
desire to make inspections more consistent with the HACCP framework. The use
of the HACCP system is not yet mandatory at meat and poultry plants in
Canada, but it is a prerequisite for participation in MPIP. In addition,
CFIA officials noted that the United States was engaging in a similar effort
to modify its slaughter inspection system and said that Canada, which was
modifying its poultry inspection system, wanted to obtain equivalency status
with the United States to facilitate bilateral trade. Finally, CFIA
officials noted that the Canadian poultry industry is investing in new
evisceration equipment with technical improvements that permit faster line
speeds without compromising food safety and quality.
MPIP was a voluntary pilot project, and now that Canada has issued
regulations to implement the program nationwide, participation continues to
be voluntary. Prior to being accepted into MPIP, poultry plants must pass a
preliminary assessment. Later, they must pass additional audits as they move
from one phase of the program to the next. The audits, which Appendix VI:
Canada?s Modernized Poultry
Inspection Program Background How the MPIP Pilot Works
Appendix VI: Canada?s Modernized Poultry Inspection Program
Page 64 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
are performed by a CFIA audit team, are an important tool for the agency and
plant managers. Clearly stated audit objectives help plant managers to plan
for and address potential deficiencies before they occur, allowing the plant
to succeed in the program. Plants participating in the program move through
three phases.
Phase 1 begins after a CFIA preliminary assessment of a volunteer plant?s
eligibility to participate in the modified inspection program. All volunteer
plants must have been operating under CPIP before participating as an MPIP
pilot plant. As with USDA?s pilot project, baseline data are collected and
analyzed before MPIP implementation. That is, microbial and quality defect-
detection data are collected at volunteer plants while they are operating
under CPIP. In addition, plant personnel are trained and accredited in their
new role as plant defect detectors.
Phase 2 resembles the transition phase in USDA?s pilot project. It is the
trial phase during which CFIA inspectors back up plant personnel in
detecting defects on the slaughter line. Plant personnel are required to
pass four weekly practical on- line tests as viscera detectors. In addition,
the plant must also pass a national compliance and verification review.
Phase 3 entails operating under MPIP and the collection and analysis of data
after MPIP?s implementation at the plant. Accredited plant personnel detect
carcass, cavity, and viscera defects, while plant preselectors cull
carcasses with specific processing and pathology defects, such as
contamination from the intestines and septicemia. On average, a Canadian
poultry plant runs one evisceration line and will have one veterinarian-
incharge and two CFIA inspectors (one evisceration floor inspector per line
working on the slaughter floor and one processing area inspector). If an
equivalency agreement with the United States requires federal inspection of
each and every carcass, CFIA will place one on- line government inspector on
each slaughter line at plants that want to export to the United States.
CFIA has established several prerequisites that plants must meet before they
are accepted to participate in MPIP. CFIA considers, at a minimum, the
following criteria for each plant:
The state of the current plant inspection system, its quality control
operations, and the gap between these and MPIP. Prerequisites for
Plants to Participate in the MPIP Pilot
Appendix VI: Canada?s Modernized Poultry Inspection Program
Page 65 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
The stage of plant- specific HACCP system development and implementation-
at a minimum, the plant must have a functioning HACCP system in place to
commence MPIP. The implementation of E. coli and Salmonella testing
protocols. The types of poultry processing conducted by the plant, as it
is desirable
to have several classes of poultry slaughtered to improve efficiency in
training staff, data collection, and implementing future pilots in the area.
The geographic location (desire to implement MPIP and collect data from
across Canada). The compliance history of the plant. The range of the
health status of flocks and average condemnation rates in
the plant selected to pilot MPIP to ensure adequate training and
accreditation of plant defect detectors to handle condemnation rates and a
range of disease conditions under an HACCP- MPIP system.
CFIA?s MPIP national performance standard categories for poultry are similar
but not identical to USDA?s pilot project standards for chickens. The
poultry standards address the following factors, among others:
Microbial contamination. Processing defects, including all types of
contamination- such as fecal
and bile- extraneous material, extensive bruising, extensive overscald/
mutilation, and inadequate bleeding. 1 Pathology defects, including
septicemia/ toxemia, airsacculitis, ascites,
emaciation, cellulitis, dark- colored carcasses, and other diseases. 2
Finished product defects, such as bruises, feathers, trachea, oil glands,
lungs, intestines, and crop. 3 To analyze microbial contamination as part of
the MPIP project, in 1997 and 1998, CFIA performed a national baseline
survey at 36, or 55 percent, of the 65 federally registered chicken plants
nationwide over a 1- year period. CFIA?s performance standards for microbial
contamination are
1 USDA?s food safety performance standard 2 and other consumer protection
(food quality) performance standards 2 and 3 for the pilot project include
these types of defects. 2 USDA?s food safety performance standard 1 and
other consumer protection (food quality) performance standard 1 for the
pilot project include these types of defects. 3 USDA?s other consumer
protection (food quality) performance standards 4 and 5 for the pilot
project include these types of defects. Performance
Standards
Appendix VI: Canada?s Modernized Poultry Inspection Program
Page 66 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
identical to USDA?s Pathogen Reduction: HACCP regulations for generic
E. coli and Salmonella.
To evaluate both processing and pathology defects, in 1997 and 1998, CFIA
conducted another national survey to establish the average level of
overlooked processing and pathology defects on carcasses passed by CFIA
inspectors or by trained and accredited plant detectors under the
traditional and CPIP methods of chicken inspection. Performance standards
for processing and pathology defects were developed from data gathered at
35, or 54 percent, of the 65 federally registered chicken plants nationwide.
The performance standards for processing defects were set at 4.0 percent;
however, as part of the MPIP project, CFIA subsequently tightened them to
2.5 percent, meaning that 97. 5 percent of the carcasses passed by plant
employees must be free of processing defects. The performance standards for
pathology defects were set at 0.4 percent, meaning that 99.6 percent of the
carcasses passed by plant employees must be free of pathology defects.
CFIA conducted an additional national survey to amend its Finished Product
Standards program. This is the same program developed and used by USDA in
all chicken plants operating under the traditional carcass- bycarcass
inspection systems. CFIA, however, has tightened the Finished Products
Standards? pass/ fail criteria for plants operating under CPIP?s and MPIP?s
inspection systems. In addition, CFIA continues to enforce a zero- tolerance
policy for fecal contamination on ready- to- chill carcasses as part of this
program.
CFIA developed an extensive training program for government and industry
personnel to ensure that everyone participating in MPIP has the necessary
skills and knowledge to perform his/ her new duties. The training includes
classroom training, a final exam, in- plant training, and the demonstration
of competency. After successfully completing the training, CFIA trainers
accredit the trainees.
The entire MPIP training program was developed by CFIA. CFIA officials
believe that no outside organization can replicate in- house expertise. In
addition, in order to encourage union cooperation, CFIA inspectors conduct
the training. At MPIP plants, every CFIA employee must participate in the
training program. MPIP Pilot Success
Attributed to Extensive Training of Inspectors and Industry
Appendix VI: Canada?s Modernized Poultry Inspection Program
Page 67 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
CFIA officials modeled MPIP?s industry- training program on the training
program they developed for their own staff. The agency adopted a trainthe-
trainer approach for industry. That is, CFIA inspectors train the plant
trainers who in turn train the plant employees (defect detectors). The
training is conducted in two phases. Phase 1 covers classroom and on- thejob
training and concludes when the defect detector is accredited to perform
his/ her new duties under MPIP. The plant managers decide which plant
employees will become defect detectors and receive MPIP training. The
training covers three types of defect detection: (1) preselection before
evisceration, (2) internal examination of carcasses, and (3) poultry viscera
examination. On- the- job training ensures that defect detectors learn how
to identify carcass pathology conditions that are indicative of disease.
Once the defect detectors are comfortable with identifying diseases, they
are tested on 20 carcasses and the corresponding viscera to see if they can
correctly decide whether the carcasses pass or need to be examined by the
veterinarian. In addition, the defect detectors have to demonstrate the
ability to detect defects on fast- moving lines. The defect detector must
pass three 15- minute tests to be accredited.
After the defect detector is certified, he/ she enters a 4- week trial
period (phase 2) where he/ she performs his/ her new MPIP duties on the
slaughter line. Defect detectors can be trained and accredited in more than
one station (cavity, carcass, and/ or viscera) and will have weekly
evaluations at each of these stations during the trial phase. In addition,
the trainer evaluates defect detectors on a periodic basis, as specified in
the plant?s HACCP plan. Plant defect detectors are reevaluated on an ongoing
basis and lose their accreditation if they fail a periodic review test.
Ongoing testing ensures continuing competence as line speeds and/ or disease
patterns change. CFIA reserves the right to conduct additional
reaccreditation tests of plant detectors at any time. In addition, CFIA
reserves the right to conduct additional tests for missed defective
carcasses at any time.
According to CFIA officials, MPIP is successful because the data generated
from the pilot plants show a downward trend in carcass defects. The plants
that participate in the MPIP pilot are performing better than the
performance set by the national standards. Part of the program?s strength is
that CFIA officials periodically review data results with plant managers to
discuss the plant?s record of performance against the standards. Through
discussions with plant managers, CFIA officials learn what types of
corrective actions are successful in addressing the plant?s system Under
MPIP Pilot,
CFIA Officials Monitor Plant Performance Trends
Appendix VI: Canada?s Modernized Poultry Inspection Program
Page 68 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
failures. In turn, CFIA officials share this knowledge with managers of
other pilot plants to help them improve their own process controls.
CFIA modified its meat- inspection regulations to implement MPIP on a
nationwide basis on May 23, 2001. The agency will continue to maintain three
types of poultry inspection systems (traditional, CPIP, and MPIP). CFIA does
not plan to make changes to the MPIP program and will continue to use a
phased approach for implementing MPIP at volunteer plants. Plants will still
be required to pass a preliminary assessment before being accepted into
MPIP. Under MPIP, CFIA officials will continually monitor the plants?
performance through informal discussions and formal audits. The agency
expects MPIP plants to perform better than the performance set by the
national standards for the microbial, food safety, and other consumer
protection defects. If a plant fails to meet these standards, the plant will
lose its license to operate under MPIP and must be relicensed to operate
either under CPIP or traditional inspections. In addition, the agency
expects MPIP plants to continually improve their performance over time.
Furthermore, CFIA will continue to emphasize the importance of training
through its mandatory training program and certification requirements for
both CFIA employees and industry personnel.
Since CFIA has formally adopted MPIP, the agency will maintain an oversight
inspector and a veterinarian- in- charge at each MPIP plant. Unlike USDA?s
pilot project, CFIA?s MPIP program does not require an on- line carcass
inspector performing carcass- by- carcass inspections. However, if the
equivalency agreement with the United States requires an on- line carcass
inspector, CFIA will comply with this requirement and place a CFIA inspector
on the slaughter line at those plants that export to the United States.
Current Status of
MPIP
Appendix VII: Comments From the U. S. Department of Agriculture
Page 69 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Appendix VII: Comments From the U. S. Department of Agriculture
Note: GAO?s comments supplementing those in the report?s text appear at the
end of this appendix.
Appendix VII: Comments From the U. S. Department of Agriculture
Page 70 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
See comment 1. See comment 2.
Appendix VII: Comments From the U. S. Department of Agriculture
Page 71 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Now on pp. 2- 3. See comment 9. Now on p. 2.
See comment 8. See comment 7.
See comment 6. See comment 5. See comment 4. See comment 3.
Appendix VII: Comments From the U. S. Department of Agriculture
Page 72 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
See comment 18. See comment 17.
See comment 16. See comment 15.
See comment 14. See comment 13. See comment 12.
See comment 11. See comment 10.
Appendix VII: Comments From the U. S. Department of Agriculture
Page 73 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Now on pp. 27- 28. See comment 23. Now on p. 26.
See comment 22. Now on p. 24.
See comment 21. Now on p. 24.
See comment 20. Now on p. 23.
See comment 19.
Appendix VII: Comments From the U. S. Department of Agriculture
Page 74 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
See comment 30. See comment 29. Now on p. 34.
See comment 28. Now on p. 30.
See comment 27. Now on p. 29.
See comment 26. Now on p. 28.
See comment 25. Now on p. 27.
See comment 24.
Appendix VII: Comments From the U. S. Department of Agriculture
Page 75 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Appendix VII: Comments From the U. S. Department of Agriculture
Page 76 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
The following are GAO?s comments on the U. S. Department of Agriculture?s
letter dated November 30, 2001.
1. We believe that the report?s title describes the overall results of our
study. Therefore, we have not modified the title as USDA suggested.
2. We do not agree that our report oversimplifies the goals of the pilot
project. The pilot project?s objective was to compare the accomplishments-
as measured by USDA?s safety and quality performance standards- of two
inspection systems at volunteer plants. We understand that the plants
participating in the pilot project made various other changes at different
times during the pilot project. However, the only change that USDA required
of plants volunteering to participate in the pilot was the replacement of
USDA inspectors with plant personnel. As USDA states, this project was
designed as a ?before- and- after? experiment. The ?before? refers to
traditional inspections with USDA inspectors performing the carcass defect
detection duties, and the ?after? refers to the plant employees? performance
of the carcass defect detection duties. As explained in our report, USDA?s
independent contractor first went to each participating plant to collect
carcass defect data under traditional inspections, waited until the plant
employees went through a transition period, and then returned to collect the
same type of data after plant employees assumed the inspectors? duties. If
USDA wants to compare the accomplishments of two different inspection
systems that include many variables, it is critical to use a control group.
Indeed, the lack of a control group in USDA?s study prevents them from
knowing the extent to which plants that do not participate in the study
(about 175 plants nationwide) made changes in their processing systems that
improved safety and quality while continuing to inspect under the
traditional system. We continue to believe that the design of the USDA?s
pilot project does not permit USDA to interpret what changes account for the
experiment?s outcomes.
3. We agree with the premise that the use of volunteer plants is a
legitimate approach and our report acknowledges that, once USDA decided to
go forward with this pilot project design, it could not compel plants to
participate and, therefore, had to rely on volunteer plants. However, given
that USDA decided to proceed with this design, our report points out that
USDA had the option of randomly selecting a subgroup from the volunteer
plants to serve as a control group. The problem with relying on volunteers
for this study is that there are a GAO?s Comments
Appendix VII: Comments From the U. S. Department of Agriculture
Page 77 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
small number of plants participating- there are only 11 chicken plants for
which before and after data have been collected. Even if the plants had been
randomly selected, they would not represent a sufficient number to enable
USDA to generalize the results to the entire population of chicken plants.
Therefore, USDA cannot assume that the results from these 11 plants would
apply to all the plants that in the future may participate in a modified
inspection system. Finally, neither USDA nor RTI provided GAO with any
evidence that the pilot project was either designed or analyzed as a time-
series study.
4. We do not agree with USDA that the plants participating in the study are
typical of the industry in that they represent diversity in geography,
corporate structure, management styles, numbers of evisceration lines,
product distribution, and other variables. As figure 2 of our report shows,
the pilot project has before- and- after data for 11 plants that are located
in the southern region of the United States. Therefore, particularly with
respect to microbial data, the results from these plants may not be
indicative of how plants in other areas of the United States may behave.
Geographic distribution and seasonal variations are important variables that
cannot be ignored in assessing a program. As a recent USDA study points out,
Salmonella and E. coli 0157: H7 rates appear to differ dramatically by
season, while geography seems to affect only Salmonella results. When, as
part of this study, USDA researchers investigated the regional patterns,
they found higher positive rates for Salmonella in warmer climates. For
example, in southern operations, the rates were about 8 percent, whereas the
rate was just under 5 percent in northern operations. This study highlights
the importance of variables such as seasonal variations and geographic
distribution. As discussed in our report, the design of USDA?s pilot project
did not appropriately consider these variables.
5. Our report accurately describes the duties of FSIS inspectors and plant
personnel as they have changed during the course of the litigation outlined
in appendix II. However, as noted in comment 9, we have modified our
characterization of current plant personnel activities in accordance with
USDA comments.
6. We made the technical changes as appropriate. 7. We disagree. Our report
does not describe the program as being
designed as a cost- saving initiative.
Appendix VII: Comments From the U. S. Department of Agriculture
Page 78 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
8. We modified our report to state that there are only 11 chicken plants
with before- and- after data.
9. We have made the first change suggested. However, USDA has not provided
any information to demonstrate that inspection resources are being utilized
elsewhere in the farm- to- table continuum other than performing for plant
inspections.
10. We disagree. We believe that the report consistently describes the
purpose and type of data collected by RTI. See comment 2.
11. We made the technical changes as appropriate. 12. We made the technical
changes as appropriate. 13. We made the technical changes as appropriate.
14. We disagree. According to the documentation provided by USDA, there
is a model for fed cattle and, as of October 2001, no cattle plants had
volunteered to participate.
15. Table 1 and its footnotes describe the number of plants participating in
the project.
16. We made the technical changes as appropriate. 17. We made the technical
changes as appropriate. 18. We agree with the agency that it should look at
this during the
rulemaking process. 19. We disagree. Our report acknowledges that
inspectors? verification
checks increased from 20 to 80 at pilot project plants, a fourfold increase
in verification checks. However, the number of noncompliance records issued
by inspectors at some of the plants far exceeded the level expected by the
increase in verification checks. At one plant, there was a twentyfold
increase in noncompliance reports. Our intent is to alert USDA of the fact
that, after shifting to modified inspections, some of the plants appear to
have problems with fecal contamination. USDA?s HACCP Rules of Practice
contemplate the Department?s taking action for noncompliance by withholding
the marks of inspection or by suspending inspections in instances where it
decides that it is necessary to do so because of multiple or recurring
Appendix VII: Comments From the U. S. Department of Agriculture
Page 79 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
regulatory noncompliances evidenced by noncompliance records. We understand
from FSIS officials that they are now considering how many repeated
instances of noncompliance will result in further regulatory action.
20. We agree that actions taken when there are multiple instances of
noncompliance are based on the same decision- making criteria in both
project pilot plants and in traditional plants.
21. About 60 percent of our survey respondents said that line speeds are
somewhat too fast or much too fast for food safety.
22. We made the technical changes as appropriate. 23. We agree with USDA?s
comment that, as shown in table 5 of our report,
the generic E. coli results show improvement after the plants shifted to
modified inspections. As the table shows, 2 of the 11 plants improved while
9 remained unchanged.
24. We disagree. As stated in our report, we believe that looking at
individual plant inspection systems? performance is relevant and appropriate
because each plant is expected to meet the standards. Therefore, in table 4,
our report shows how each plant performed, as measured by USDA?s performance
standards.
25. We made the technical changes as appropriate. 26. We disagree. As stated
in our report, we believe that looking at
individual plant inspection systems? performance is relevant and
appropriate. Therefore, in table 6, our report shows how each plant
performed, as measured by USDA?s performance standards.
27. We agree. Our original criterion for analyzing changes in the data
presented in- table 7 was .01 percent- which we used because it reflects the
level of precision in the data that USDA provided to us. However, in
response to USDA?s comment that declines in performance of fractions of 1
percent may not be meaningfully different, we modified our criterion to 0.5
percent. This means that a plant?s change from ?same? to ?worse? or ?better?
would now represent 215 chickens per line per shift compared with 4.3
chickens per line per shift in terms of quality and safety. By changing our
analysis, we now find that in 16 instances, the plants? performance under
modified inspections moved from ?better? to ?same? and in 8
Appendix VII: Comments From the U. S. Department of Agriculture
Page 80 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
instances the plants? performance moved from ?worse? to ?same.? In summary,
even after changing our criterion, we find that in several instances, the
plants are not meeting USDA?s goal- that is, that modified inspections
should achieve at least the same level of performance as traditional
inspections. We modified table 6 to reflect these changes. In table 6, we
clearly identify which plants have or have not met each individual
performance standard.
28. We agree. As stated in our report, we believe that training of plant
personnel is an issue that must be addressed by USDA.
29. We made the technical changes as appropriate. 30. We made the technical
changes as appropriate.
Appendix VIII: GAO Contact and Staff Acknowledgments
Page 81 GAO- 02- 59 Meat and Poultry Pilot Inspection Project
Robert C. Summers (404) 679- 1839 In addition to the name above, Maria
Cristina Gobin, John M. Nicholson, Diana P. Cheng, Shana B. Wallace, Fran A.
Featherston, Karen K. Keegan, and Cynthia Norris made key contributions to
this report. Appendix VIII: GAO Contact and Staff
Acknowledgments GAO Contact Acknowledgments
(360024)
The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO?s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO?s Web site (www. gao. gov) contains abstracts and full- text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
site daily. The list contains links to the full- text document files. To
have GAO E- mail this list to you every afternoon, go to our home page and
complete the easy- to- use electronic order form found under ?To Order GAO
Products.?
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U. S. General Accounting Office P. O. Box 37050 Washington, D. C. 20013
To order by phone: Voice: (202) 512- 6000 TDD: (301) 413- 0006 Fax: (202)
258- 4066
GAO Building Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)
Washington, D. C. 20013
Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail:
fraudnet@ gao. gov, or 1- 800- 424- 5454 (automated answering system).
Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S.
General Accounting Office, 441 G. Street NW, Room 7149, Washington, D. C.
20548 GAO?s Mission
Obtaining Copies of GAO Reports and Testimony
Order by Mail or Phone Visit GAO?s Document Distribution Center
To Report Fraud, Waste, and Abuse in Federal Programs
Public Affairs
*** End of document. ***