    Food Safety: Weaknesses in Meat and Poultry Inspection Pilot
    Should Be Addressed Before Implementation (17-DEC-01, GAO-02-59).

    In 1997, the Department of Agriculture (USDA) announced that it
    would modify its meat and poultry slaughter inspection program to
    make industry more responsible for identifying carcass defects.
    Before making the change permanent, USDA developed a model to
    test whether a prevention-oriented inspection system that uses
    plant personnel to examine each carcass and USDA inspectors to
    verify that quality standards are met would continue to ensure
    the safety of meat and poultry products. USDA's pilot project for
    chickens has several design and methodology limitations that
    compromise the overall validity and reliability of its results.
    First, the chicken pilot that USDA designed lacks a control
    group--a critical design flaw that precluded a comparison between
    the performance of the inspection systems at those plants that
    volunteered to participate in the pilot and that of plants that
    did not participate. Second, the chicken plants that volunteered
    to participate in the baseline measurement phase of the pilot
    were not randomly selected, and they did not include plants from
    all chicken-producing ares or plants of all sizes. Third, the
    pilot project's methodology did not take into account such
    variables as seasonal changes and plant modifications that could
    affect project results. Finally, USDA's pilot project did not
    include features of the modified inspection systems in Australia
    and Canada that would be important considerations in ensuring the
    successful implementation of a modified inspection system
    nationwide. Notwithstanding the project's design problems, the
    data themselves do not conclusively demonstrate that modified
    inspections are at least equal to traditional inspections.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-02-59
        ACCNO:   A02437
      TITLE:     Food Safety: Weaknesses in Meat and Poultry Inspection
    Pilot Should Be Addressed Before Implementation
         DATE:   12/17/2001
      SUBJECT:   Food and drug law
    	     Meat inspection
    	     Poultry inspection
    	     Contaminated foods
    	     Consumer protection
    	     Safety regulation
    	     Food facilities
    	     Listeria monocytogenes Bacteria
    	     Salmonella Enteritidis Bacteria
    	     USDA Meat and Poultry Inspection Program
    	     Australia Meat Safety Enhancement
    	     Program
    	     Canada Modernized Poultry Inspection
    	     Program
    	     E.coli Bacteria
    	     FSIS Hazard Analysis and Critical
    	     Control Point System

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Testimony.                                               **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-02-59

    United States General Accounting Office GAO Report to the Committee on

    Agriculture, Nutrition, and Forestry, U. S. Senate

    December 2001 FOOD SAFETY Weaknesses in Meat and Poultry Inspection Pilot
    Should Be Addressed Before Implementation

    GAO- 02- 59

    Page i GAO- 02- 59 Meat and Poultry Pilot Inspection Project Letter 1

    Results in Brief 2 Background 5 Pilot Project Makes Industry More
    Responsible and Accountable

    for Product Safety and Quality, While Continuing Government Monitoring With
    Fewer Resources 9 Design and Methodology Limitations Compromise Overall
    Validity

    and Reliability of Pilot Project Results 15 Pilot Project Results Are
    Inconclusive, but FSIS Inspection

    Personnel Generally Support Modified Inspections 25 Conclusions 33
    Recommendations for Executive Action 35 Agency Comments and Our Evaluation
    35

    Appendix I Scope and Methodology 37

    Appendix II Chronology of Pilot Project Litigation 40

    Appendix III Performance Standards for Hog Plants 43

    Appendix IV Survey Results 44

    Appendix V Australia?s Meat Safety Enhancement Program 59 Background and
    Trends That Led to the Program 59 How the MSEP System Works 60 Prerequisites
    for Plants to Participate in MSEP 61 AQIS? Evaluation of Its Program 62

    Appendix VI Canada?s Modernized Poultry Inspection Program 63 Background 63
    How the MPIP Pilot Works 63 Prerequisites for Plants to Participate in the
    MPIP Pilot 64 Performance Standards 65 Contents

    Page ii GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    MPIP Pilot Success Attributed to Extensive Training of Inspectors and
    Industry 66 Under MPIP Pilot, CFIA Officials Monitor Plant Performance

    Trends 67 Current Status of MPIP 68

    Appendix VII Comments From the U. S. Department of Agriculture 69 GAO?s
    Comments 76

    Appendix VIII GAO Contact and Staff Acknowledgments 81 GAO Contact 81
    Acknowledgments 81

    Tables

    Table 1: Number and Types of Pilot Plants That Participated in Traditional
    and Modified Inspections 11 Table 2: Traditional Inspection System?s Results
    (Baseline Data)

    From 16 Chicken Pilot Plants: Defects per Shift in Percents 13 Table 3:
    Performance Standards for Inspection Systems at Chicken

    Plants in the Pilot Project 26 Table 4: Eleven Chicken Plants? Inspection
    Systems Compliance

    With Performance Standards (RTI?s Data) 27 Table 5: Eleven Chicken Plants?
    Inspection Systems Performance

    After Shifting to Modified Inspections (RTI?s Data) 28 Table 6: Ten Chicken
    Plants? Inspection Systems Compliance With

    Performance Standards (FSIS? Data) 29 Table 7: Changes in Chicken Plants?
    Inspection Systems

    Performance Over Two Time Periods After Shifting to Modified Inspections
    (FSIS? Data) 30 Table 8: Summary of Responses to GAO?s Survey of USDA

    Inspectors and Veterinarians 31 Table 9: Performance Standards for
    Inspection Systems at Hog

    Plants in the Pilot Project 43 Table 10: Division of Responsibility Under
    Australia?s Meat Safety

    Enhancement Program 60

    Page iii GAO- 02- 59 Meat and Poultry Pilot Inspection Project Figures

    Figure 1: Traditional and Modified Inspection Systems on a Chicken Slaughter
    Line 8 Figure 2: Location of Chicken Pilot Plants 18 Figure 3: Fecal
    Noncompliance Records Under Traditional and

    Modified Inspections 23

    Abbreviations

    CCP Critical Control Point CDC Centers for Disease Control and Prevention
    FSIS Food Safety and Inspection Service HACCP Hazard Analysis and Critical
    Control Point HIMP HACCP- Based Inspection Models Project RTI Research
    Triangle Institute USDA U. S. Department of Agriculture

    Page 1 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    December 17, 2001 The Honorable Tom Harkin Chairman The Honorable Richard G.
    Lugar Ranking Minority Member

    Committee on Agriculture, Nutrition, and Forestry United States Senate

    While the food supply in the United States is generally safe, foodborne
    bacteria, viruses, and other pathogens can cause a significant number of
    illnesses, a small fraction of which are life- threatening. According to
    estimates from the Centers for Disease Control and Prevention (CDC),
    foodborne diseases cause 76 million illnesses in the United States each
    year, including 325,000 hospitalizations and 5,000 deaths. Meat and poultry
    products contaminated with bacteria such as Salmonella, Campylobacter,
    Escherichia coli (E. coli), and Listeria cause most foodborne illnesses and
    deaths. To control the spread of foodborne illnesses through meat and
    poultry products, approximately 3,400 U. S. Department of Agriculture (USDA)
    inspectors at 1,300 slaughter plants are stationed along slaughter lines to
    provide continuous inspection and conduct organoleptic examinations- using
    sight, touch, and smell- of each and every carcass.

    In 1997, USDA announced the need to modify its meat and poultry slaughter
    inspection program to make industry more responsible for identifying carcass
    defects. This approach is consistent with the agency?s previous adoption of
    the Pathogen Reduction: Hazard Analysis and Critical Control Point (HACCP)
    regulations. The HACCP approach is risk- based and makes industry, rather
    than federal inspectors, responsible for identifying steps in food
    production where food safety hazards are most likely to occur and for
    establishing controls that prevent or reduce it. USDA had not extended the
    HACCP principles to slaughter inspections because the agency has
    traditionally provided continuous inspection of each and every carcass.
    However, USDA believes that changing its traditional inspection system would
    also reduce inspectors? reliance on organoleptic inspections, allow for a
    shift to prevention- oriented inspection systems based on risk, and permit
    redeployment of its resources to better protect the public from foodborne
    diseases.

    Before making a permanent change to its slaughter inspections system, USDA
    developed a model to test whether such a change would continue to ensure the
    safety of meat and poultry products. This model is being tested

    United States General Accounting Office Washington, DC 20548

    Page 2 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    at several volunteer chicken and hog plants. At plants participating in the
    project, plant personnel, instead of USDA inspectors, examine each carcass
    to make an initial determination whether it is unacceptable and should be
    removed from the slaughter line. At the participating plants, USDA first
    measures the performance of traditional inspection systems and then compares
    it with the performance of modified inspections. USDA will propose
    regulations to modify its inspection system nationwide if the project shows
    that the modified inspection system performs as well as or better than
    traditional inspections. A reduced number of USDA inspectors are still at
    each plant to verify that safety and quality standards are met. In addition,
    at least one USDA inspector is positioned near the end of each slaughter
    line in order to comply with a federal court ruling that USDA cannot
    delegate the inspection of carcasses to plant employees. As of July 2001,
    the federal cost associated with this project was approximately $5.7
    million, and USDA estimates that it will spend another $1.2 million.
    Australia and Canada have tested similar modified inspection programs for
    meat and poultry and approved their use on a nationwide basis.

    Concerned about the design, methodology, and reliability of the pilot
    project, you asked us to (1) describe the objectives, design, and scope of
    the project; (2) identify limitations, if any, in the project?s design and
    methodology; and (3) any design and methodology limitations notwithstanding,
    determine if the data generated by the project will allow USDA to reach
    valid conclusions on the relative effectiveness of modified and traditional
    inspection methods in ensuring food safety and quality. In addressing these
    objectives, you also asked us to obtain information from similar projects in
    Australia and Canada.

    In response to your request, we reviewed USDA?s pilot project as implemented
    at chicken and hog plants. However, we were able only to analyze data from
    chicken plants because complete data from hog plants were not yet available.
    We also visited Australia and Canada to learn about similar pilot projects.
    Appendix I describes our methodology in detail.

    USDA?s original objective in implementing the pilot project was to test
    whether a prevention- oriented inspection system that uses plant personnel
    to examine each carcass for safety and quality and USDA inspectors to verify
    that safety and quality standards are met can provide a level of product
    safety and quality equal to or better than traditional inspections. At 11
    chicken and 3 hog plants that are voluntarily participating in the before
    and after phases of the project, plant personnel, instead of USDA
    inspectors, initially determine which carcasses and parts are unacceptable
    Results in Brief

    Page 3 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    and should be removed from the slaughter line because they are diseased or
    unwholesome. USDA is, therefore, able to use fewer inspection personnel at
    these plants. Using safety and quality performance standards developed for
    the project, an independent contractor measured how well pilot plants?
    inspection systems performed against these standards. The contractor
    measured how well USDA inspectors identified carcass defects under
    traditional inspections and then conducted similar measurements once plant
    personnel assumed those duties. Also, as a part of the pilot project, USDA
    inspectors at the project plants have examined carcasses to evaluate how
    well plant employees detect carcass defects. USDA will analyze these data to
    determine if at least the same level of safety and quality is maintained
    under the modified inspections. USDA has announced that preliminary results
    from the chicken pilot project show that plants? inspection systems perform
    better under the modified inspections, and that it expects to propose
    regulations to modify its slaughter inspection system for all chicken plants
    early in 2002. Both Australia and Canada have recently tested and adopted
    modified inspection programs that resemble USDA?s pilot project in that they
    include the replacement of some government inspectors with plant personnel.

    USDA?s pilot project for chickens has several design and methodology
    limitations that compromise the overall validity and reliability of its
    results. Hence, it is questionable whether the data generated by the project
    are indicative of how all of the chicken plants? inspection systems would
    perform if modified inspections were adopted nationwide. First, the chicken
    pilot that USDA designed lacks a control group- a critical design flaw that
    precludes a comparison between the performance of the inspection systems at
    those plants that volunteered to participate in the pilot and that of plants
    that did not participate. Without a control group, USDA cannot determine
    whether changes in inspections systems are due to personnel changes or other
    possible explanations, such as the addition of chlorine rinses. Second, the
    chicken plants that volunteered to participate in the baseline measurement
    phase of the pilot were not randomly selected, and they did not include
    plants from all chickenproducing areas or plants of all sizes. Thus, the
    results cannot be generalized to the entire population of chicken slaughter
    plants in the United States. Third, the pilot project?s methodology did not
    take into account variables such as seasonal changes and plant modifications
    that could affect project results. For example, after the project began,
    many plants added antimicrobial rinses and washers, which usually reduce the
    levels of microbial contamination. USDA acknowledges some of the project?s
    limitations but maintains that the design is consistent with that of

    Page 4 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    other USDA studies and that it is difficult to conduct this type of
    experiment without relying on volunteer participants. Finally, USDA?s pilot
    project did not include features of the modified inspection systems in
    Australia and Canada that would be important considerations in ensuring the
    successful implementation of a modified inspection system nationwide. For
    example, during the pilot project, USDA did not require the training of
    plant employees. Such training would be important in ensuring that plant
    personnel are as competent as federal inspectors in identifying carcass
    defects that can affect product safety and quality. Similarly, the pilot
    project did not require the use of statistical process controls as required
    in Australia and Canada. Statistical process controls provide a means to
    determine whether the plant?s production processes are performing within
    established performance standards. This technique would enable USDA and the
    plants to better measure and control their performance daily and over time
    to ensure continuous improvement. Additionally, USDA allowed plants with
    repeated noncompliance records to continue participating in the pilot
    project. If USDA decides to permanently modify its inspection system, a
    phased- in approach such as Canada?s would help to ensure that plants
    continually improve their ability to ensure product safety and quality and
    that those that have difficulty return to traditional inspections.

    Notwithstanding the project?s design problems, which we believe make the
    results unreliable, we found that, so far, the data themselves do not
    conclusively demonstrate that modified inspections are at least equal to
    traditional inspections. Part of the difficulty is that the two data sets
    collected for this pilot show somewhat different results. These data show
    whether the pilot plants met the seven organoleptic performance standards-
    two food safety standards and five other consumer protection (quality)
    standards- that USDA developed to compare, first the inspection systems?
    performance at plants under traditional inspections and then under modified
    inspections. The modified inspection systems at plants in the pilot project
    were expected to perform at least as well as they did under traditional
    inspections. However, the contractor?s data show that while the majority of
    the chicken plants? inspection systems met or exceeded four or more of the
    seven organoleptic standards adopted by USDA, none met all seven.
    Specifically, the contractor?s data show that most of the inspection systems
    at these plants did not meet the safety standard for the presence of fecal
    material, which could contain harmful bacteria such as E. coli. However, the
    traditional inspection system at these plants was also unable to meet this
    standard. Most inspection systems also did not meet quality standards for
    defects such as feathers and oil glands, but USDA officials pointed out that
    such defects are of a

    Page 5 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    lesser concern because they are not associated with foodborne illness and
    they can generally be removed from the carcass after it leaves the slaughter
    line and moves to the processing area. USDA?s contractor also measured
    whether the modified inspection systems at these plants met two other
    standards that existed before the pilot project- the pathogen reduction
    performance standard for Salmonella and generic E. coli. The data show that
    9 of the 11 inspection systems met the Salmonella standard and that 80
    percent of the samples collected for generic E. coli were within the
    acceptable ranges. On the other hand, the data collected by USDA inspectors
    at these chicken plants show better results than those shown by the
    contractor?s data. According to USDA?s data, 7 of 10 plants? inspection
    systems met 6 of the 7 standards. While the data generated by the pilot
    project yielded inconclusive results, the pilot project?s inspectors and
    veterinarians that we surveyed for the most part believe that a modified
    inspection system results in safer products. More than half of those
    surveyed said that modified inspections are equal to or somewhat better than
    traditional inspections in ensuring product safety and quality. Modified
    inspections allow for additional time to inspect slaughter line operations
    more thoroughly because inspectors, except for the carcass inspector, are no
    longer tied to a fixed location.

    This report reiterates our previous recommendation for legislative revisions
    aimed at reducing the potential for further legal challenges by providing
    USDA with clear authority to modify its inspection system. In addition, this
    report makes several recommendations to the Secretary of Agriculture aimed
    at ensuring that, if USDA decides to implement a modified inspection system,
    such a system will effectively ensure product safety and quality. In
    commenting on a draft of this report, USDA said that the report?s
    recommendations for executive action are appropriate and that the issues
    noted in the report can be satisfactorily addressed. USDA also provided
    technical comments which we incorporated as appropriate.

    The Federal Meat Inspection Act and the Poultry Products Inspection Act, 21
    U. S. C. sections 604 and 455, respectively, give USDA overall
    responsibility for ensuring the safety and wholesomeness of meat and poultry
    products that enter interstate commerce. Acting under these legislative
    authorities, USDA has engaged in continuous government inspection of each
    and every carcass at slaughter plants throughout the United States. Within
    USDA, the Food Safety and Inspection Service (FSIS) is responsible for
    inspections at all meat and poultry slaughter and processing plants and for
    ensuring plants? compliance with regulatory requirements. Background

    Page 6 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    At slaughter plants, FSIS inspectors are located at fixed stations and
    examine carcasses to identify and remove defects that impair product safety
    and quality. Defects that are likely to present a public health risk, such
    as the presence of potentially pathogenic fecal material, are considered
    food safety hazards. Product quality defects, such as feathers or bruises on
    the carcasses, are aesthetic flaws that rarely present a direct safety risk.

    Inspectors perform different tasks, depending on the animal species being
    slaughtered. For example, in chicken plants, the inspectors are located at
    fixed stations immediately after the place on the slaughter line where
    carcasses are opened and the organs removed. From these positions, the
    inspectors examine the exterior, interior cavity, and organs of each
    carcass. The inspectors then determine which carcasses must be condemned,
    which have conditions that may be trimmed, and which may proceed to the next
    stage in the slaughter line. Carcasses are then trimmed, rinsed, and, as the
    final step of the slaughter line, placed into a chiller. From here,
    carcasses move to the processing area of the plant, where they are cut up
    and packaged. In contrast, in hog plants, the inspectors are located at
    three fixed stations on the slaughter line. At the head station, the
    inspectors excise and evaluate the head lymph nodes; at the viscera station,
    they evaluate most organs; and at the final carcass station, they examine
    each carcass for possible contamination, disease conditions, and other
    adulterants before it goes into the chiller or cooler.

    For a variety of reasons, including responding to recommendations from the
    National Academy of Sciences and GAO that FSIS shift its resources to a
    prevention- oriented, risk- based inspection system, in 1996 USDA embarked
    on regulatory reform efforts to reduce foodborne hazards in meat and poultry
    products. The agency adopted a new science- based process control system,
    the Pathogen Reduction: Hazard Analysis and Critical Control Point system.
    HACCP is a risk- based approach that makes industry, not federal inspectors,
    responsible for identifying steps in food production where contamination is
    most likely to occur and for establishing controls that prevent or reduce
    contamination.

    USDA anticipated that the HACCP framework would also require changes in the
    roles and responsibilities of its inspection workforce, as industry, not
    government, becomes increasingly responsible for the safety of its products.
    For slaughter line operations, however, making industry more directly
    responsible for the safety and quality of their products was problematic
    because of a legal requirement for continuous carcass- bycarcass inspection.
    USDA believed that, to more fully integrate the HACCP

    Page 7 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    concept into meat and poultry slaughter operations, its traditional
    inspection system would have to be modified to transfer some of the carcass
    defect detection responsibilities to industry personnel. Before implementing
    such an approach, FSIS decided to conduct a pilot to test and evaluate
    whether product safety and quality could be maintained at a comparable level
    if plant employees perform the carcass defect detection duties and a reduced
    number of federal inspectors remain to verify product safety and quality
    through increased testing and observation.

    The pilot project is known as the HACCP- Based Inspection Models Project
    (HIMP) and its original design was intended to measure the effect of
    completely removing FSIS inspectors from their fixed locations on the
    slaughter line while maintaining one oversight and one verification
    inspector. In April 1998, the inspectors? union filed suit in the United
    States District Court for the District of Columbia to enjoin the USDA from
    proceeding with the pilot project, on the grounds that it violated the
    Federal Meat Inspection Act, 21 U. S. C. section 604, and the Poultry
    Products Inspection Act, 21 U. S. C. section 455, by not requiring federal
    government officials to perform carcass- by- carcass postmortem inspections.
    In June 2000, the United States Court of Appeals for the District of
    Columbia Circuit ruled that delegating the task of inspecting carcasses to
    plant employees violated the acts because both statutes require that federal
    inspectors, rather than private employees, determine whether a product is
    adulterated. As a result of this ruling, in September 2000, FSIS redesigned
    the pilot project for chickens and hogs and placed at least one FSIS
    inspector back at a fixed location on each slaughter line to inspect each
    carcass.

    Following a district court determination in January 2001 that the redesigned
    pilot project does not violate the acts, the inspectors? union appealed in
    February 2001. That appeal is still pending. A detailed description of the
    litigation surrounding the pilot project is contained in appendix II.

    Figure 1 depicts inspection responsibilities under FSIS? traditional and
    modified inspections on a chicken slaughter line.

    Page 8 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Figure 1: Traditional and Modified Inspection Systems on a Chicken Slaughter
    Line

    Note: Under the traditional inspection system, the slaughter line?s speed at
    most chicken plants is 91 birds per minute. Each inspector is responsible
    for about 35 birds per minute. Therefore, three FSIS inspectors are required
    to inspect carcasses on a slaughter line that runs at a speed of 91 birds
    per minute. The figure above shows only one FSIS inspector?s station in
    detail. Under the pilot project, a plant employee (sorter) replaces the FSIS
    inspector who examined the carcasses to determine their condition.

    As figure 1 shows, under traditional inspection systems, three FSIS
    inspectors are present at fixed locations on the slaughter line. In
    contrast, at chicken plants that participate in the pilot project, one FSIS
    inspector is present at a fixed location on each slaughter line, while a
    verification inspector monitors the entire line. The verification inspector
    is free to move along the slaughter line to continuously observe and
    evaluate the plant?s implementation of its HACCP system and process
    controls. This inspector is also responsible for randomly selecting and
    examining 80 carcasses per line per shift to verify that the plant is
    complying with the performance standards for food safety and quality. A
    slaughter line

    Page 9 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    averaging 91 chickens per minute would process over 43,000 chicken carcasses
    during each shift. The carcass inspector is located at the end of the
    slaughter line, immediately before carcasses enter the chiller. This
    inspector visually examines each carcass to comply with a court mandate that
    each carcass should receive FSIS inspection. At participating hog plants,
    carcass inspectors are positioned at the head, viscera, and final carcass
    inspection stations. Most of the inspectors and some veterinarians at the
    pilot project plants received promotions.

    The pilot project?s hypothesis was that a prevention- oriented inspection
    system that uses plant personnel to examine each carcass can provide a level
    of product safety and quality equal to or better than traditional
    inspections. At selected chicken and hog plants that voluntarily agreed to
    participate in the project, plant personnel, instead of FSIS inspectors,
    initially determine which carcasses and parts are unacceptable and should be
    removed from the slaughter line. Under this project, FSIS is able to use
    fewer inspection personnel at each plant. FSIS used an independent
    contractor to collect and analyze organoleptic and microbial data under the
    traditional inspection system first and then under the modified inspection
    system. FSIS developed organoleptic performance standards to measure how
    well the inspection system at participating plants performed after shifting
    from traditional to modified inspections. In addition, as part of their
    duties, FSIS inspectors continued to collect organoleptic performance data
    at these plants. Recently, both Australia and Canada tested and adopted
    modified inspection programs that resemble USDA?s pilot project. Pilot
    Project Makes

    Industry More Responsible and Accountable for Product Safety and Quality,
    While Continuing Government Monitoring With Fewer Resources

    Page 10 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    To initiate the pilot project, FSIS invited meat and poultry plants that
    slaughter young, healthy animals- market hogs, fed cattle, and young poultry
    (including young turkeys)- to volunteer to participate in the project. 1
    Sixteen chicken, 5 hog, and 5 turkey plants initially volunteered to
    participate. 2 No cattle plants volunteered. In general, plant managers that
    volunteered to participate in the pilot said that they elected to do so
    because they believe they can enhance product safety and quality if they
    have better control of slaughter line operations and, at the same time,
    increase productivity.

    FSIS contracted with the Research Triangle Institute 3 (RTI) to collect and
    analyze organoleptic and microbial data at plants that volunteered to
    participate in the pilot project. The purpose of this data collection is to
    evaluate whether the inspection systems at plants operating under modified
    inspections perform at least as well as the traditional inspection systems.
    In August 2000, RTI completed its collection of organoleptic and microbial
    data at 16 chicken, 5 hog, and 5 turkey plants under the traditional
    inspection. These baseline data documented the accomplishments of the
    traditional inspections system, and FSIS decided to adopt the results as
    performance standards for measuring the accomplishments of the same plants?
    inspection systems after they shifted to modified inspections. After a
    transition period during which plant personnel practiced their new roles,
    RTI again collected organoleptic and microbial data under the modified
    inspection system to provide a beforeand- after comparison. 4

    After RTI collected data on the performance of the traditional inspection
    system at the volunteer plants (baseline data), 5 of the 16 chicken plants

    1 FSIS designed the pilot project for meat and poultry plants that slaughter
    young animals because this group comprises approximately 90 percent of the
    animals slaughtered and those carcasses do not have complex pathology or
    other problems associated with older animals.

    2 Originally, 17 chicken plants volunteered to participate, but 1 of the
    chicken plants elected to drop out of the project shortly thereafter. 3 The
    Research Triangle Institute is an independent nonprofit organization that
    conducts multidisciplinary research on issues including health and
    pharmaceuticals, environment, and education and training. RTI?s contract
    with USDA for this pilot project contains modifications for increased future
    work.

    4 After FSIS placed at least one carcass inspector back on the slaughter
    line, RTI had to return to collect a new set of food safety and quality
    data. Under Pilot, Industry Is

    More Responsible and Accountable for Product Safety and Quality, While FSIS
    Provides Inspection Using Fewer Government Resources

    Page 11 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    and 2 of the 5 hog plants dropped out of the project. 5 As a result, RTI has
    collected modified inspections data at 11 chicken and 3 hog plants. These
    data were collected over a 5- to 6- week period. Some managers from plants
    that dropped out cited uncertainty about the future of the pilot project
    because of the court?s action, which resulted in modifications to the
    pilot?s original design.

    Ten additional chicken plants have since volunteered to participate in the
    pilot, bringing the total number of participating plants to 21, but RTI will
    collect data at only 5 of the additional 10 plants. According to FSIS, no
    traditional inspection performance data (baseline data) will be collected at
    these additional plants. Table 1 shows the number of slaughter plants that
    participated both in the baseline data collection phase (traditional
    inspections) and in data collection after they switched to modified
    inspections.

    Table 1: Number and Types of Pilot Plants That Participated in Traditional
    and Modified Inspections

    Species Plants that completed baseline data

    collection under traditional inspections

    Plants that completed data collection under modified

    inspections

    Chickens 16 11 (10) a Hogs 5 3 Turkeys 5 0 a One plant dropped out after it
    completed data collection under the modified inspection system.

    Source: USDA, FSIS.

    Because turkey plants have not participated in modified inspections and
    because data from hog plants under modified inspections are not yet
    available, this report discusses pilot project results from chicken plants
    only. (See app. III for information on the development of performance
    standards for hogs.)

    At the pilot project plants, FSIS has been able to use fewer inspection
    resources than would otherwise be used under traditional inspection systems,
    resulting in an overall 22 percent reduction (from 259 to 202 inspectors) in
    the number of FSIS inspectors on the slaughter floor. At most of the 14
    chicken and hog plants, the reduction of inspectors ranged

    5 All five turkey plants dropped out after the baseline data were collected.

    Page 12 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    from 1 and 10 inspectors, depending on the number of slaughter lines and the
    speed of those lines. Under traditional inspections, there were three
    inspectors per line on average, compared with two inspectors per line under
    modified inspections. At three of the pilot project plants, the change in
    the number of inspectors was not entirely due to the modified inspection
    system.

    To collect and analyze organoleptic data on the defects that the FSIS
    inspectors had overlooked under traditional inspections at chicken plants,
    RTI veterinarians at each volunteer plant examined approximately 2,000
    randomly selected carcasses that had passed FSIS? slaughter inspection. RTI
    veterinarians examined carcasses to record the same type of defects that
    FSIS inspectors identify when they examine each carcass. These included
    safety defects, such as visible fecal material and evidence of septicemia
    and/ or toxemia, and other consumer protection defects (quality defects)
    such as feathers, hairs, or bruises. FSIS grouped this information into two
    food safety and five other consumer protection (quality) categories:

     Food Safety 1: Infectious conditions such as septicemia and/ or toxemia.
    Food Safety 2: Contamination with fecal material.  Other Consumer
    Protection 1: Animal diseases such as airsaculitis.  Other Consumer
    Protection 2: Defects such as bruises and sores.  Other Consumer Protection
    3: Contamination with digestive tract contents

    (ingesta).  Other Consumer Protection 4: Dressing defects such as feathers
    and oil

    glands.  Other Consumer Protection 5: Dressing defects of the digestive
    tract tissue

    such as bursa and cloaca. FSIS then developed performance standards for each
    of these seven categories. The quality performance standards were set at
    position 12 (out of 16) on the basis of measurement of the traditional
    inspection system at the 16 chicken plants. That is, 25 percent of the
    plants? inspection systems (4 different plants) would have to improve on
    their baseline results in each of the 7 performance categories. FSIS
    officials explained that they decided to adopt the 12th position of the
    baseline results as the standard to ensure that most participants would be
    able to meet them. The officials stated that this represents a reasonable
    tightening of the traditional system?s accomplishments. The Pilot?s Scope
    Included

    the Development of Safety and Quality Standards to Measure Performance of
    Traditional Inspections as Compared With Modified Inspections

    Page 13 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    To better protect consumers from defects that may be harmful, FSIS set the
    two food safety standards at zero rather than adopting the results of the
    12th position. Thus, a performance standard of zero for fecal contamination
    means that FSIS expects all carcasses slaughtered during a shift to be free
    of fecal material. Conversely, an 80- percent performance standard for
    dressing defects such as feathers and oil glands means that if 80 of 100
    carcasses have these defects, the performance standard is met. The
    performance standards represent the percentage of carcasses per shift with
    each defect that FSIS considers acceptable. Table 2 shows the traditional
    inspection system?s baseline results for the 16 chicken plants.

    Table 2: Traditional Inspection System?s Results (Baseline Data) From 16
    Chicken Pilot Plants: Defects per Shift in Percents

    Note: Ranking numbers represent different plants for different food safety
    and OCP standards. Although the 12th position?s performance was 0.05 percent
    for Food Safety 1 and 1.5 percent for Food Safety 2, FSIS adopted a zero
    tolerance policy for these two standards. For Food Safety 2, the zero
    tolerance standard was used because it is already a regulatory requirement.

    Source: RTI.

    RTI also measured the prevalence of generic E. coli and Salmonella under
    traditional inspections and modified inspections at the 11 chicken plants to
    determine if changes in the inspection system would have an effect on the
    microbial profile of carcasses. RTI collected and analyzed 300 samples

    Page 14 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    for both microorganisms over a 6- week period. RTI analyzed the

    Salmonella samples to determine if they met FSIS? already existing
    regulatory standard for Salmonella and the generic E. coli samples to
    determine if the results were in the acceptable, marginal, and unacceptable
    ranges established by FSIS.

    FSIS is also collecting data on product safety and quality defects at
    chicken plants in the pilot project and is analyzing these data to compare
    how well traditional inspection systems perform versus modified inspection
    systems. FSIS inspectors collect the data as they conduct scheduled and
    unscheduled verification tests at each chicken plant. These tests evaluate,
    among other things, how well plant employees detect carcass defects. When
    the pilot project is concluded, FSIS plans to analyze these data to
    determine if at least the same level of safety and quality is maintained
    under the pilot program. FSIS has announced that preliminary results from
    the chicken pilot project show that inspection systems perform better after
    the plants shifted to modified inspections, and that it expects to propose
    regulations to modify its traditional slaughter inspection system for all
    chicken plants in 2002.

    The Australian Quarantine and Inspection Service and the Canadian Food
    Inspection Agency have tested and adopted alternative inspection systems
    similar to that tested in the FSIS pilot project. (See apps. V and VI for
    additional details.)

    Australia has tested and adopted a modified inspection system for all meat
    plants that produce products for domestic consumption. In addition,
    Australia is now in the process of implementing the Meat Safety Enhancement
    Program at plants that export meat. Under the program, plants are
    responsible for plant process controls, including sanitation, the microbial
    monitoring of pathogens, and detecting product defects. The program
    incorporates the use of government- licensed company employees, acting under
    the supervision and oversight of government veterinarians, to replace
    government inspectors. In order to be licensed by the government, the
    company?s employees are required to undertake about 600 hours of training
    and pass an examination. The Australian Quarantine and Inspection Service
    sets public health product standards and audits plant operations to ensure
    that food safety standards are met. A key feature of the program is the
    mandatory use of statistical process controls to track plants? performance
    over time. Pilot Projects in Australia

    and Canada Used a Similar Strategy

    Page 15 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Canada has adopted a new poultry inspection system, known as the Modernized
    Poultry Inspection Program that allows plant personnel to assume defect
    detection duties and remove defective carcasses from the slaughter line
    under the supervision of government inspectors. The program is being
    implemented in a phased approach. Performance standards for Canadian poultry
    plants participating in this program are similar but not identical to FSIS?
    standards for chicken plants in the U. S. pilot project. Like the Australian
    program, the Canadian program has stringent training requirements for
    industry personnel replacing inspectors. Industry personnel are required to
    demonstrate competency through testing and certification.

    The pilot project has several design and methodology limitations that
    compromise the overall validity and reliability of its results. For example,
    the lack of a control group prevents valid comparisons between the
    inspection system of participating plants and that of nonparticipating
    plants. In addition, the participating plants were not randomly selected;
    therefore, results from these plants cannot be generalized to the entire
    population. Finally, the pilot project does not appropriately explain how
    variables, such as seasonal changes and plant modifications, could affect
    the project?s results. In addition, the pilot project does not include
    important features of similar programs in Australia and Canada.

    FSIS officials characterized the design of the pilot project as a classic
    before- and- after experiment that is typical of other FSIS experiments. A
    classic before- and- after experiment involves both a control and an
    experimental group and thus provides a basis for comparing the results of
    the two groups. Ideally, the groups are as similar as possible, except for
    the variable that the experiment is attempting to measure. However, because
    this level of rigor is rarely achievable in applied settings,
    quasiexperimental designs are often used in which comparison groups are
    established to compare the impact of a new program with the status quo.
    According to our discussion with FSIS officials, the volunteer plants were
    not separated into a control group and an experimental group because the
    agency felt that it would have been unfair to exclude volunteer plants from
    taking part in the modified inspection aspect of the experiment.
    Nevertheless, under a quasi- experimental design, FSIS could have used as a
    control group the five chicken plants that decided to discontinue
    participation in the pilot project after their baseline performance under
    the traditional system was measured. The agency could have then compared the
    results from these plants with the results obtained from the Design and

    Methodology Limitations Compromise Overall Validity and Reliability of Pilot
    Project Results

    Pilot Project Lacks a Control Group

    Page 16 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    plants that participated in the modified inspections. Lacking such a
    comparison, it is difficult to determine how the inspection systems at the
    plants that did not participate in the pilot project would have performed
    under the performance standards established for the pilot project.

    The managers of two large chicken plants that dropped out of the pilot said
    that they proposed another approach to FSIS. Namely, these managers wanted
    to conduct what could have been a controlled experiment within their own
    plant. Both of these plants have multiple slaughter lines, and the managers
    wanted to operate half of their slaughter lines under traditional
    inspections and the other half under modified inspections. However, FSIS
    would not authorize such an arrangement because the agency wanted the entire
    plant to be under one inspection system. Officials said that it would have
    been too difficult to maintain two separate inspection systems in the same
    plant. As a result, both of these plants withdrew from the pilot project.

    The Australian pilot project compared three plants that volunteered to
    participate in the pilot project with three plants operating under
    traditional inspections, so the pilot had both an experimental group and a
    control group. The plants in the experimental group removed the government
    inspectors from the slaughter line and replaced them with plant personnel,
    while the control group plants made no changes to their inspection system.
    The pilot project measured both microbial and organoleptic data at plants in
    both groups to determine how well they complied with standards. The results
    were then compared in order to determine how the modified inspections
    affected food safety and quality. The Canadian Food Inspection Agency did
    not use a control group.

    Because the volunteer chicken plants were not randomly selected, the results
    cannot be generalized to the population of plants that slaughter young
    chickens. FSIS is not authorized to compel plants to participate in any
    pilot project; therefore, it had to rely on volunteer plants instead of
    randomly selecting participants. However, FSIS did have the option to
    randomly select a subgroup to serve as a control group from the volunteer
    plants, but decided not to pursue that option. Australia and Canada also did
    not select plants on a random basis.

    FSIS officials acknowledge that the number of plants in the pilot was not
    statistically derived. The officials believe, however, that the 16 chicken
    plants that originally volunteered constitute a sufficient number to
    estimate results for the population. However, only 11 chicken plants have
    Plants Participating Are

    Not Randomly Selected and Do Not Represent the Population

    Page 17 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    participated in all phases of the pilot project. These plants represent
    about 6 percent of the population of young chicken plants nationwide, or 11
    of 186 plants in 1999. We estimate that 125 randomly selected plants would
    need to participate in the project in order for FSIS to obtain results that
    can be generalized to the population of chicken plants. 6

    As a result of the pilot project?s reliance on volunteer plants, the results
    of the pilot do not provide sufficient geographic representation. That is,
    the 11 chicken plants that have participated in all phases of the pilot
    project are located in 7 of 35 states that produce young chickens, or about
    20 percent of the chicken- producing states. These plants are concentrated
    in the southern region of the United States and account for about 57 percent
    of U. S. young chicken production. (See fig. 2.) Unlike FSIS, the Canadian
    Food Inspection Agency designed its pilot to include chicken plants from the
    western, eastern, and central provinces. As a result, the Canadian pilot
    project has better geographic representation.

    6 This sample size is calculated to provide for results of plus or minus 5
    percent at the 95- percent confidence interval.

    Page 18 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Figure 2: Location of Chicken Pilot Plants

    Note: Numbers indicate how many chicken plants participated in the baseline
    data collection phase of the pilot project.

    Source: USDA, FSIS.

    In addition, there are eight large plants, three small plants, and no very
    small plants in the pilot project. According to FSIS, there are 112 large,
    61 small, and 13 very small plants that slaughter young chickens in the
    United States. We discussed with officials at all participating plants in
    the pilot project whether the plants represent the population of chicken
    plants in the United States. Most of them said that, in their opinion, the
    plants are representative of the industry nationwide. FSIS officials are
    also of the opinion that all plants that volunteered to participate are
    similar to nonparticipants in all respects except for the fact that they
    volunteered. However, since only volunteer plants participated in the pilot,
    potential selection bias is introduced because such plants may be
    predisposed to better performance.

    Page 19 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    The pilot project does not adequately control for the effect that seasonal
    variations and plant modifications can have on the its results. Because
    seasonal changes can cause animals to develop diseases and/ or poor health,
    two poultry science experts that we consulted said that it is imperative to
    collect samples over sufficiently long periods of time to cover all seasons.
    Seasonal variations significantly affect microbial results, especially in
    the case of bacteria such as Salmonella, which was measured in the pilot
    project. However, the pilot project?s design called for 6 consecutive weeks
    of microbial data collection. Accordingly, the experts we consulted believe
    that microbial data collected during the pilot project are of limited value
    because the data cover less than 2 months at each plant.

    The pilot project?s results may also be affected by another set of
    variables. Most of the plant managers at pilot project plants told us that
    they introduced changes to their slaughter lines to improve product safety
    and quality after the baseline performance of the traditional system was
    measured. These changes included the addition of new brushes and scrubbers,
    plain water washes, and antimicrobial washes. Some managers also modified
    their evisceration equipment and developed computerized systems to track
    carcass safety and quality defects. All the inspectors and veterinarians at
    pilot project plants that we surveyed indicated that at least one change had
    been made after modified inspections were implemented at the plant. In
    effect, it is not possible to discern whether the before- andafter results
    measured by the two food safety and five quality standards are to be solely
    attributed to the shift from traditional to modified inspections at these
    plants.

    According to FSIS officials, the pilot project was designed to measure the
    accomplishments of the traditional inspection system and the accomplishments
    of the models inspection system allowing for measuring multiple process
    changes- not isolated comparisons between FSIS inspector performance and
    plant employee performance. If this is the case, then the pilot project
    should have included the measurement of all the process changes introduced,
    not just the only change introduced by the agency; namely, the transfer of
    carcass defect detection from FSIS inspectors to plant personnel. We
    understand that, at their initiative, plants decided to make changes to
    their process during this pilot. However, FSIS did not collect data on what
    changes were made or on what the effect of those changes may be on the
    overall pilot project results.

    USDA officials disagree that the pilot project does not take into account
    variables that could affect its results. They stated that, if the results
    from Pilot Project?s

    Methodology Does Not Take Into Account Variables That Could Affect Results

    Page 20 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    all plants are viewed collectively, their sample extended over a sufficient
    period of time to consider the effect of seasonal variations, evisceration
    line effects, and other non- quantifiable variables. Even if all plants are
    considered collectively, FSIS is still missing microbial data for the entire
    year. According to officials, from January through April, no data were
    collected at any of the pilot plants. Moreover, according to the experts we
    consulted, FSIS should have a year?s worth of microbial data from each plant
    in the pilot to be able to account for seasonal variations.

    In discussing these limitations with USDA officials, they explained that the
    focus of the project is studying an entire system, including all changes
    made by the plants, and not simply comparing the work of company employees
    with that of government inspectors. However, we note that the project?s
    stated objective was to determine whether a modified inspection system is at
    least as good as the traditional system. In order for such a comparison to
    be valid, USDA would have to determine that other variables introduced into
    the slaughter line, such as the addition of rinses or brushes, are not the
    cause of changes noted after the shift in inspection systems took place.

    The pilot project did not address three important features of similar
    programs in Australia and Canada. Those features would help to ensure that
    plant personnel are able to discharge their carcass defect detection duties
    as well as federal inspectors and that plants are capable of continuing to
    ensure the safety and quality of their products under a modified inspection
    system. First, the pilot project did not require that, prior to assuming
    their new duties, plant personnel receive any training and pass competency
    tests. Second, the pilot project did not require the use of statistical
    process controls as a means to ensure that plants continually improve their
    processes under the modified inspection system. Third, the plants continued
    in the pilot project irrespective of repetitive noncompliance with
    regulatory requirements.

    FSIS did not require that plant employees complete training before assuming
    the carcass defect detection duties that FSIS inspectors? performed.
    Furthermore, FSIS did not establish a way to measure plant employee?s
    knowledge and competence. FSIS officials told us that training for plant
    personnel will not be required if the system is adopted nationwide. Plant
    personnel and FSIS inspectors at each pilot project plant did participate,
    however, in a transition phase during which they practiced their new roles.
    FSIS officials told us that the transition phase was also Pilot Project
    Lacks Key

    Elements Project Does Not Require Formal Employee Training and Certification

    Page 21 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    used to explain how and when data were to be collected during the project.

    If FSIS implements modified inspections nationwide, plant personnel will be
    assuming carcass defect detection duties previously performed by FSIS
    inspectors who have received formal training on how to identify carcass
    defects and determine whether they are suitable for human consumption.
    However, unless FSIS institutes training requirements, plant personnel will
    not have received comparable training. We asked plant managers participating
    in the pilot project about the training that they have provided for plant
    employees. They replied that they provided various levels of training.
    Individual plants have sole discretion for determining the type of training
    provided for their employees as FSIS does not approve or monitor plants?
    training programs. According to plant officials, FSIS is supportive but does
    not endorse a training program that the International HACCP Alliance
    developed for plant personnel participating in this pilot project. 7 In some
    instances, the Alliance, not FSIS, has accredited plants? training programs
    but it does not certify trainees who complete these courses.

    Seven of the 14 pilot project managers whom we interviewed support a
    training program for plant personnel. About two- thirds of the managers also
    were in favor of developing a joint training program between industry and
    FSIS. At nine pilot project plants, plant employees were required to pass a
    test before being assigned to their new jobs, and seven plants adopted some
    kind of continuous on- the- job training. The managers told us that
    classroom training for plant employees ranged from 4 to 80 hours. The
    duration of on- the- job training ranged from 8 to 160 hours.

    In contrast, the Australian modified inspection program requires plant
    personnel to complete about 600 hours of classroom training as a
    prerequisite to participation in the modified inspection program. Plant
    personnel must also pass a competency test and be certified. Upon completion
    of the on- the- job training, state authorities license the plant personnel.
    The Australian government?s veterinarian- in- charge at each plant is
    authorized to withdraw or deny license renewal of plant personnel on the
    basis of performance.

    7 The International HACCP Alliance membership includes over 120 members from
    nine countries representing industry associations, professional
    associations, educational foundations, universities, third party/ private
    companies, and government cooperators. Their mission is to promote
    international public health and safety by facilitating uniform development
    and implementation of HACCP programs from farm to table.

    Page 22 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Similarly, under Canada?s modified inspection program, government inspectors
    train plant personnel. The required training includes standardized classroom
    training, in- plant training, a final examination, and a demonstration of
    competency on verification tests and finished product standards. At the
    completion of the training, the Canadian Food Inspection Agency accredits
    plant employees. Plant employees must be reaccredited every 6 months, and
    the agency?s inspectors reserve the right to conduct additional
    reaccreditation of plant employees at any time.

    Sixty- seven percent of the FSIS inspectors and veterinarians whom we
    surveyed believe FSIS should develop a training program that includes
    mandatory testing and certification of plant employees.

    Although FSIS has encouraged plants in the pilot project to use statistical
    process controls, it is not a requirement of the pilot. However,
    threequarters of the inspectors and veterinarians responding to our survey
    indicated that their plants use statistical process controls. About a third
    of these respondents believe that the use of statistical process controls
    have the effect of making food safety and quality better, while a quarter of
    them believe that it does not make a difference. At the pilot project
    plants, FSIS inspectors do not review or analyze plant data using
    statistical process controls. However, by applying process controls, FSIS
    could obtain a more comprehensive assessment of the effectiveness of the
    modified inspection system because it could review the performance of an
    individual plant?s data to determine whether the plant?s systems are
    continually improving over time.

    In contrast, the Australian and Canadian food inspection agencies use
    statistical process control methods to audit a plant?s performance over
    time. This method entails verification by the plant?s management that its
    production process is stable, that it is capable of producing products that
    meet performance standards, that it takes appropriate actions if changes
    occur in the process before it results in unacceptable products, and that it
    takes actions to continuously improve its process. Project Does Not Require
    Use

    of Statistical Process Controls

    Page 23 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    FSIS allowed all plants to continue participating in the pilot project even
    though several had multiple instances of noncompliance with regulatory
    requirements for fecal contamination. 8 Our analysis of noncompliance
    records issued by FSIS inspectors at 11 chicken pilot plants showed an
    increased number of such records after the plants shifted to modified
    inspections for fecal noncompliance. In two instances, however, FSIS took
    additional action and notified plants of problems with repeated
    noncompliance. FSIS officials told us that they are now considering how many
    repeated instances of noncompliance will result in further regulatory
    action.

    Figure 3: Fecal Noncompliance Records Under Traditional and Modified
    Inspections

    Note: Data are provided for a 12- month period before and after the pilot
    project?s implementation. However, at some plants, data reported after the
    implementation of the pilot project covered periods of less than 12 months,
    data for plant 7 cover 8 months, data for plants 1 and 10 cover 10 months,
    and data for plant 9 cover 11 months. It is likely that a full twelve months
    of data for these plants would result in an even higher number of
    noncompliance records.

    Source: USDA, FSIS.

    As figure 3 shows, the number of noncompliance records resulting from fecal
    material increased significantly at several plants after they shifted from
    traditional inspections to modified inspections. At one plant, FSIS

    8 FSIS inspectors document each instance of noncompliance with regulatory
    requirements in a noncompliance record that plant managers must address
    immediately. If this is done, the plant continues to operate without
    interruption. When deficiencies occur repeatedly, FSIS may take further
    action to withhold products or to suspend inspections. If FSIS suspends
    inspections, the plant cannot operate. Project Does Not Fully Take

    Into Account Plants? Repeated Instances of Noncompliance With Regulatory
    Requirements

    Page 24 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    inspectors issued 469 fecal material noncompliance records during a 10-
    month period after the plant shifted to modified inspections. Many
    noncompliance records reflect multiple violations recorded within the same
    record. At one plant, FSIS inspectors reported that over 90 percent of the
    fecal material noncompliance records reflected repetitive failures.

    FSIS officials acknowledge the rise in the number of noncompliance records
    and attribute it to the fact that at pilot project plants, FSIS inspectors
    have more time to verify compliance with regulatory requirements and to
    document instances of noncompliance when they occur. The officials also
    point out that, as part of the their verification duties, FSIS inspectors
    now examine 80 carcasses per line per shift compared with 20 carcasses under
    the traditional inspection system. 9 In addition, unscheduled verification
    tests of additional carcasses are conducted under each inspection system.
    Officials also told us that staffing shortages existed under traditional
    inspections and that in some instances, it was not possible to complete
    verification activities on 20 carcasses for each line. Even accounting for
    the fourfold increase in the number of verification checks taking place
    under the modified inspections system, one plant experienced a more than
    twentyfold increase in the number of noncompliance records for fecal
    contamination. At other plants, however, the number of noncompliance records
    did not even double.

    Another factor may also account for the increases in noncompliance records
    issued for fecal failures. Although inspectors and veterinarians working at
    pilot project plants who responded to our survey were not comparing line
    speeds to those of the traditional inspection system, about 60 percent of
    them think that line speeds are too fast under modified inspections to
    ensure product safety. For example, one inspector stated that fast line
    speeds cause inexperienced employees to miss carcasses with visible fecal
    contamination. FSIS does not regulate line speeds under this pilot. However,
    under traditional inspections, each inspector is responsible for about 35
    carcasses per inspector per minute. Canadian inspectors told us that line
    speeds need to be regulated under modified inspection systems.

    9 Each carcass is inspected under both traditional and modified inspections.
    In addition, under both systems, inspectors also select a sample of
    carcasses at the end of each line to verify that the inspection system is
    working as intended.

    Page 25 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    In contrast to FSIS, the Canadian Food Inspection Agency decided to proceed
    with a phased approach to nationwide implementation of Canada?s modified
    inspection program, the Modernized Poultry Inspection Program. Under this
    approach, poultry plants must pass a preliminary assessment before being
    accepted into the program and are required to pass additional audits as they
    move from one phase of the program to the next. Producers that choose not to
    participate in or do not qualify for a partially modified or fully modified
    inspection system are inspected under the traditional system.

    Notwithstanding the design and methodology limitations, we found that data
    from the chicken pilot that RTI and FSIS inspectors collected show somewhat
    different results and do not provide a conclusive basis for FSIS to make a
    decision regarding the merits of one inspection system compared with
    another. However, 71 percent of the FSIS inspectors and veterinarians that
    we surveyed believe that product safety is equal to or somewhat better under
    modified inspections. Some of them commented that under the modified system,
    they are able to oversee the entire slaughter line and have more time to
    collect carcass samples for detailed examination. However, they also raised
    concerns about the modified inspection system.

    RTI?s data do not conclusively show that inspection systems at chicken
    plants perform as well as or better than they did after they switched to
    modified inspections. Under modified inspections, none of the inspection
    system at these plants met all of the performance standards that FSIS
    developed to measure their performance. Table 3 shows the performance
    standards that the inspection systems at these plants were expected to meet
    after shifting to modified inspections. 10

    10 These standards, set by data gathered at the 16 plants under the
    traditional inspection system, reflect the performance level of the 12th
    position. (See p. 13.) Pilot Project Results

    Are Inconclusive, but FSIS Inspection Personnel Generally Support Modified
    Inspections

    Contractor?s Data Do Not Conclusively Show That Modified Inspection Systems
    at Chicken Pilot Plants Performed ?As Well As or Better Than? Traditional
    Inspection Systems

    Page 26 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Table 3: Performance Standards for Inspection Systems at Chicken Plants in
    the Pilot Project

    Categories Performance standards

    (percentage of defects allowed per shift)

    Food Safety (FS) 1: Infectious condition such as septicemia and toxemia 0
    FS2: Digestive content contamination (visible fecal material) 0 Other
    Consumer Protection (OCP) 1: Animal diseases such as airsaculitis 1.7 OCP2:
    Miscellaneous such as bruises, sores, etc. 52.5 OCP3: Digestive content
    contamination (ingesta) 18.6 OCP4: Dressing defects such as feathers and oil
    glands 80.0 OCP5: Dressing defects of digestive tract such as bursa and
    cloaca 20.8

    Source: USDA, FSIS.

    Data from the inspection systems at the 11 chicken plants whose performance
    RTI measured under both inspection systems show that, while the majority of
    the systems met four or more of the performance standards for food safety
    and quality under modified inspections, none met all seven of the standards.
    11 The data show that 10 of the 11 plants? inspection systems met the food
    safety standard for septicemia and/ or toxemia, but only 1 of the 11 systems
    met the zero- tolerance food safety standard for visible fecal material,
    which could contain harmful bacteria such as E. coli. Most plants?
    inspection systems, however, were also unable to meet the fecal material
    standard under the traditional inspection system. All 11 systems at these
    plants met the quality standard for bruises and sores (OCP2), but only 3 met
    the quality standard relating to the presence of feathers (OCP4). According
    to FSIS officials, carcasses with product quality defects are of a lesser
    concern than those with food safety defects because they are not associated
    with foodborne illness and quality defects can generally be removed from the
    carcass after it leaves the slaughter line and moves to the cut- up and
    processing area.

    11 We analyzed the statistical significance of inspection system changes at
    each plant using the 95- percent confidence level.

    Page 27 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Three of the 11 plants? inspection systems met six of the seven standards, 3
    plants met five of the standards, 4 plants met four standards, and 1 plant
    met two standards. Table 4 shows which systems met the performance standards
    that FSIS set for the pilot project.

    Table 4: Eleven Chicken Plants? Inspection Systems Compliance With
    Performance Standards (RTI?s Data) Food safety standards Food quality
    standards (in percent)

    Plant FS1:

    septicemia and/ or toxemia (zero allowed)

    FS2: visible fecal material (zero allowed)

    OCP1: animal disease, e. g., airsaculitis (1.7 allowed)

    OCP2: bruises, sores, etc. (52.5 allowed) OCP3: ingesta

    (18.6 allowed) OCP4:

    feathers, oil glands, etc. (80.0 allowed)

    OCP5: bursa, cloaca, etc. (20.8 allowed)

    1 Yes No Yes Yes Yes No Yes 2 Yes No Yes Yes Yes No No 3 Yes No Yes Yes Yes
    No No 4 Yes No Yes Yes Yes Yes Yes 5 Yes No Yes Yes Yes Yes Yes 6 Yes No Yes
    Yes Yes No No 7 Yes No Yes Yes Yes No No 8 Yes Yes Yes Yes Yes No Yes
    9NoNoYesYesNo No No 10 Yes No Yes Yes Yes No Yes 11 Yes No Yes Yes No Yes
    Yes

    Plants that met standard

    10111 11 936

    Source: RTI.

    As discussed earlier, under its contract with FSIS, RTI collected microbial
    data as well. Specifically, it analyzed samples for the presence of

    Salmonella and generic E. coli. The results show that 9 of the 11 chicken
    plants? inspection systems met the performance standard for Salmonella.

    Also, about 80 percent of the samples collected at the11 plants were within
    the acceptable ranges that FSIS specifies for generic E. coli.

    Irrespective of whether plants? inspection systems met the performance
    standards, we analyzed these data to determine if, under modified
    inspections, the systems performed at least as well as they did under
    traditional inspections. Table 5 summarizes the results of this analysis.

    Page 28 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Table 5: Eleven Chicken Plants? Inspection Systems Performance After
    Shifting to Modified Inspections (RTI?s Data) Microbial standards Food
    safety standards Food quality standards

    Performance Changes Generic

    E. coli Salmonella

    FS1: septicemia

    and/ or toxemia

    FS2: visible

    fecal material

    OCP1: animal disease, e. g.

    airsaculitis OCP2:

    bruises, sores, etc. OCP3:

    ingesta OCP4:

    feathers, oil glands,

    etc. OCP5:

    bursa, cloaca

    , etc.

    Better 2 2 1 4 5 5 4 3 4 Same 9 4 10 7 5 3 1 1 1 Worse 0 5 0 0 1 3 6 7 6

    Total 11 11 11 11 11 11 11 11 11

    Source: GAO?s analysis of RTI?s data.

    The data show that some of the plants? inspection systems improved or
    remained unchanged in certain categories such as the food safety standard
    for septicemia and/ or toxemia. On the other hand, the systems? ability to
    meet the quality performance standards deteriorated over time. In general,
    most measures of performance showed deterioration in one or more plants. In
    fact, every inspection system showed deterioration for at least one of the
    nine standards measured. At 5 of the 11 plants, results for

    Salmonella were worse under modified inspections than they were under the
    traditional inspection system- only 2 improved their performance in this
    category. FSIS has stated that any new inspection system should perform at
    least as well as the current system of inspections.

    FSIS? data for 10 chicken plants in the pilot project also show that none of
    their inspection systems met all seven standards. 12 In contrast to the RTI
    data, however, the FSIS data show that 7 of the 10 systems met six of seven
    standards, and the remaining 3 plants met five of the seven standards. FSIS
    officials stated that the differences in results might be caused, at least
    in part, by differences in data collection. As part of FSIS inspectors?
    verification duties at the chicken plants participating in the pilot
    project, the inspectors sample and analyze carcasses to determine whether
    they meet performance standards. Inspectors are not required to collect and
    analyze microbial samples, as RTI was required to do for the pilot project.
    The data collected cover a longer period than RTI?s and include much larger
    samples per plant. The FSIS officials acknowledge,

    12 We analyzed FSIS data from 10 of the 11 plants at which RTI collected
    data for both traditional and modified inspections because one plant dropped
    out of the pilot shortly after RTI completed data collection. FSIS? Data Do
    Not

    Conclusively Show That Plants? Inspection Systems Perform ?As Well As or
    Better? Under Modified Inspections

    Page 29 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    however, that disparities in the FSIS and RTI data for some performance
    standards, such as dressing defects (e. g., feathers and oil glands),
    suggest a need for further investigation.

    The FSIS data show that 7 of the 10 chicken plants? inspection systems met
    six of the seven performance standards, but none of the plants met the zero-
    tolerance standard for fecal material over time. It should be noted,
    however, that the actual performance of the 12th position for this standard
    was 1.5 percent, not zero as required by FSIS? performance standard. So
    noncompliance with this standard may not provide a basis for comparing
    performance of the modified with traditional inspection systems. After
    shifting to modified inspections, each of the 10 plants? systems had reached
    a performance that was better than 1.5 percent for this standard. We
    analyzed data on individual inspection systems? performance from the time
    that each plant implemented modified inspections through June of 2001. Table
    6 shows a summary of the inspection systems? performance at individual
    plants.

    Table 6: Ten Chicken Plants? Inspection Systems Compliance With Performance
    Standards (FSIS? Data) Food safety standards Food quality standards

    Plant FS1:

    septicemia and/ or toxemia FS2: visible

    fecal material OCP1: animal

    disease, e. g., airsaculitis)

    OCP2: bruises, sores, etc. OCP3:

    ingesta OCP4:

    feathers, oil glands, etc. OCP5: bursa,

    cloaca, etc.

    1 Yes No Yes Yes Yes Yes Yes 2 Yes No Yes Yes No Yes Yes 3 Yes No Yes Yes
    Yes Yes Yes 4 Yes No Yes Yes Yes Yes Yes 5 Yes No Yes Yes Yes Yes Yes 6 No
    No Yes Yes Yes Yes Yes 7 Yes No Yes Yes Yes Yes Yes 8 Yes No No Yes Yes Yes
    Yes 9 Yes No Yes Yes Yes Yes Yes 10 Yes No Yes Yes Yes Yes Yes

    Plants that met standard

    9 0 9 10 9 10 10

    Note: FSIS inspectors do not collect and analyze generic E. coli and
    Salmonella as part of their verification duties in this pilot project.

    Source: FSIS inspectors? verification data from November 2000 through June
    2001.

    Page 30 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Table 6 shows which plants? inspection systems did not meet specific
    performance standards. That is, plant 6 did not meet the food safety
    standard for infectious conditions (e. g., septicemia and/ or toxemia);
    plant 8 did not meet the food quality standard for animal diseases such as
    airsaculitis; and plant 2 did not meet the food quality standard for
    digestive content contamination (ingesta). At the pilot plants, FSIS
    inspectors are issuing noncompliance records when plants do not meet the
    performance standards; however, FSIS officials have not yet decided how many
    instances of noncompliance will be tolerated before the agency can decide to
    take further action to ensure regulatory compliance.

    We reviewed data collected by FSIS inspectors after the plants shifted to
    modified inspections during two different time periods. In comparing these
    two periods, we found that in several categories, the inspection system?s
    performance at each plant deteriorated over time. Table 7 summarizes the
    changes for these 10 plants.

    Table 7: Changes in Chicken Plants? Inspection Systems Performance Over Two
    Time Periods After Shifting to Modified Inspections (FSIS? Data)

    Food safety standards Food quality standards Performance

    FS1: septicemia and/ or toxemia FS2: visible

    fecal material OCP1:

    animal disease, e. g.,

    airsaculitis OCP2:

    bruises, sores, etc. OCP3:

    ingesta OCP4:

    feathers, oil glands, etc.

    OCP5: bursa, cloaca, etc.

    Better 0 1 1 5 2 05 Same 1 0 9 9 0 2 12 Worse 0 0 0 5 6 93

    Total 10 10 10 10 10 10 10

    Note: During the first period, the plants were implementing the original
    pilot project design in which there were no FSIS inspectors stationed at
    fixed positions on the slaughter lines. During the second, period, a carcass
    inspector was reinstated at each slaughter line to comply with a court
    ruling. The first period starts just after each plant shifted from
    traditional to modified inspections and ends on October 31, 2000. The second
    period starts November 1, 2000, and ends on June 30, 2001.

    Source: GAO?s analysis of FSIS inspectors? verification data from
    implementation of modified inspections until June 2001. We used a difference
    of plus or minus 0.5 percent to analyze the changes for the seven categories
    described in the table. We estimated that, on average, 43,000 chickens are
    processed per line, per shift.

    On the basis of this information, in addition to providing plant managers
    with feedback on a daily basis, FSIS inspectors could identify plants whose
    inspection systems need to improve. In addition, FSIS could aggregate data
    from these plants and use this type of analysis to evaluate whether a
    specific performance standard needs to be revised. For example, we made two
    observations by analyzing the data as follows:

    Page 31 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

     As shown in table 7, 9 of the 10 plants? performance under the OCP4
    standard deteriorated over time. While this standard permits 80 percent of
    carcasses per shift to have this defect and still meet the standard, this
    trend may suggest the need for FSIS to investigate why the inspection
    system?s performance at an individual plant is deteriorating.

     As shown in table 7, 9 of the 10 plants maintained their performance and
    one improved under the FS2 standard.

    Australian officials told us that under their modified inspection system,
    the Australian Quarantine and Inspection Service requires plants to use
    process controls to demonstrate continuous improvement in plant sanitation,
    microbial pathogen reduction, and quality defect reduction. The agency uses
    these data to monitor plant performance and enforce compliance with
    standards.

    About 70 percent of the respondents to our survey of FSIS inspectors and
    veterinarians believe that modified inspections are equal to or somewhat
    better than traditional inspections for product safety while more than half
    believe that modified inspections are the same as or better than traditional
    inspections for product quality. Table 8 summarizes inspectors? and
    veterinarians? responses to major topics covered by our survey. (See app. IV
    for further detail on the survey responses.)

    Table 8: Summary of Responses to GAO?s Survey of USDA Inspectors and
    Veterinarians

    Topic Responses in percentages

    Product safety 71: same as or better under modified inspections versus
    traditional inspections 24: better under traditional inspections 5: unsure
    Product quality 57: same as or better under modified inspections versus

    traditional inspections 39: better under traditional inspections

    5: unsure Effect of plants? adding anti- microbial rinses and washes a

    62: food safety the same as or better than without rinses and washes

    3: food safety worse than without rinses and washes 10: don?t know its
    effect on food safety Line speeds: Concerning product safety 59: line speed
    is too fast

    39: line speed is about right 0: line speed is too slow 2: unable to
    determine

    Most FSIS Inspectors and Veterinarians Believe That Modified Inspections Are
    Equal to or Better Than Traditional Inspections for Ensuring Product Safety

    Page 32 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Topic Responses in percentages

    Concerning product quality 64: line speed is too fast

    33: line speed is about right 0: line speed is too slow 2: unable to
    determine Original modified inspection system versus modified system after
    court order: Concerning product safety

    Concerning product quality

    57: modified system after court order is better 27: original modified system
    is better

    8: they are about the same 8: no opinion on which is better

    45: modified system after court order is better 25: original modified system
    is better 19: they are about the same 11: no opinion on which is better.
    Noncompliance records 76: increased under the modified inspections versus

    traditional inspections b 11: decreased under the modified inspections
    versus traditional inspections 10: the same under the modified inspections
    and traditional inspections

    3: don?t know Training of plant personnel: Concerning adequacy of plant
    sorter training

    Concerning USDA?s development of standardized training curriculum for plant
    personnel

    Concerning USDA?s development of a testing and certification program for
    plant personnel

    15: plant?s training helped to a great extent 51: plant?s training helped to
    some extent 28: plant?s training helped to little or no extent

    6: unable to determine 67: FSIS should develop curriculum 17: FSIS should
    not develop curriculum 15: neutral

    2: unable to determine 64: FSIS should develop this program 18: FSIS should
    not develop this program 16: neutral

    1: unable to determine. FSIS training for pilot project inspectors 40:
    helped to a great extent

    51: helped to some extent 8: helped to little or no extent 1: have not
    received training for the pilot project 0: unable to determine. a This
    change was experienced by 75 percent of respondents at their plants after
    modified inspections

    were implemented. b Some inspectors cited more time and freedom to find
    defects and write noncompliance records and

    more carcasses are sampled under modified inspections. Note: Because of
    rounding, numbers may not add up to 100 percent.

    Page 33 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Nearly all of the inspectors and veterinarians provided additional written
    comments on several issues of concern and made suggestions for improvement.
    The inspectors commented that under the modified system, they are able to
    oversee the entire slaughter line and have more time to collect carcass
    samples for detailed examination. In addition, 27 of the 210 inspectors and
    veterinarians commented that they were concerned about regulatory
    noncompliance and enforcement issues. Particularly noteworthy are comments
    that indicate that stronger measures should be in place to reduce repetitive
    instances of fecal material noncompliance and that inspectors should have
    more authority to hold a plant accountable for multiple and/ or repetitive
    problems. In addition, four inspectors would like each failure of the zero-
    tolerance standard for fecal material to be documented in separate
    noncompliance records rather than recording multiple instances in a single
    record, which may obscure the frequency of the problem. Thirty- nine
    inspectors and veterinarians also raised concerns regarding product quality.
    In addition, seven inspectors stated that the quality performance standards
    need to be tightened so that fewer defects are overlooked.

    In addition, 30 inspectors and veterinarians commented that they are
    concerned that the zero- tolerance standard for fecal material is not
    adequately enforced, especially since the carcass inspector examines only
    the back of the carcass and does not look inside the cavity. Seven suggested
    that FSIS place a mirror or provide some other mechanism to facilitate
    observation by the carcass inspector.

    Finally, 62 inspectors commented that working conditions under the modified
    inspection system are better because the inspectors no longer need to make
    repetitive motions, as they did when they continuously examined each carcass
    on the slaughter line by touch. Sixteen inspectors also said that they like
    the ability to physically move from their location at least once every hour.
    Twenty- five of them mentioned, however, that the carcass inspectors?
    location might expose them to agents, such as chlorine or trisodium
    phosphate that could adversely affect their health.

    We believe that a risk- based inspection system- such as the one that USDA
    is pilot- testing at chicken plants and is starting at hog plants- has merit
    in concept and is consistent with the existing risk- based framework for
    HACCP. However, while we support this approach, we believe that the design
    of this pilot will not permit USDA to reach conclusions about whether the
    new system of modified inspections performs as well as the traditional
    system. Conclusions

    Page 34 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    If, as planned, USDA undertakes to modify its current inspection system for
    chicken plants, it will need to consider several factors that we discuss in
    this report and that are also key features of the Australian and Canadian
    modified inspection systems. First, if USDA issues regulations that require
    all chicken plants to modify their inspection systems concurrently without
    first determining whether individual plants are able to meet standards, it
    may risk including those plants with repeated records of noncompliance. Such
    plants may have difficulty meeting their new responsibilities. Continued
    participation in a modified inspection system should depend on the plants?
    ability to maintain good performance. Second, adding a requirement that
    plants use statistical process control systems to identify variations in
    performance will allow the plants to better manage and control their
    production processes and will also allow USDA to appropriately monitor and
    verify inspection systems? performance at each plant over time. Third,
    without requiring that plant personnel receive adequate training to
    undertake carcass defect detection responsibilities, USDA may jeopardize
    product safety and quality. At the very least, USDA should ensure that
    personnel conducting carcass defect detection duties have knowledge and
    training comparable to that of USDA inspectors who were previously
    responsible for these duties. Last, inspectors and veterinarians responding
    to our survey provided several useful comments on how to enhance a modified
    inspection system. For example, they believe that USDA needs to address
    multiple instances of plant noncompliance with regulatory requirements. As
    discussed in this report, many plants had repeated instances of
    noncompliance with a critical food safety performance standard.

    In addition to the current legal challenges, USDA faces future potential
    legal challenges unless its statutory inspection authorities are revised.
    These challenges will likely hinder the Department?s objective of modifying
    its inspection system at meat and poultry slaughter plants. We continue to
    believe that, as we have recommended before, the Congress should consider
    revising the Meat and Poultry Acts, 21 U. S. C. sections 604 and 455, to
    provide FSIS with the flexibility and discretion to target its inspection
    resources for the most serious food safety risks. Such revisions would
    eliminate the requirements that USDA has traditionally implemented through
    continuous carcass- by- carcass government inspection and replace them with
    a risk- based inspection system that includes government oversight and
    verification.

    Page 35 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    If USDA decides to implement modifications to its inspection system, we
    recommend that the Secretary of Agriculture direct FSIS to

     phase in the implementation of modified inspections so that only plants
    with a good history of regulatory compliance are eligible to participate and
    continue in the program;

     require plants to adopt statistical process control systems to manage and
    control their production and require FSIS personnel to monitor and verify
    these systems;

     in conjunction with industry, develop a training and certification program
    for personnel involved in tasks previously performed by federal inspectors
    and require that only trained and certified plant personnel perform these
    duties; and

     consider the merits of adopting suggestions from inspectors and
    veterinarians at pilot project plants contained in this report, such as how
    to address repetitive instances of noncompliance with regulatory
    requirements.

    We further recommend that, if in addition to the current pilot project for
    chickens, USDA decides to conduct similar pilots for other species- hogs,
    turkeys, or cattle- the Department take steps to ensure that the pilot?s
    design and methodology are sufficiently rigorous to allow more valid
    conclusions than those in this chicken pilot.

    We provided USDA with a draft of this report for review and comment. In
    written comments, USDA said that the report?s recommendations were
    appropriate and that the agency will address them when it seeks public
    comment prior to making any regulatory changes. USDA provided technical
    comments, which we incorporated in the report as appropriate. USDA?s
    comments and our responses are contained in appendix VII.

    We performed our review from December 2000 through October 2001, in
    accordance with generally accepted government auditing standards.

    As agreed with your offices, unless you publicly announce its contents
    earlier, we plan no further distribution of this report until 30 days from
    the date of this letter. At that time, we will send copies to congressional
    committees with jurisdiction over food safety issues; the embassies of
    Recommendations for

    Executive Action Agency Comments and Our Evaluation

    Page 36 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Australia and Canada; the Secretary of Agriculture; the Director, Office of
    Management and Budget; and other interested parties. We will make copies
    available to others on request.

    If you have any questions about this report, please contact me at (202) 512-
    3841. Major contributors to this report are listed in appendix VIII.

    Lawrence J. Dyckman Director, Natural Resources

    and Environment

    Appendix I: Scope and Methodology Page 37 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    To describe the objectives, design, and scope of the pilot project, we
    interviewed cognizant government and industry officials. Specifically, we
    interviewed officials and/ or reviewed documents from the U. S. Department
    of Agriculture?s (USDA) Food Safety and Inspection Service (FSIS); Office of
    Policy, Program Development, International Programs, and Field Operations;
    and Foreign Agriculture Service. We also consulted with the National Joint
    Council of Food Inspection Locals, Research Triangle Institute (RTI), Center
    for Science in the Public Interest, Government Accountability Project,
    European Union Commission, Consumer Federation of America, National Chicken
    Council, National Turkey Federation, National Veterinarian Association, and
    American Meat Institute. We obtained information as well from Federal
    Register notices and court filings. Additionally, we visited and interviewed
    plant managers, USDA inspectors, and veterinarians at five chicken plants
    and one hog plant participating in the pilot project. To provide information
    on similar projects in Australia and Canada, we visited and toured their
    meat and poultry plants. We also interviewed Australian and Canadian
    government officials, inspectors, and veterinarians, consumer groups, and
    plant managers to obtain their perspectives on their country?s modified
    inspection systems. We did not evaluate the validity of the Australian and
    Canadian projects? design and methodology.

    To identify limitations, if any, in the project?s design and methodology, we
    reviewed USDA?s contract with RTI and discussed data collection and data
    analysis methods with statisticians from USDA, the Australian Quarantine and
    Inspection Service, and the Canadian Food Inspection Agency. We obtained
    input on the pilot project?s design from consumer groups, industry
    associations, poultry science experts, and our design methodologists. We
    also reviewed the sample sizes selected by USDA and determined whether the
    results generated from the volunteer plants can be generalized to entire
    population. We interviewed managers of pilot project plants to identify the
    length of industry training provided for plant personnel who replaced USDA
    inspectors on the slaughter line and any modifications made to the slaughter
    line before and during the pilot project that could affect the project?s
    results. Moreover, we participated in FSIS? three- day training course
    covering pilot project inspections.

    Additionally, we analyzed noncompliance records issued by USDA inspectors
    for failures to meet the zero- tolerance standard for fecal material at
    chicken plants. We asked USDA to provide a complete set of noncompliance
    records for each of the 11 chicken plants that participated in the baseline
    and modified inspection phases of the pilot project. That is, we asked for
    noncompliance records 1 year before and 1 year after Appendix I: Scope and
    Methodology

    Appendix I: Scope and Methodology Page 38 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    implementation of the modified inspections at each plant. To the best of our
    knowledge, we received complete data from all the plants. However, for some
    plants, we did not receive 12 months? worth of data reflecting the modified
    inspections because the plants had been operating under modified inspections
    for less than a year.

    We began our analysis by removing duplicate noncompliance records and
    noncompliance records resulting from sanitation and other nonfecal problems.
    We then counted the number of noncompliance records resulting from fecal
    material violations by month for each plant to determine whether the total
    number of noncompliance records was increasing or decreasing after
    implementation of the modified inspections. Since each noncompliance record
    can document multiple violations, we also analyzed, on a monthly basis,
    whether the total number of carcasses contaminated with fecal material for
    each plant was increasing or decreasing under the modified versus
    traditional inspections. Furthermore, we counted and compared the number of
    noncompliance records generated from the same type of system failure
    (repetitive failures) before and after implementation of the modified
    inspections. For some months, we did not have any noncompliance records
    documenting problems with fecal material.

    To evaluate whether the data generated by the pilot project will allow USDA
    to reach valid conclusions on the relative effectiveness of the modified
    inspection system tested in the pilot project, we reviewed, compared, and
    contrasted pilot project data gathered by USDA inspectors and RTI. We
    requested and received pilot project data gathered by RTI at the 11 chicken
    plants that participated in both the baseline and modified inspection phases
    of the pilot project. We compared the individual plants? results with the
    two food safety and five food quality performance standards to determine if
    each plant was able to meet the standards. We also determined whether plants
    improved their performance over time under modified inspections by comparing
    individual plants? results under modified inspections with their results
    under traditional inspections. We also requested USDA inspectors?
    verification data from the same 11 chicken plants. Because one plant dropped
    out of the pilot project in early 2001, we did not evaluate the data from
    this plant. USDA averaged the verification results from all of the plants
    together to determine if the plants collectively met the performance
    standards. We chose to average each plant?s results individually to
    determine its ability to meet or exceed the performance standards over time.

    Appendix I: Scope and Methodology Page 39 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    In addition, we surveyed all pilot project inspectors and veterinarians who
    the USDA identified as working at pilot project plants to obtain their views
    and concerns regarding the modified inspection system. USDA officials
    provided the survey respondents with 1 hour of compensatory time for
    completing our survey. In designing our survey, we interviewed pilot project
    program officials, inspectors, and veterinarians to determine what issues
    should be included in the survey. During the pretesting phase of our survey,
    it became evident that respondents considered the survey questions to be
    highly sensitive. To address these concerns, we used procedures to guarantee
    the anonymity of all survey responses. However, the use of a separate return
    postcard for follow- up purposes allowed us to track which respondents did
    and did not mail back survey responses. After developing an initial draft of
    the survey questionnaire, we visited and pretested the questionnaire with 10
    respondents at two chicken plants and one hog plant to obtain comments from
    pilot project inspectors and veterinarians and to ensure that the survey was
    appropriate for both species of animals. We revised the questionnaire in
    accordance with their comments. Then we incorporated input from union
    officials and USDA officials before sending out the questionnaires. We
    distributed them by mail according to a list provided by USDA officials.
    However, we sent additional questionnaires to those pilot project inspectors
    and veterinarians who were missing from the initial list and who called us
    to say they did not receive one. We also sent three follow- up reminders in
    the weeks following the initial distribution of the questionnaire. We
    reviewed each survey response to identify internal data inconsistencies. We
    sent out questionnaires to 225 persons and received 210 responses, for a 93-
    percent- response rate.

    We performed our review from December 2000 through October 2001, in
    accordance with generally accepted government auditing standards.

    Appendix II: Chronology of Pilot Project Litigation

    Page 40 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    USDA?s Food Safety and Inspection Service sought public comment on the
    development of new inspection models for the slaughter and processing of
    meat and poultry in connection with the implementation of its previously
    promulgated HACCP rule. 1 FSIS stated that the inspection models would not
    include fixed FSIS inspection stations on slaughter lines, as under existing
    procedures. Instead, FSIS inspectors would provide oversight at the
    slaughter lines and verification that plants were properly implementing
    HACCP, among other things. FSIS further stated that, upon completion of the
    models project, it will initiate rulemaking, as appropriate, to change
    existing inspection procedures.

    American Federation of Government Employees (AFGE) filed a complaint in the
    U. S. District Court for the District of Columbia seeking to enjoin the USDA
    from proceeding with the pilot project, on the grounds that it violated the
    Federal Meat Inspection Act, 21 U. S. C. section 604, and the Poultry
    Products Inspection Act, 21 U. S. C. section 455, by not requiring federal
    government officials to perform carcass- by- carcass postmortem inspection.

    U. S. District Court (Lamberth, R.) granted USDA?s motion for summary
    judgment, holding that neither the Federal Meat Inspection Act nor the
    Poultry Products Inspection Act was violated by the HACCP- Based Inspection
    Models Project. American Federation of Government Employees v. Glickman, No.
    98- 0893 (D. D. C. Sept. 23, 1999). The court reasoned that even though the
    statutes require postmortem inspections by USDA inspectors, the word
    ?inspection? in these statutes does not necessarily mandate direct, physical
    ?organoleptic?- involving sight, touch, and smell- examination by USDA
    inspectors.

    AFGE appealed the district court?s ruling to the U. S. Court of Appeals for
    the District of Columbia Circuit, which reversed the district court?s
    decision. The court of appeals held that delegating the task of inspecting
    carcasses to plant employees violated the Federal Meat Inspection Act and
    the Poultry Products Inspection Act because both statutes require that
    federal inspectors, rather than private employees, determine whether a
    product is adulterated. American Federation of Government Employees v.
    Glickman, 215 F. 3d 7, 9 (2000). The court of appeals remanded the case to
    the district court for further proceedings consistent with its opinion.

    1 ?HACCP- Based Meat and Poultry Inspection Concepts,? 62 Fed. Reg. 31553
    (June 10, 1997). Appendix II: Chronology of Pilot Project

    Litigation 6/ 10/ 97

    4/ 8/ 98 9/ 23/ 99 6/ 30/ 00

    Appendix II: Chronology of Pilot Project Litigation

    Page 41 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    AFGE moved the district court to enter an order declaring that the pilot
    project violated the Federal Meat Inspection Act and the Poultry Products
    Inspection Act and permanently enjoining USDA from delegating its statutory
    duty to inspect each meat and poultry carcass and its parts to anyone other
    than federal inspectors.

    USDA responded to AFGE?s motion, stating that to comply with the court of
    appeals? decision, FSIS was modifying the pilot project so that an FSIS
    inspector would directly inspect every carcass. Specifically, USDA stated
    that in the 11 chicken plants in the pilot project, one FSIS inspector would
    be stationed toward the end of the slaughter line, between the final wash
    and the chiller. This inspector would be responsible for examining each
    carcass and determining whether it is adulterated. In the three hog plants,
    FSIS inspectors would be placed at up to three fixed locations on the
    slaughter line and would be responsible for examining the carcass, head, and
    viscera, and determining whether they were adulterated.

    All pilot project plants had implemented the redesigned inspection model. 2
    U. S. District Court for the District of Columbia ruled that the redesigned
    pilot project does not violate the court of appeals ruling in American
    Federation of Government Employees v. Glickman, 215 F. 3d 7 (2000) and does
    not violate the Federal Meat Inspection Act and the Poultry Products
    Inspection Act. American Federation of Government Employees v. Glickman, No.
    98- 893 (Jan. 17, 2001). According to the district court, the redesigned
    pilot project is consistent with the court of appeals decision because the
    thrust of that opinion was that to satisfy its inspection duties, FSIS must
    do more than merely observe others performing inspections, and in the
    redesigned project, federal inspectors will make the critical determination
    as to whether a product is adulterated. Furthermore, the redesigned pilot
    project satisfied the requirements of the federal meat and poultry
    inspection statutes, according to the district court, because it reflects a
    reasonable interpretation of the meaning of the term ?inspection? within
    those statutes.

    2 See FSIS Briefing Paper: The History of the HACCP- Based Inspection Models
    Project (Jan. 19, 2001). 8/ 25/ 00

    9/ 15/ 00 11/ 6/ 00 1/ 17/ 01

    Appendix II: Chronology of Pilot Project Litigation

    Page 42 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    AFGE filed a notice of appeal to the U. S. Court of Appeals for the District
    of Columbia Circuit. The appeal is still pending. 2/ 5/ 01

    Appendix III: Performance Standards for Hog Plants

    Page 43 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    For hog plants, FSIS developed three food safety and three food quality
    standards from traditional inspections data (baseline data) gathered at five
    plants. (See table 9.) These standards were set slightly below the
    performance of the fourth (out of five) plants. 1 At the time of our review,
    FSIS did not have complete pilot project data available for the hog plants.

    Table 9: Performance Standards for Inspection Systems at Hog Plants in the
    Pilot Project

    Categories Performance standards

    (percentage of defects allowed per shift)

    Food Safety 1: Infectious conditions such as septicemia and toxemia 0 Food
    Safety 2: Digestive content such as fecal material, ingesta, milk

    0 Food Safety 3: Antemortem symptoms such as neurologic conditions 0 Other
    Consumer Protection 1: Carcass pathology such as arthritis, emaciation, etc.
    4.1 Other Consumer Protection 2: Visceral pathology such as enteritis/
    gastritis, fecal contamination of viscera, etc.

    7.2 Other Consumer Protection 3: Miscellaneous such as anemia, bile,
    bruises, scabs, etc. 20.5

    Source: USDA, FSIS.

    USDA is using the safety and quality performance standards developed for
    this project to determine, among other things, if the plants can perform at
    least as well as they did under traditional inspections.

    1 USDA developed the hog performance standards from the following formula:
    (0. 25 * 3rd plant?s performance) + (0. 75 * 4th plant?s performance). These
    standards are comparable to the 25th percentile established for the chicken
    pilot. Appendix III: Performance Standards for Hog

    Plants

    Appendix IV: Survey Results Page 44 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results

    Appendix IV: Survey Results Page 45 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 46 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 47 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 48 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 49 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 50 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 51 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 52 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 53 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 54 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 55 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 56 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 57 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix IV: Survey Results Page 58 GAO- 02- 59 Meat and Poultry Pilot
    Inspection Project

    Appendix V: Australia?s Meat Safety Enhancement Program

    Page 59 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Australia?s Meat Safety Enhancement Program (MSEP) is a pilot meat
    inspection program developed by the Australian Quarantine and Inspection
    Service (AQIS) for use in meat export plants. MSEP is patterned after the
    Australian domestic meat inspection system, which is grounded in the theory
    of Hazard Analysis Critical Control Point procedures and uses a quality
    systems approach to control for hazards in food processing. Under the
    Australian inspection approach, slaughter plants are clearly identified as
    being responsible for plant process control, including sanitation, microbial
    monitoring, and inspection of the product for quality defects. AQIS sets
    public health product standards and audits plant operations to ensure that
    food safety standards are being achieved.

    Australia?s traditional system of meat inspection had an end- product
    inspection focus and employed government staff to perform inspections.
    However, Australian authorities were concerned that this system did not
    adequately address all of the risks to food safety. For example, public
    health officials, among others, thought that proper emphasis was not being
    given to microbial contaminants, such as E. coli and Salmonella, or residues
    of veterinary and agricultural chemicals and antibiotics, which had emerged
    as threats to the safety of fresh meat and meat products.

    Australia first moved in 1991 to address these emerging food safety issues
    with a comprehensive review of domestic meat inspection. Upon the completion
    of this review, the review task team outlined changes to existing meat
    inspection procedures that it thought would result in a more cost- effective
    system while maintaining adequate safeguards to public health. The
    recommended changes included strategies for implementing quality assurance
    programs and HACCP procedures.

    Over an 18- month period, commencing in early 1992, three domestic slaughter
    plants developed quality assurance systems that included replacing
    regulatory agency inspectors with company employees. AQIS subsequently
    approved this new inspection approach for trial. In the trial, each plant
    was paired with a nearby similar plant as a control plant. The control
    plants continued operations under the traditional meat inspection system.
    According to AQIS, the trial results showed that all three plants operating
    under their quality assurance systems and using companyemployed meat
    inspectors were capable of maintaining standards of food safety at least
    equal to those of the control plants. Appendix V: Australia?s Meat Safety

    Enhancement Program Background and Trends That Led to the Program

    Appendix V: Australia?s Meat Safety Enhancement Program

    Page 60 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    These findings were examined and endorsed by a scientific panel composed of
    officers from AQIS, universities, and meat inspector representatives of the
    Public Sector Union and the Australian Meat Research Council. All Australian
    domestic meat establishments in each state subsequently moved to a self-
    regulated system of meat inspection, whereby companies assumed
    responsibility for meat safety. MSEP, which was implemented in 1996 for meat
    export facilities, was an outgrowth of the domestic movement to quality
    assurance systems using HACCP procedures and company- employed inspectors.

    The MSEP model incorporates the use of routine meat inspection performed by
    qualified company employees working under a system approved and legislated
    by the government. The government, acting through AQIS, maintains full- time
    oversight, verification, and certification of the final product. As a part
    of Australia?s MSEP system, a full- time AQIS veterinary officer, whose
    salary is reimbursed by the company, is required to be on duty at all times
    in each participating export plant. At establishments producing products for
    export to the United States, an AQIS inspector assists the veterinary
    officer in performing the zero tolerance and end- product inspection.
    Company management is responsible for on- line inspection using company-
    paid inspectors that have been trained in accordance with a standardized
    curriculum and are under the general supervision of the full- time AQIS
    veterinary officer. The officer provides continuous oversight, verification,
    and daily audit of plant operations and production control systems. The
    company- paid inspectors must also be licensed by their respective states
    before they may act as inspectors. To be eligible for a license, they must
    complete about 600 hours of formal classroom training and pass examinations.
    The division of responsibilities between AQIS and companies under MSEP are
    as shown in table 10.

    Table 10: Division of Responsibility Under Australia?s Meat Safety
    Enhancement Program Function Company?s responsibility AQIS? responsibility

    Antemortem inspection Sort healthy animals for slaughter and separate
    animals identified as potential suspects for inspection by AQIS.

    Inspect suspect animals and decide on their disposition.

    Postmortem inspection Develop procedures for carcass, head, and offal
    inspection using all the inspection activities required by AQIS.

    Set up monitoring checks for all inspection activities conducted by company-
    employed inspectors.

    Conduct regular oversight and verification checks of post- mortem inspection
    activity at least daily.

    Conduct independent checks of the inspection procedures during the monthly
    audit review.

    How the MSEP System Works

    Appendix V: Australia?s Meat Safety Enhancement Program

    Page 61 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Function Company?s responsibility AQIS? responsibility

    E. coli and Salmonella monitoring Company Quality Assurance officers or
    inspectors will take samples for testing.

    Check the E. coli samples to ensure that processing is consistent with good
    laboratory practice.

    Supervise each Salmonella test. Review microbiological results in the
    monthly review of the plant. End- of- line inspection Take appropriate
    corrective action if the AQIS

    veterinary officer detects defects. Perform regular ?check the checker?
    audits of company inspection and quality assurance personnel. Monitoring
    Monitor all procedures and all aspects of the

    HACCP program. Continuously monitor and recheck

     antemortem and postmortem inspections and

     microbiological testing and the HACCP program.

    Source: AQIS.

    According to AQIS officials, the service has established several
    prerequisites for companies that wish to participate in MSEP and sell
    products for export. In addition, plants must demonstrate the following
    before they can be a part of the program. Specifically, plants must

     have a good operational/ coregulatory trade record;  have demonstrated a
    commitment to and appreciation of HACCP/ Quality

    Assurance principles, to include an adequate company infrastructure and
    training competencies;  accept responsibility for performing the sorting
    (inspection) function;  develop and maintain a plant- specific, AQIS-
    approved Meat Safety Quality

    Assurance System;  act as a promotional site for industry/ overseas
    officials; and  maintain a comprehensive plant quality assurance manual
    that was

    developed by and is ?owned? by relevant areas of plant operations. The
    manual must be a working document and receive AQIS? approval. When approved,
    the HACCP/ Quality Manual acts as a legal contract between the plant and
    AQIS on the specific procedures, controls, and standards at the plant.

    Participation in the program is currently voluntary. However, eventually,
    all plants will be required to move to the new inspection system.
    Prerequisites for

    Plants to Participate in MSEP

    Appendix V: Australia?s Meat Safety Enhancement Program

    Page 62 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    According to AQIS officials, much empirical evidence on the effectiveness of
    HACCP quality assurance systems has accumulated since MSEP was implemented
    in Australia. AQIS officials told us they are fully confident that the MSEP
    approach to regulatory inspection control achieves guarantees of food safety
    that are at least equivalent to those of the traditional approach to meat
    inspection. They said the system incorporates a reengineering of meat
    inspection practices and results in various changes to the culture or
    philosophy of food inspection, including changes that

     prevent rather than correct,  build quality in rather than rely on end-
    product inspection,  place more reliance on microscopic verification,
    transfer responsibility from government to industry for performing certain

    functions and maintaining standards, and  incorporate government
    verification and certification. AQIS? Evaluation of

    Its Program

    Appendix VI: Canada?s Modernized Poultry Inspection Program

    Page 63 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    The Canadian Food Inspection Agency (CFIA) recently adopted a new poultry
    inspection system that allows industry to progressively assume defect
    detection duties and remove defective carcasses from the slaughter line. The
    agency began pilot testing this modified inspection system in September
    1997. The pilot project is similar in design to USDA?s pilot project for
    poultry. The new system, known as the Modernized Poultry Inspection Program
    (MPIP), has goals to reduce microbial pathogens in raw poultry products and
    to enhance the use of science and risk- based management in Canada?s poultry
    inspection system. MPIP builds on a precursor program- the Canadian Poultry
    Inspection Program (CPIP)- which also allows industry operators to assume
    some defect- detection duties. The significant difference between the two
    programs is that under MPIP, industry personnel are also responsible for
    detecting viscera defects. Under both programs, CFIA inspectors monitor
    poultry plants? slaughter operations and verify industry?s compliance with
    food safety and quality standards. For poultry plants that choose not to
    participate in either of the programs, government inspectors continue to
    perform all carcass defect detection and removal duties.

    Before CFIA amended its meat inspection regulations to adopt the MPIP
    program in May 2001, the agency pilot tested the modified inspection system
    at four chicken plants that represented about 10 percent of the total
    chicken production. CFIA officials told us that several factors contributed
    to their decision to modify the poultry inspection system, including the
    desire to make inspections more consistent with the HACCP framework. The use
    of the HACCP system is not yet mandatory at meat and poultry plants in
    Canada, but it is a prerequisite for participation in MPIP. In addition,
    CFIA officials noted that the United States was engaging in a similar effort
    to modify its slaughter inspection system and said that Canada, which was
    modifying its poultry inspection system, wanted to obtain equivalency status
    with the United States to facilitate bilateral trade. Finally, CFIA
    officials noted that the Canadian poultry industry is investing in new
    evisceration equipment with technical improvements that permit faster line
    speeds without compromising food safety and quality.

    MPIP was a voluntary pilot project, and now that Canada has issued
    regulations to implement the program nationwide, participation continues to
    be voluntary. Prior to being accepted into MPIP, poultry plants must pass a
    preliminary assessment. Later, they must pass additional audits as they move
    from one phase of the program to the next. The audits, which Appendix VI:
    Canada?s Modernized Poultry

    Inspection Program Background How the MPIP Pilot Works

    Appendix VI: Canada?s Modernized Poultry Inspection Program

    Page 64 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    are performed by a CFIA audit team, are an important tool for the agency and
    plant managers. Clearly stated audit objectives help plant managers to plan
    for and address potential deficiencies before they occur, allowing the plant
    to succeed in the program. Plants participating in the program move through
    three phases.

    Phase 1 begins after a CFIA preliminary assessment of a volunteer plant?s
    eligibility to participate in the modified inspection program. All volunteer
    plants must have been operating under CPIP before participating as an MPIP
    pilot plant. As with USDA?s pilot project, baseline data are collected and
    analyzed before MPIP implementation. That is, microbial and quality defect-
    detection data are collected at volunteer plants while they are operating
    under CPIP. In addition, plant personnel are trained and accredited in their
    new role as plant defect detectors.

    Phase 2 resembles the transition phase in USDA?s pilot project. It is the
    trial phase during which CFIA inspectors back up plant personnel in
    detecting defects on the slaughter line. Plant personnel are required to
    pass four weekly practical on- line tests as viscera detectors. In addition,
    the plant must also pass a national compliance and verification review.

    Phase 3 entails operating under MPIP and the collection and analysis of data
    after MPIP?s implementation at the plant. Accredited plant personnel detect
    carcass, cavity, and viscera defects, while plant preselectors cull
    carcasses with specific processing and pathology defects, such as
    contamination from the intestines and septicemia. On average, a Canadian
    poultry plant runs one evisceration line and will have one veterinarian-
    incharge and two CFIA inspectors (one evisceration floor inspector per line
    working on the slaughter floor and one processing area inspector). If an
    equivalency agreement with the United States requires federal inspection of
    each and every carcass, CFIA will place one on- line government inspector on
    each slaughter line at plants that want to export to the United States.

    CFIA has established several prerequisites that plants must meet before they
    are accepted to participate in MPIP. CFIA considers, at a minimum, the
    following criteria for each plant:

     The state of the current plant inspection system, its quality control
    operations, and the gap between these and MPIP. Prerequisites for

    Plants to Participate in the MPIP Pilot

    Appendix VI: Canada?s Modernized Poultry Inspection Program

    Page 65 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

     The stage of plant- specific HACCP system development and implementation-
    at a minimum, the plant must have a functioning HACCP system in place to
    commence MPIP.  The implementation of E. coli and Salmonella testing
    protocols.  The types of poultry processing conducted by the plant, as it
    is desirable

    to have several classes of poultry slaughtered to improve efficiency in
    training staff, data collection, and implementing future pilots in the area.
     The geographic location (desire to implement MPIP and collect data from

    across Canada).  The compliance history of the plant.  The range of the
    health status of flocks and average condemnation rates in

    the plant selected to pilot MPIP to ensure adequate training and
    accreditation of plant defect detectors to handle condemnation rates and a
    range of disease conditions under an HACCP- MPIP system.

    CFIA?s MPIP national performance standard categories for poultry are similar
    but not identical to USDA?s pilot project standards for chickens. The
    poultry standards address the following factors, among others:

     Microbial contamination.  Processing defects, including all types of
    contamination- such as fecal

    and bile- extraneous material, extensive bruising, extensive overscald/
    mutilation, and inadequate bleeding. 1  Pathology defects, including
    septicemia/ toxemia, airsacculitis, ascites,

    emaciation, cellulitis, dark- colored carcasses, and other diseases. 2
    Finished product defects, such as bruises, feathers, trachea, oil glands,

    lungs, intestines, and crop. 3 To analyze microbial contamination as part of
    the MPIP project, in 1997 and 1998, CFIA performed a national baseline
    survey at 36, or 55 percent, of the 65 federally registered chicken plants
    nationwide over a 1- year period. CFIA?s performance standards for microbial
    contamination are

    1 USDA?s food safety performance standard 2 and other consumer protection
    (food quality) performance standards 2 and 3 for the pilot project include
    these types of defects. 2 USDA?s food safety performance standard 1 and
    other consumer protection (food quality) performance standard 1 for the
    pilot project include these types of defects. 3 USDA?s other consumer
    protection (food quality) performance standards 4 and 5 for the pilot
    project include these types of defects. Performance

    Standards

    Appendix VI: Canada?s Modernized Poultry Inspection Program

    Page 66 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    identical to USDA?s Pathogen Reduction: HACCP regulations for generic

    E. coli and Salmonella.

    To evaluate both processing and pathology defects, in 1997 and 1998, CFIA
    conducted another national survey to establish the average level of
    overlooked processing and pathology defects on carcasses passed by CFIA
    inspectors or by trained and accredited plant detectors under the
    traditional and CPIP methods of chicken inspection. Performance standards
    for processing and pathology defects were developed from data gathered at
    35, or 54 percent, of the 65 federally registered chicken plants nationwide.
    The performance standards for processing defects were set at 4.0 percent;
    however, as part of the MPIP project, CFIA subsequently tightened them to
    2.5 percent, meaning that 97. 5 percent of the carcasses passed by plant
    employees must be free of processing defects. The performance standards for
    pathology defects were set at 0.4 percent, meaning that 99.6 percent of the
    carcasses passed by plant employees must be free of pathology defects.

    CFIA conducted an additional national survey to amend its Finished Product
    Standards program. This is the same program developed and used by USDA in
    all chicken plants operating under the traditional carcass- bycarcass
    inspection systems. CFIA, however, has tightened the Finished Products
    Standards? pass/ fail criteria for plants operating under CPIP?s and MPIP?s
    inspection systems. In addition, CFIA continues to enforce a zero- tolerance
    policy for fecal contamination on ready- to- chill carcasses as part of this
    program.

    CFIA developed an extensive training program for government and industry
    personnel to ensure that everyone participating in MPIP has the necessary
    skills and knowledge to perform his/ her new duties. The training includes
    classroom training, a final exam, in- plant training, and the demonstration
    of competency. After successfully completing the training, CFIA trainers
    accredit the trainees.

    The entire MPIP training program was developed by CFIA. CFIA officials
    believe that no outside organization can replicate in- house expertise. In
    addition, in order to encourage union cooperation, CFIA inspectors conduct
    the training. At MPIP plants, every CFIA employee must participate in the
    training program. MPIP Pilot Success

    Attributed to Extensive Training of Inspectors and Industry

    Appendix VI: Canada?s Modernized Poultry Inspection Program

    Page 67 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    CFIA officials modeled MPIP?s industry- training program on the training
    program they developed for their own staff. The agency adopted a trainthe-
    trainer approach for industry. That is, CFIA inspectors train the plant
    trainers who in turn train the plant employees (defect detectors). The
    training is conducted in two phases. Phase 1 covers classroom and on- thejob
    training and concludes when the defect detector is accredited to perform
    his/ her new duties under MPIP. The plant managers decide which plant
    employees will become defect detectors and receive MPIP training. The
    training covers three types of defect detection: (1) preselection before
    evisceration, (2) internal examination of carcasses, and (3) poultry viscera
    examination. On- the- job training ensures that defect detectors learn how
    to identify carcass pathology conditions that are indicative of disease.
    Once the defect detectors are comfortable with identifying diseases, they
    are tested on 20 carcasses and the corresponding viscera to see if they can
    correctly decide whether the carcasses pass or need to be examined by the
    veterinarian. In addition, the defect detectors have to demonstrate the
    ability to detect defects on fast- moving lines. The defect detector must
    pass three 15- minute tests to be accredited.

    After the defect detector is certified, he/ she enters a 4- week trial
    period (phase 2) where he/ she performs his/ her new MPIP duties on the
    slaughter line. Defect detectors can be trained and accredited in more than
    one station (cavity, carcass, and/ or viscera) and will have weekly
    evaluations at each of these stations during the trial phase. In addition,
    the trainer evaluates defect detectors on a periodic basis, as specified in
    the plant?s HACCP plan. Plant defect detectors are reevaluated on an ongoing
    basis and lose their accreditation if they fail a periodic review test.
    Ongoing testing ensures continuing competence as line speeds and/ or disease
    patterns change. CFIA reserves the right to conduct additional
    reaccreditation tests of plant detectors at any time. In addition, CFIA
    reserves the right to conduct additional tests for missed defective
    carcasses at any time.

    According to CFIA officials, MPIP is successful because the data generated
    from the pilot plants show a downward trend in carcass defects. The plants
    that participate in the MPIP pilot are performing better than the
    performance set by the national standards. Part of the program?s strength is
    that CFIA officials periodically review data results with plant managers to
    discuss the plant?s record of performance against the standards. Through
    discussions with plant managers, CFIA officials learn what types of
    corrective actions are successful in addressing the plant?s system Under
    MPIP Pilot,

    CFIA Officials Monitor Plant Performance Trends

    Appendix VI: Canada?s Modernized Poultry Inspection Program

    Page 68 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    failures. In turn, CFIA officials share this knowledge with managers of
    other pilot plants to help them improve their own process controls.

    CFIA modified its meat- inspection regulations to implement MPIP on a
    nationwide basis on May 23, 2001. The agency will continue to maintain three
    types of poultry inspection systems (traditional, CPIP, and MPIP). CFIA does
    not plan to make changes to the MPIP program and will continue to use a
    phased approach for implementing MPIP at volunteer plants. Plants will still
    be required to pass a preliminary assessment before being accepted into
    MPIP. Under MPIP, CFIA officials will continually monitor the plants?
    performance through informal discussions and formal audits. The agency
    expects MPIP plants to perform better than the performance set by the
    national standards for the microbial, food safety, and other consumer
    protection defects. If a plant fails to meet these standards, the plant will
    lose its license to operate under MPIP and must be relicensed to operate
    either under CPIP or traditional inspections. In addition, the agency
    expects MPIP plants to continually improve their performance over time.
    Furthermore, CFIA will continue to emphasize the importance of training
    through its mandatory training program and certification requirements for
    both CFIA employees and industry personnel.

    Since CFIA has formally adopted MPIP, the agency will maintain an oversight
    inspector and a veterinarian- in- charge at each MPIP plant. Unlike USDA?s
    pilot project, CFIA?s MPIP program does not require an on- line carcass
    inspector performing carcass- by- carcass inspections. However, if the
    equivalency agreement with the United States requires an on- line carcass
    inspector, CFIA will comply with this requirement and place a CFIA inspector
    on the slaughter line at those plants that export to the United States.
    Current Status of

    MPIP

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 69 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Appendix VII: Comments From the U. S. Department of Agriculture

    Note: GAO?s comments supplementing those in the report?s text appear at the
    end of this appendix.

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 70 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    See comment 1. See comment 2.

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 71 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Now on pp. 2- 3. See comment 9. Now on p. 2.

    See comment 8. See comment 7.

    See comment 6. See comment 5. See comment 4. See comment 3.

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 72 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    See comment 18. See comment 17.

    See comment 16. See comment 15.

    See comment 14. See comment 13. See comment 12.

    See comment 11. See comment 10.

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 73 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Now on pp. 27- 28. See comment 23. Now on p. 26.

    See comment 22. Now on p. 24.

    See comment 21. Now on p. 24.

    See comment 20. Now on p. 23.

    See comment 19.

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 74 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    See comment 30. See comment 29. Now on p. 34.

    See comment 28. Now on p. 30.

    See comment 27. Now on p. 29.

    See comment 26. Now on p. 28.

    See comment 25. Now on p. 27.

    See comment 24.

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 75 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 76 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    The following are GAO?s comments on the U. S. Department of Agriculture?s
    letter dated November 30, 2001.

    1. We believe that the report?s title describes the overall results of our
    study. Therefore, we have not modified the title as USDA suggested.

    2. We do not agree that our report oversimplifies the goals of the pilot
    project. The pilot project?s objective was to compare the accomplishments-
    as measured by USDA?s safety and quality performance standards- of two
    inspection systems at volunteer plants. We understand that the plants
    participating in the pilot project made various other changes at different
    times during the pilot project. However, the only change that USDA required
    of plants volunteering to participate in the pilot was the replacement of
    USDA inspectors with plant personnel. As USDA states, this project was
    designed as a ?before- and- after? experiment. The ?before? refers to
    traditional inspections with USDA inspectors performing the carcass defect
    detection duties, and the ?after? refers to the plant employees? performance
    of the carcass defect detection duties. As explained in our report, USDA?s
    independent contractor first went to each participating plant to collect
    carcass defect data under traditional inspections, waited until the plant
    employees went through a transition period, and then returned to collect the
    same type of data after plant employees assumed the inspectors? duties. If
    USDA wants to compare the accomplishments of two different inspection
    systems that include many variables, it is critical to use a control group.
    Indeed, the lack of a control group in USDA?s study prevents them from
    knowing the extent to which plants that do not participate in the study
    (about 175 plants nationwide) made changes in their processing systems that
    improved safety and quality while continuing to inspect under the
    traditional system. We continue to believe that the design of the USDA?s
    pilot project does not permit USDA to interpret what changes account for the
    experiment?s outcomes.

    3. We agree with the premise that the use of volunteer plants is a
    legitimate approach and our report acknowledges that, once USDA decided to
    go forward with this pilot project design, it could not compel plants to
    participate and, therefore, had to rely on volunteer plants. However, given
    that USDA decided to proceed with this design, our report points out that
    USDA had the option of randomly selecting a subgroup from the volunteer
    plants to serve as a control group. The problem with relying on volunteers
    for this study is that there are a GAO?s Comments

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 77 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    small number of plants participating- there are only 11 chicken plants for
    which before and after data have been collected. Even if the plants had been
    randomly selected, they would not represent a sufficient number to enable
    USDA to generalize the results to the entire population of chicken plants.
    Therefore, USDA cannot assume that the results from these 11 plants would
    apply to all the plants that in the future may participate in a modified
    inspection system. Finally, neither USDA nor RTI provided GAO with any
    evidence that the pilot project was either designed or analyzed as a time-
    series study.

    4. We do not agree with USDA that the plants participating in the study are
    typical of the industry in that they represent diversity in geography,
    corporate structure, management styles, numbers of evisceration lines,
    product distribution, and other variables. As figure 2 of our report shows,
    the pilot project has before- and- after data for 11 plants that are located
    in the southern region of the United States. Therefore, particularly with
    respect to microbial data, the results from these plants may not be
    indicative of how plants in other areas of the United States may behave.
    Geographic distribution and seasonal variations are important variables that
    cannot be ignored in assessing a program. As a recent USDA study points out,
    Salmonella and E. coli 0157: H7 rates appear to differ dramatically by
    season, while geography seems to affect only Salmonella results. When, as
    part of this study, USDA researchers investigated the regional patterns,
    they found higher positive rates for Salmonella in warmer climates. For
    example, in southern operations, the rates were about 8 percent, whereas the
    rate was just under 5 percent in northern operations. This study highlights
    the importance of variables such as seasonal variations and geographic
    distribution. As discussed in our report, the design of USDA?s pilot project
    did not appropriately consider these variables.

    5. Our report accurately describes the duties of FSIS inspectors and plant
    personnel as they have changed during the course of the litigation outlined
    in appendix II. However, as noted in comment 9, we have modified our
    characterization of current plant personnel activities in accordance with
    USDA comments.

    6. We made the technical changes as appropriate. 7. We disagree. Our report
    does not describe the program as being

    designed as a cost- saving initiative.

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 78 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    8. We modified our report to state that there are only 11 chicken plants
    with before- and- after data.

    9. We have made the first change suggested. However, USDA has not provided
    any information to demonstrate that inspection resources are being utilized
    elsewhere in the farm- to- table continuum other than performing for plant
    inspections.

    10. We disagree. We believe that the report consistently describes the
    purpose and type of data collected by RTI. See comment 2.

    11. We made the technical changes as appropriate. 12. We made the technical
    changes as appropriate. 13. We made the technical changes as appropriate.
    14. We disagree. According to the documentation provided by USDA, there

    is a model for fed cattle and, as of October 2001, no cattle plants had
    volunteered to participate.

    15. Table 1 and its footnotes describe the number of plants participating in
    the project.

    16. We made the technical changes as appropriate. 17. We made the technical
    changes as appropriate. 18. We agree with the agency that it should look at
    this during the

    rulemaking process. 19. We disagree. Our report acknowledges that
    inspectors? verification

    checks increased from 20 to 80 at pilot project plants, a fourfold increase
    in verification checks. However, the number of noncompliance records issued
    by inspectors at some of the plants far exceeded the level expected by the
    increase in verification checks. At one plant, there was a twentyfold
    increase in noncompliance reports. Our intent is to alert USDA of the fact
    that, after shifting to modified inspections, some of the plants appear to
    have problems with fecal contamination. USDA?s HACCP Rules of Practice
    contemplate the Department?s taking action for noncompliance by withholding
    the marks of inspection or by suspending inspections in instances where it
    decides that it is necessary to do so because of multiple or recurring

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 79 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    regulatory noncompliances evidenced by noncompliance records. We understand
    from FSIS officials that they are now considering how many repeated
    instances of noncompliance will result in further regulatory action.

    20. We agree that actions taken when there are multiple instances of
    noncompliance are based on the same decision- making criteria in both
    project pilot plants and in traditional plants.

    21. About 60 percent of our survey respondents said that line speeds are
    somewhat too fast or much too fast for food safety.

    22. We made the technical changes as appropriate. 23. We agree with USDA?s
    comment that, as shown in table 5 of our report,

    the generic E. coli results show improvement after the plants shifted to
    modified inspections. As the table shows, 2 of the 11 plants improved while
    9 remained unchanged.

    24. We disagree. As stated in our report, we believe that looking at
    individual plant inspection systems? performance is relevant and appropriate
    because each plant is expected to meet the standards. Therefore, in table 4,
    our report shows how each plant performed, as measured by USDA?s performance
    standards.

    25. We made the technical changes as appropriate. 26. We disagree. As stated
    in our report, we believe that looking at

    individual plant inspection systems? performance is relevant and
    appropriate. Therefore, in table 6, our report shows how each plant
    performed, as measured by USDA?s performance standards.

    27. We agree. Our original criterion for analyzing changes in the data
    presented in- table 7 was .01 percent- which we used because it reflects the
    level of precision in the data that USDA provided to us. However, in
    response to USDA?s comment that declines in performance of fractions of 1
    percent may not be meaningfully different, we modified our criterion to 0.5
    percent. This means that a plant?s change from ?same? to ?worse? or ?better?
    would now represent 215 chickens per line per shift compared with 4.3
    chickens per line per shift in terms of quality and safety. By changing our
    analysis, we now find that in 16 instances, the plants? performance under
    modified inspections moved from ?better? to ?same? and in 8

    Appendix VII: Comments From the U. S. Department of Agriculture

    Page 80 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    instances the plants? performance moved from ?worse? to ?same.? In summary,
    even after changing our criterion, we find that in several instances, the
    plants are not meeting USDA?s goal- that is, that modified inspections
    should achieve at least the same level of performance as traditional
    inspections. We modified table 6 to reflect these changes. In table 6, we
    clearly identify which plants have or have not met each individual
    performance standard.

    28. We agree. As stated in our report, we believe that training of plant
    personnel is an issue that must be addressed by USDA.

    29. We made the technical changes as appropriate. 30. We made the technical
    changes as appropriate.

    Appendix VIII: GAO Contact and Staff Acknowledgments

    Page 81 GAO- 02- 59 Meat and Poultry Pilot Inspection Project

    Robert C. Summers (404) 679- 1839 In addition to the name above, Maria
    Cristina Gobin, John M. Nicholson, Diana P. Cheng, Shana B. Wallace, Fran A.
    Featherston, Karen K. Keegan, and Cynthia Norris made key contributions to
    this report. Appendix VIII: GAO Contact and Staff

    Acknowledgments GAO Contact Acknowledgments

    (360024)

    The General Accounting Office, the investigative arm of Congress, exists to
    support Congress in meeting its constitutional responsibilities and to help
    improve the performance and accountability of the federal government for the
    American people. GAO examines the use of public funds; evaluates federal
    programs and policies; and provides analyses, recommendations, and other
    assistance to help Congress make informed oversight, policy, and funding
    decisions. GAO?s commitment to good government is reflected in its core
    values of accountability, integrity, and reliability.

    The fastest and easiest way to obtain copies of GAO documents is through the
    Internet. GAO?s Web site (www. gao. gov) contains abstracts and full- text
    files of current reports and testimony and an expanding archive of older
    products. The Web site features a search engine to help you locate documents
    using key words and phrases. You can print these documents in their
    entirety, including charts and other graphics.

    Each day, GAO issues a list of newly released reports, testimony, and
    correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
    site daily. The list contains links to the full- text document files. To
    have GAO E- mail this list to you every afternoon, go to our home page and
    complete the easy- to- use electronic order form found under ?To Order GAO
    Products.?

    The first copy of each printed report is free. Additional copies are $2
    each. A check or money order should be made out to the Superintendent of
    Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
    copies mailed to a single address are discounted 25 percent. Orders should
    be sent to:

    U. S. General Accounting Office P. O. Box 37050 Washington, D. C. 20013

    To order by phone: Voice: (202) 512- 6000 TDD: (301) 413- 0006 Fax: (202)
    258- 4066

    GAO Building Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)
    Washington, D. C. 20013

    Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail:
    fraudnet@ gao. gov, or 1- 800- 424- 5454 (automated answering system).

    Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S.
    General Accounting Office, 441 G. Street NW, Room 7149, Washington, D. C.
    20548 GAO?s Mission

    Obtaining Copies of GAO Reports and Testimony

    Order by Mail or Phone Visit GAO?s Document Distribution Center

    To Report Fraud, Waste, and Abuse in Federal Programs

    Public Affairs
    *** End of document. ***