Consumer Product Safety Commission: Additional Steps Needed to Assess
Fire Hazards of Upholstered Furniture (Letter Report, 11/17/1999,
GAO/HEHS-00-3).
Pursuant to a legislative requirement, GAO provided information on the
Consumer Product Safety Commission's (CPSC) study on whether to set a
flammability standard for sofas, chairs, and other upholstered
furniture, focusing on the: (1) methodology CPSC used to estimate the
magnitude of the fire hazard that the standard would address; and (2)
reliability of this methodology for producing sound estimates of the
hazard that the standard would address.
GAO noted that: (1) because no single national data source exists on the
magnitude of hazards and losses caused by upholstered furniture fires,
CPSC blends information from two different sources; (2) one source
provides national estimates on the total number of fires in four general
categories and the extent of losses, but it provides no information
about specific types of fires, such as upholstered furniture fires; (3)
the second source provides detailed information for specific types of
fires, but for only a portion of all fires in the United States; (4)
CPSC uses the details from the second source of data and the national
estimates from the first source to calculate national estimates of fire
losses from the kinds of upholstered furniture fires the standard would
address; (5) CPSC cannot ensure that its methodology provides a complete
picture of the national fire losses that the potential standard would
address; (6) CPSC does not develop a statement of precision for the
estimated losses from upholstered furniture fires; (7) without this,
CPSC's estimates of fire losses do not adequately disclose the range of
benefits that may be associated with its potential standard; (8) CPSC's
methodology for calculating fire losses has the effect of including
losses that are not likely to be addressed by the standard; (9) fire
losses involving upholstered furniture are counted even though other
factors not addressed by the standard may have been responsible, such as
fires that are intentionally set; (10) also, for those fires for which
the cause or origin is unknown, CPSC assumes that upholstered furniture
fires will occur in the same proportion they occur in fires with a known
cause; (11) GAO's testing shows that these methods are likely to
substantially overstate fire losses that the standard would address, and
as a result, they could have a material effect on the associated
benefits expected from the potential standard; and (12) various analyses
can be used to assess the validity of underlying assumptions and
ultimately strengthen CPSC estimates, but so far CPSC has not used them.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-00-3
TITLE: Consumer Product Safety Commission: Additional Steps
Needed to Assess Fire Hazards of Upholstered Furniture
DATE: 11/17/1999
SUBJECT: Household goods
Product safety
Flammability standards
Safety regulation
Cost effectiveness analysis
Evaluation methods
Statistical data
IDENTIFIER: FEMA National Fire Incident Reporting System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Committees
November 1999
CONSUMER PRODUCT SAFETY COMMISSION
- ADDITIONAL STEPS NEEDED TO
ASSESS FIRE HAZARDS OF UPHOLSTERED
FURNITURE
GAO/HEHS-00-3
Furniture Flammability Standard
(101823)
Abbreviations
=============================================================== ABBREV
CDC - Centers for Disease Control and Prevention
CPSC - Consumer Product Safety Commission
NFIRS - National Fire Incident Reporting System
NFPA - National Fire Protection Association
USFA - U.S. Fire Administration
Letter
=============================================================== LETTER
B-282713
November 17, 1999
The Honorable Christopher Bond
Chairman
The Honorable Barbara A. Mikulski
Ranking Minority Member
Subcommittee on VA, HUD,
and Independent Agencies
Committee on Appropriations
United States Senate
The Honorable James T. Walsh
Chairman
The Honorable Alan B. Mollohan
Ranking Minority Member
Subcommittee on VA, HUD,
and Independent Agencies
Committee on Appropriations
House of Representatives
In 1994, the Consumer Product Safety Commission (CPSC) began
preliminary work on setting a flammability standard that would make
it harder for sofas, chairs, and other upholstered furniture to catch
fire. CPSC estimated at that time that nearly 700 people died
annually from such fires. During the 5 years that the agency has
been studying the potential standard, the need for such a standard
has remained an issue of considerable debate. Proponents, such as
fire protection groups, contend that without such a standard, the
public is subject to an unnecessary risk. Opponents, including
upholstered furniture manufacturers and the Small Business
Administration, respond that the magnitude of the problem is not
great enough to warrant the risks or added expense involved in
treating fabrics with flame-retardant chemicals or taking other steps
to make fabrics more flame-resistant. Opponents also contend that
more cost-effective solutions may exist, such as making more use of
smoke detectors to warn when furniture has caught fire.
To issue a flammability standard, the CPSC commissioners must
determine that the standard's benefits bear a reasonable relationship
to its costs.\1 To do this, CPSC assesses in quantitative terms
whether the lives and property saved would justify the additional
expense or risks associated with building furniture that complies
with the standard. In the conference report for CPSC's 1999
appropriations act,\2 the Congress directed us to review CPSC's
process for establishing a potential flammability standard.\3 Because
CPSC's work on the potential standard is still in process, we did not
conduct a comprehensive analysis of all aspects of the agency's
preliminary computation of the risks, costs, and benefits. As agreed
with your offices, we focused our efforts primarily on analyzing
CPSC's approach to quantifying the fire hazards that are the basis
for computing the standard's potential benefits. We addressed the
following questions:
-- What methodology does CPSC use to estimate the magnitude of the
fire hazard that the standard would address?
-- How reliable is this methodology for producing sound estimates
of the hazard that the standard would address?
To answer these questions, we reviewed the process, including the
underlying assumptions, that CPSC used to develop its initial
estimates of the magnitude of fire hazards from upholstered
furniture. We conducted limited tests of the effects of certain
assumptions on the estimated benefits of the standard. We also
looked at the capacity of CPSC's methodology to produce sound fire
hazard estimates when using the most current data available. We
conducted our work between May and October 1999 in accordance with
generally accepted government auditing standards. Appendix I
describes our scope and methodology in more detail.
--------------------
\1 15 U.S.C. 1193 (j)(2)(B) (1994).
\2 H.R. Rep. No. 105-769 at 267 (1998).
\3 CPSC's appropriations act directed the National Academy of
Sciences to study the potential toxicological risks of all
flame-retardant chemicals identified by CPSC and the Academy as
likely candidates for use in making upholstered furniture resistant
to ignition.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
Because no single national data source exists on the magnitude of
hazards and losses caused by upholstered furniture fires, CPSC blends
information from two different sources. One source provides national
estimates on the total number of fires in four general categories and
the extent of losses, but it provides no information about specific
types of fires, such as upholstered furniture fires. The second
source provides detailed information for specific types of fires, but
for only a portion of all fires in the United States. CPSC uses the
details from the second source of data and the national estimates
from the first source to calculate national estimates of fire losses
from the kinds of upholstered furniture fires the standard would
address.
At present, CPSC cannot ensure that its methodology provides a
complete picture of the national fire losses that the potential
standard would address. For example,
-- CPSC has not developed a statement of precision for the
estimated losses from upholstered furniture fires. Without
this, CPSC's estimates of fire losses do not adequately disclose
the range of benefits that may be associated with its potential
standard.
-- CPSC's methodology for calculating fire losses has the effect of
including losses that are not likely to be addressed by the
standard. Fire losses involving upholstered furniture are
counted even though other factors not addressed by the standard
may have been responsible, such as fires that are intentionally
set. Also, for those fires for which the cause or origin is
unknown, CPSC assumes that upholstered furniture fires will
occur in the same proportion they occur in fires with a known
cause. Our testing shows that these methods are likely to
overstate fire losses that the standard would address, and as a
result, they could have a material effect on the associated
benefits expected from the potential standard. Various analyses
can be used to assess the validity of underlying assumptions and
ultimately strengthen CPSC estimates, but so far CPSC has not
used them.
We are recommending that, as CPSC continues to consider the need for
a mandatory flammability standard for upholstered furniture, it
should conduct additional analyses to identify the level of
imprecision in the methodology's fundamental assumptions and apply
any necessary revisions to its cost-benefit analysis of the potential
standard.
BACKGROUND
------------------------------------------------------------ Letter :2
Although CPSC has reported that upholstered furniture fires account
for only 3 percent of all residential fires that occur each year,
these fires take a high toll in human life and property damage. CPSC
reports that of all the products the agency regulates, upholstered
furniture is the leading cause of household fire death. Under the
Flammable Fabrics Act,\4 which authorizes the issuance of
flammability standards for clothing, upholstery, and other products,
CPSC has the authority to issue mandatory performance and labeling
standards for upholstered products.
To address the hazards of upholstered furniture fires, the National
Association of State Fire Marshals petitioned CPSC in 1993 to issue a
flammability standard for residential upholstered furniture by
adopting three California standards.\5 In 1994, after studying
national estimates on the incidence and severity of household fires,
CPSC granted part of the petition. This part of the petition dealt
with small open flames such as matches and candles. By a 2-to-1
vote, CPSC published an Advance Notice of Proposed Rulemaking to
announce its consideration of issuing a flammability standard or
other regulation to address the risks posed by these upholstered
furniture fires. CPSC commissioners deferred action on the cigarette
portion of the petition in order to first assess the effectiveness of
and industry compliance with an existing voluntary furniture
flammability standard for cigarettes. However, on the basis of their
initial laboratory testing, CPSC staff now believe a small open-flame
standard will also address cigarette-caused fires.\6 As a result,
they have taken that assumption into account in developing estimates
of the fire losses the standard would address and the benefits it
would produce.
Proponents of a mandatory standard point to its ability to prevent
death, injury, and property damage as the major benefits. Opponents
recognize that these losses should be avoided, but they believe the
potential costs associated with the proposed standard are too great.
Possible costs cited include those related to health risks--both to
employees of furniture manufacturers and to consumers--posed by
flame-retardant chemicals that may be used; increased prices
consumers would have to pay for upholstered furniture to cover a
variety of additional manufacturing costs; diminished feel and
texture of fabric treated with flame-retardant chemicals; and loss of
consumer choice because some materials may be eliminated if they
cannot be made flame-resistant.
CPSC activities conducted since the Advance Notice of Proposed
Rulemaking have included in-depth fire investigations, technical
analyses and laboratory tests, development of a draft standard, and
initial estimates of the standard's potential costs and benefits.
CPSC's 1999 appropriations act prevents the standard from proceeding
past this stage until CPSC fully considers the fundings and
conclusions of the National Academy of Sciences' congressionally
mandated review on the potential toxicity of flame-retardant
chemicals. This review is to be completed in January 2000.
--------------------
\4 15 U.S.C. 1191 (1994).
\5 California has standards for upholstered furniture fires started
by large open flames in public buildings that lack a sprinkler
system, upholstered furniture fires started by smoking materials, and
those started by small open flames, such as candles. Separate
standards were requested for small open flames and cigarettes because
some materials have different probabilities of catching fire,
depending on the heat source.
\6 CPSC staff said that, for this reason, they will not take further
action on the cigarette portion of the petition until after the
technical work for the small-open-flame portion is completed.
CPSC USES TWO DATA SOURCES TO
ESTIMATE THE MAGNITUDE OF THE
UPHOLSTERED FURNITURE FIRE
PROBLEM
------------------------------------------------------------ Letter :3
To estimate the incidence of fires and losses resulting from
upholstered furniture being ignited by cigarettes or small open
flames, CPSC uses aggregate fire data from a national survey,
detailed data on a portion of individual fires, and a methodology to
combine the two databases. This approach is necessary because no
single data source exists that provides the information needed to
estimate the magnitude of the problem the standard is intended to
address. The aggregate survey data provide national fire loss
estimates for four general categories of fires (such as residential
or vehicle fires) but provide no information about detailed
characteristics of the fires. This information is from a survey
conducted by the National Fire Protection Association (NFPA), a
nonprofit fire protection association. In contrast, the second
source consists of detailed information linking fire losses--deaths,
injuries, and damages--with specific types of fires, such as those
originating in upholstered furniture. However, the detailed data do
not provide national totals and are not collected in a way that
permits them to be used by themselves for national projections. The
detailed data are collected from more than a third of the nation's
fire departments and are found in the National Fire Incident
Reporting System (NFIRS), a federal database. This database is
maintained by the U.S. Fire Administration (USFA), within the
Federal Emergency Management Agency. The NFIRS data contain
information on more than 20 different fire characteristics, such as
the source of the fire and what caught fire first.
Because neither NFPA estimates nor the NFIRS data can produce a
nationwide estimate of fire losses for specific types of fires, CPSC
uses a methodology for combining the two. The process of blending
the two sources of data was developed jointly by NFPA, CPSC, and USFA
to approximate and report on fire trends for the fire-fighting
community and the general public. CPSC, NFPA, and USFA use this
general process to track and report on fires and fire losses
nationally. They have all used this process for many years and try
to apply it consistently so that conflicting estimates are not
produced, which could confuse the public and those who use the data.
The methodology is based on the proportional relationship of the
NFIRS data to the NFPA estimates. If NFIRS data contain half of the
total number of fires estimated by NFPA, for example, NFIRS numbers
for specific types of fires are doubled to produce national estimates
for these specific fires. The NFIRS data also include a number of
fires for which the first item ignited is unknown. The methodology
adds a proportional number of these fires and fire losses to the
NFIRS fires known to have started in upholstered furniture. In the
1994 NFIRS data, for example, upholstered furniture fires constituted
3 percent of all fires with a known item of origin. Therefore, 3
percent of the fires with an unknown item of origin also would be
allotted to upholstered furniture fires.
EXTENT OF FIRE LOSSES IS
UNCERTAIN
------------------------------------------------------------ Letter :4
CPSC has not fully addressed the uncertainty surrounding key data and
underlying assumptions that it uses in developing national fire loss
estimates. Particularly important are (1) the need to account for
imprecision surrounding the data used to make national fire loss
estimates and (2) the assumption that the potential standard could
address all upholstered furniture fires classified as being ignited
by either small open flames or cigarettes. So far, CPSC has not
accounted for this imprecision or tested the soundness of this
assumption, and as a result, the full range of fire losses is not
known. Our limited testing demonstrates the importance of these
steps in developing sound estimates of fire losses that the standard
is designed to address.
CPSC HAS NOT TESTED KEY
ISSUES
---------------------------------------------------------- Letter :4.1
Several issues raise uncertainties about the underlying assumptions
related to the NFPA and NFIRS data (see table 1). These issues
include the low response rates to the NFPA survey, the voluntary
nature of reporting NFIRS data, and the fact that some upholstered
furniture fires started by small open flames or cigarettes will not
be addressed by the standard. These issues raise questions about
whether the fire and fire loss data provided by NFIRS and NFPA result
in a representative picture of the incidence and severity of
residential fires that occur nationally. In relying on these data
for standard-setting, it is important to understand whether the data
are representative and, if they are not, whether they overestimate or
underestimate the national fire problem.
Table 1
Key Issues Potentially Affecting the
Validity of CPSC Estimates
Subject of
preliminary
Issues that raise testing by
Assumption uncertainty GAO
------------------ ------------------------ ------------
NFPA survey data
----------------------------------------------------------
NFPA national fire Low response rate (21 Yes
and fire loss percent of fire
estimates are departments surveyed)\a
representative of and limited review of
fires nationwide. nonrespondents results
in uncertainty about
extent that national
projections accurately
represent the national
fire problem.
No outside review of the No
analysis used to produce
national estimates was
conducted. Such a
review, usually
conducted by a party
with a background in
methodology and
statistics, is commonly
used to help identify
flaws or constraints in
estimating
methodologies.
Potential exists for No
survey respondents to
provide inaccurate
information. Limited
follow-up or
corroboration of survey
responses is a way to
ascertain the degree to
which these responses
are accurate and
supported.
NFIRS data
----------------------------------------------------------
Any upholstered Some upholstered Yes
furniture fire furniture fires that
classified as were started by
being ignited by a cigarettes or small open
small, open flame flames also involve
or cigarette could other factors that place
be addressed by them outside the
the potential category of fires the
standard. standard addresses,
raising concern that too
many fire losses may be
linked to the potential
standard.
Fire and fire loss Voluntary reporting Yes
data reported by carries the potential
local fire that certain types of
departments are fire departments, such
representative of as large fire
fires that occur departments, will be
nationally. over-or underrepresented
in the data, or will
only report major fires,
which would affect the
extent to which data are
representative of the
fire problem nationally.
There is potential for No
data reported by local
fire departments to be
inaccurate. Limited
verification and
corroboration of
reported data is a way
to ascertain the level
of accuracy of the
reported data.
----------------------------------------------------------
\a Fire departments protecting a larger number of people (over
50,000) responded to the survey at a rate more than twice that of
departments protecting fewer people (under 50,000). As a result,
NFPA reported that it received responses from fire departments that
serve 40 percent of the population.
TESTING OF ISSUES SHOWED
POTENTIAL FOR INACCURACY
---------------------------------------------------------- Letter :4.2
We selectively tested three key issues that affect the validity of
CPSC's estimates:
-- the extent to which the data used to project NFPA estimates
provide nationally representative and precise estimates,
-- the extent to which data on fire losses that the potential
standard does not address are included in developing estimates
of fire losses the standard is expected to address, and
-- the extent to which fire departments in the NFIRS database are
representative of fire departments nationally and provide
nationally representative fire data.
Our selective testing showed mixed results, indicating that CPSC
needs to more fully analyze some of its assumptions in order to
provide greater assurance that its fire loss estimates are valid.
REPRESENTATIVENESS AND
PRECISION OF NFPA
ESTIMATES
-------------------------------------------------------- Letter :4.2.1
Because of the overall low response rate to the NFPA survey (21
percent), we conducted a limited test to assess the accuracy of NFPA
estimates. Corroborating survey projections to another data source
is a common way of assessing the representativeness of survey data.
This type of comparison can be a general gauge of how well the survey
represents the nation as a whole. To do this, we compared the NFPA
death estimates with the Centers for Disease Control and Prevention
(CDC) death estimates. CDC obtains this information from death
certificates recorded in the 50 states and the District of Columbia.
The data classify deaths by the underlying cause of death, which is
determined from the death certificate information.\7 Our testing
indicated that CDC death data fall within the bounds of NFPA's
estimates. On the basis of this general test, the data appear to
provide representative estimates of fire deaths. We did not assess
the representativeness of the NFPA national estimates of injuries and
property damage from fires.
To provide a complete picture of fire losses, CPSC needs to account
for the range of precision around NFPA's estimates. Because the
estimates are based on a survey, they have a margin of error that
needs to be incorporated into national upholstered furniture fire
loss estimates. Survey researchers routinely report the margin of
error in the results of their surveys and polls. CPSC has not
accounted for the margin of error around NFPA estimates for deaths,
injuries, and property damage. Although CPSC stated that the margin
of error for the total number of fires (2.5 percent) was not
significant for its estimates, we found that the margins of error for
deaths and injuries were considerably larger than the margin of error
for the total number of fires. In 1997, the largest margin of error
was for the estimated number of deaths, which could be off by as much
as 350 deaths (10 percent) in either direction. Because deaths,
injuries, and property damages are the factors for estimating losses
from the kinds of fires covered under a potential standard, they all
need to be considered. By not obtaining and using all of this
information in developing its estimates of upholstered furniture fire
losses, CPSC runs the risk of conveying a false sense of precision
about its results.
--------------------
\7 While these data may not be 100-percent accurate either, they are
subject to numerous tests and checks and are widely used within the
health research community.
CPSC'S CALCULATION OF
FIRE LOSSES
-------------------------------------------------------- Letter :4.2.2
CPSC's methodology for estimating fire losses that the potential
standard is expected to address also warrants additional refinement.
First, in estimating total fire losses, CPSC identifies all fires in
NFIRS data known to have involved the ignition of upholstered
furniture by small open flames or cigarettes.\8 Second, when NFIRS
data lack complete information to link fire losses to a specific type
of fire, CPSC classifies a portion of these fires as upholstered
furniture fires involving small open flames or cigarettes. Our work
shows that both of these procedures are likely to overstate the fire
losses the potential standard is designed to address, and as a
result, they have a material effect on the benefits ascribed to the
standard.
With regard to counting fires for which the origin and cause are
known, CPSC includes upholstered furniture fires that were classified
in NFIRS as being started by small open flames or cigarettes.
However, our examination of the same data shows that other factors
were involved in the cause of some of these fires, making them fires
the standard would not appear to address. For example, the standard
is not designed to address fires intentionally set or those in which
a small open flame ignites a paper or flammable liquid that is on or
near an upholstered couch that in turn ignites the couch. CPSC,
however, has not yet excluded these types of fires in its estimate.
CPSC's allocation of unknown fire losses also warrants additional
attention. In 1997, about 35 percent of the residential fire deaths
in the NFIRS data were not identified by what first caught fire, such
as upholstered furniture, draperies, chimneys, or walls. When NFIRS
data lack complete information to link fire losses to a specific type
of fire, CPSC allocates these losses on the basis of how the fire
loss data are distributed across all fires in which the specific type
of fire is known. In other words, if 19 percent of all residential
fire deaths are attributed to upholstered furniture, the same
percentage of deaths that stem from fires not attributed to specific
types of fires is assigned to upholstered furniture fires. CPSC
officials told us they proportionally allocated unknown fire loss
data because they had no evidence that the data should be allocated
differently.
Methods and data exist that could categorize the origin and cause of
fires better before calculating the fire losses the standard could
possibly address. The NFIRS data are useful in this regard because
information available in NFIRS allows certain types of fires to be
ruled out as likely to be addressed by the standard. As a result,
the NFIRS data can be used to test the effect of CPSC's method of
calculating total fire loss data.
We tried such a test using NFIRS data for 1997. We analyzed detailed
information on fires--those known to be upholstered furniture fires
started by a small open flame or cigarette and those with an unknown
origin or cause--looking for characteristics to identify fires that
the potential standard would not appear to address. For example,
fires we recategorized include those that were intentionally set,
those involving electrical short circuits, and those involving
flammable materials not used to upholster furniture (such as gases,
flammable liquids, or cardboard). Our reclassification of fires
reduced CPSC's estimate of fire losses that the standard is designed
to address by up to 152 deaths, 434 injuries, and $70 million in
property damages (see app. I for more details on these reductions).
Adjusting for this reclassification of fires could reduce the
benefits CPSC attributes to the standard from $885 million to $610
million, or 31 percent of the total benefits. On the other hand,
additional and more thorough investigation of some fires by CPSC
could identify other factors that show that some of these fire losses
would be prevented by the standard.
Also, our test should not be taken as the most definitive or
comprehensive approach to this issue. For example, more
comprehensive analytical approaches to assess and allocate unknown
fire losses could yield results that would make the estimates more
precise. Our selective testing was done mainly to determine whether
the underlying assumption of CPSC's approach appears sound. Our
results indicate that some of CPSC's current assumptions have a level
of uncertainty that is material and as such warrants attention.
--------------------
\8 As pointed out earlier, CPSC has determined that the potential
standard will address both small-open-flame and cigarette fires, and
as a result, CPSC estimates fire losses for both types of fires.
REPRESENTATIVENESS OF THE
FIRE DEPARTMENTS THAT REPORT
NFIRS DATA
---------------------------------------------------------- Letter :4.3
NFIRS data are based on reports submitted by a little more than a
third of the 31,000 fire departments nationwide. Because the fire
departments reporting to NFIRS do so voluntarily, it is important to
determine how representative this group is of fire departments and
fire experiences as a whole. For example, 11 states have no fire
departments reporting information to NFIRS, and only a portion of the
fire departments in the other states participate. If certain types
of fire departments, such as large urban fire departments, are more
likely or less likely to report to NFIRS, or if fire departments tend
to report only major fires, this could skew estimates of the fire
problem.
Testing the extent that NFIRS data are representative of fire and
fire losses that occur nationally can be conducted in several ways.
We did so by assessing the extent to which the types of fire
departments reporting data to NFIRS resembled the types of fire
departments nationally on the basis of two factors: the size of the
fire department and whether it is predominantly a volunteer or a paid
fire department. This analysis indicated that NFIRS-reporting fire
departments fall into the various fire department size and type
categories at roughly the same proportions as all fire departments
across the country. As a result, any differences in fire losses that
is related to fire department size or type are probably appropriately
represented in the NFIRS data.
EFFECT OF SELECTED TESTING
ON COST-BENEFIT ESTIMATES
---------------------------------------------------------- Letter :4.4
Although the selected testing we conducted resulted in a decline in
estimated benefits expected from the potential standard, the tests
should not be taken as a comprehensive assessment of CPSC's overall
cost-benefit calculation. For example, we did not review or evaluate
CPSC's assumptions about the effectiveness of the potential standard
in preventing upholstered furniture fires ignited by small open
flames or cigarettes, which also have a major impact on the estimated
benefits of the standard. We also did not assess the validity or
scientific basis for CPSC's assumptions concerning the extent to
which a small-open-flame standard will also address cigarette-caused
fires. CPSC estimated that the potential standard would prevent 80
percent of the small-open-flame losses and 50 percent of the
cigarette-related losses.\9 However, this key assumption is subject
to further laboratory testing by CPSC.
Also, as stated earlier we did not analyze assumptions and
methodologies used to estimate the costs of the standard. It is
important to recognize that the testing of other issues and
assumptions--especially those relating to costs--may offset, at least
in part, the decline in estimated potential benefits that our testing
found.\10 As a result, we present no estimates of the effect our
testing has on the net costs or benefits of the potential standard.
Rather, our testing demonstrates the need for CPSC to further refine
its analysis.
--------------------
\9 In applying this rate of effectiveness, CPSC also reduced
cigarette-related losses by an additional 24 percent to account for
the effect of the existing voluntary industry flammability standard.
\10 It is also possible that this testing would lower the estimates
even further.
CONCLUSIONS
------------------------------------------------------------ Letter :5
To promulgate a mandatory standard, the CPSC commissioners must
determine that benefits produced by the standard bear a reasonable
relationship to its costs. A high degree of assurance about the
soundness of these estimates is an important part of accurately
assessing this relationship. As matters stand, CPSC's current
approach is not likely to generate the necessary degree of assurance.
To provide this assurance, CPSC needs to demonstrate, to a greater
degree than now exists, the validity of the assumptions on which the
estimate is based. This will require testing that so far has not
been part of CPSC's approach.
RECOMMENDATIONS
------------------------------------------------------------ Letter :6
To resolve issues surrounding the data and assumptions used in
preparing the cost-benefit analysis for a potential standard to
protect against fire hazards associated with upholstered furniture,
we recommend that the Chairman, CPSC, direct CPSC staff to conduct
additional and more detailed analyses of key assumptions including,
but not limited to,
-- assessing the precision surrounding NFPA national fire loss
estimates and their impact on estimated benefits attributable to
the standard and
-- identifying a more accurate method to calculate fire losses that
could be addressed by the standard.
We also recommend that any necessary revisions identified by these
analyses should be incorporated into the cost-benefit analysis of the
potential standard.
AGENCY COMMENTS AND OUR
RESPONSE
------------------------------------------------------------ Letter :7
In its comments on our draft report, CPSC agreed with our
recommendations. It stated that CPSC staff have already begun to
address the issues we raised as they refine their fire loss estimates
for upholstered furniture. (See app. II for the complete text of
CPSC's comments.)
However, CPSC stated that the report overall downplayed the positive
findings about CPSC data and overemphasized the data problems. In
fact, after reviewing our report, CPSC provided us with new
information and perspective on the representativeness of the NFPA
data. After considering this additional information, we modified our
draft to agree with CPSC that our limited testing did not disclose
any obvious data problems. However, this does not indicate that the
data are fully reliable, because our testing did not address all the
issues surrounding the underlying data. For example, NFPA and NFIRS
data are based on data reported by individual fire departments that
have not been subject to verification or corroboration. As a result,
we believe our report, as modified, presents an appropriate
characterization of our results and the remaining uncertainties
surrounding the underlying data.
CPSC also commented that our analysis overstated the significance of
the issues surrounding NFIRS data. CPSC concluded that it was not
appropriate for us to quantify the effect of these data problems and
said that our analysis was flawed for two reasons:
-- Our analysis eliminated those fires that available information
indicated were inconsistent with the scope of the potential
standard. However, CPSC stated that this process could result
in eliminating too many fires.
-- Our analysis erroneously adjusted the fire data where the cause
was unknown. CPSC commented that we did not include the correct
proportion of fires with unknown origin or source in arriving at
a total number of fires that the potential standard is designed
to address.
We do not agree with CPSC's conclusion. First, our approach was to
determine whether data classification issues could have a material
effect on CPSC's estimated losses that the potential standard would
address. We believe our analysis was a reasonable one for this
purpose and that it shows a likelihood of significant effect that
warrants attention. The report appropriately qualifies the results
and acknowledges that our tests should not be taken as the most
definitive or comprehensive approach. Rather, it is CPSC's
responsibility to develop more precise estimates, as it proceeds in
its rulemaking process.
Second, we believe that our approach properly distributed the number
of "unknown" fires to the total number of fires that the potential
standard is designed to address. In our opinion CPSC's approach
overstates the true proportion because it classifies as unknown some
fires that available data indicate are not subject to the proposed
regulation. Consequently, our methodology corrects the data to
account for the greater chance of containing upholstered furniture
cases. In making this adjustment, contrary to CPSC's comments, we
did not eliminate fires from consideration. Rather, we reclassified
fires as known not to be upholstered furniture fires that the
potential standard is designed to address. We then recalculated the
proportion of unknown fire losses on the basis of our
reclassification.
Finally, CPSC commented that our recalculation of its estimated fire
losses, adjusted for reclassification of unknown fire origin or
cause, would overstate the reduction in potential benefits. Our
report indicates that, while this recalculation is an upper-bound
estimate for this factor, further investigation and analysis could
also influence the magnitude of estimated fire losses. CPSC's
continuing efforts to refine its analysis--of estimated costs as well
as benefits associated with its proposed standard--will be central to
its ongoing work to determine the merits of proceeding with a
mandatory flammability standard. In addition to CPSC's written
comments, CPSC staff provided us with oral comments, which we
incorporated, where appropriate, in the final report.
---------------------------------------------------------- Letter :7.1
We are sending copies of this report to the Honorable Ann Brown,
Chairman, and the Honorable Thomas H. Moore and the Honorable Mary
Sheila Gall, Commissioners, CPSC; and appropriate congressional
committees. We will also make copies available to others upon
request.
If you or your staffs have any questions about this report, please
contact me at (202) 512-7118 or Frank Pasquier at (206) 287-4861.
Major contributors to this report include Tim S. Bushfield, Evan
Stoll, and Stan Stenersen.
Kathryn G. Allen
Associate Director, Health Financing
and Public Health Issues
SCOPE AND METHODOLOGY
=========================================================== Appendix I
Our work focused on the methodology and data that the Consumer
Product Safety Commission (CPSC) is using to estimate national fire
losses from the kinds of upholstered furniture fires that would be
covered by the potential flammability standard now being considered.
We used a variety of sources and methods to gather and analyze data
on this issue.
RANGE OF INFORMATION SOURCES
USED
--------------------------------------------------------- Appendix I:1
We obtained information from a wide variety of agencies, interest
groups, and other sources. Among the key sources were the following:
-- CPSC officials, including those responsible for calculating
national fire loss estimates and estimating the potential
benefits that would result from the standard;
-- National Fire Protection Association (NFPA) officials
responsible for conducting NFPA's annual fire surveys,
calculating the survey's national fire and fire loss estimates,
and involved in developing the methodology that combines NFPA's
survey projections with detailed fire data collected by local
fire departments;
-- U.S. Fire Administration officials responsible for overseeing
and analyzing the detailed fire and fire loss data reported to
the agency's National Fire Incident Reporting System (NFIRS);
-- officials from the National Center for Health Statistics and the
National Center for Injury Prevention and Control at the Centers
for Disease Control and Prevention (CDC);
-- officials representing various segments of the upholstered
furniture manufacturing industry, such as fiber manufacturers
and upholstered furniture manufacturers; and
-- private consultants who use the NFPA and NFIRS data to estimate
national fire problems.
Our work also included a review of relevant legislation and other
documents related to CPSC's approach to considering the potential
standard.
ANALYZING THE METHODOLOGY FOR
MAKING NATIONAL FIRE LOSS
ESTIMATES
--------------------------------------------------------- Appendix I:2
To address the study's first question (the methodology used to
estimate the magnitude of the fire hazard that would be addressed by
the standard), we obtained documentation describing the process and
examples illustrating the data and major steps in the process from
CPSC, NFPA, and the U.S. Fire Administration. In addition to
reviewing documents and conducting interviews to learn how these
estimates had been developed, we also obtained and analyzed the 1997
NFIRS database of actual fire incidents as well as detailed
information about the 1997 NFPA survey, including some of the
characteristics of the fire departments that responded to it.
TESTING THE VALIDITY OF
UNDERLYING ASSUMPTIONS AND THE
SOUNDNESS OF NATIONAL ESTIMATES
--------------------------------------------------------- Appendix I:3
To address the study's second question (the reliability of this
methodology to produce sound fire hazard estimates), we first
identified the assumptions underlying key data and methodological
steps in the process. We reviewed each data source separately to
identify the analytical and statistical limitations accompanying the
assumptions on which the information was based. When we identified
limitations, we compared CPSC's process to date with various
analytical and statistical methods for testing reliability. For
example, we assessed the potential implication of NFPA's response
rate on the soundness of the data, determined what statistical
techniques could help assess the effect of these uncertainties, and
spoke with CPSC staff to determine the extent to which they had
conducted their own tests of these uncertainties. We focused more
detailed testing on three specific assumptions: the
representativeness of data used to make NFPA's national fire
estimates; the representativeness of data in NFIRS; and CPSC's
classification of fires the standard would address, including fires
with and without a known item of origin or ignition source.
To assess the representativeness of the data used in the national
fire estimates, we conducted two types of analysis. First, we
compared NFPA's national estimate of total fire deaths with the
number of national fire deaths compiled by CDC. Because CDC obtains
information from all death certificates in all 50 states and the
District of Columbia, we concluded that the data were a valid basis
for comparison. Second, we obtained from NFPA information on the
margin of error around its national estimates of fires, deaths,
injuries, and property damages.
To assess the representativeness of data in NFIRS, we compared the
distribution of NFIRS fire departments by size with the distribution
of fire departments nationally. We did so by categorizing fire
departments by using the size of the population they protected as a
proxy for fire department size. To do this, we needed more data than
were available in NFIRS. We matched NFIRS fire departments with the
NFPA database and used the NFPA population data in those cases where
a fire department appeared on both databases. We also conducted a
similar analysis comparing the distribution of fire departments based
on the type of fire fighting personnel employed (paid or volunteer)
and the severity of fires reported by each category of fire
department.
To evaluate CPSC's calculation of fire losses expected to be
addressed by the potential standard, we used data available in NFIRS
to recategorize fires the standard is not designed to address. We
conducted this analysis using information from CPSC and 1997 NFIRS
data. To identify those fires the potential standard is not designed
to address, we used other information in the NFIRS database such as
the type of material ignited first, the situation that resulted in
contact between a heat source and flammable material, the equipment
involved in the ignition, and the area of the house where the fire
started.
As a result of these analyses, adjustments were made to (1) the raw
number of upholstered furniture fires and fire losses known to have
resulted from small open flames or cigarettes; (2) the raw number of
fires with an unknown origin and/or heat source that are allocated to
upholstered furniture fires the standard is intended to address; and
(3) the weights, or multipliers, used to project the raw NFIRS
numbers into national estimates. Table I.1 shows the effect of these
adjustments on fire loss estimates that the standard is designed to
address.
Table I.1
Reduction in Fire Loss Estimates Based
on Recategorization of Fires that the
Standard Is Not Designed to Address
Recategorize Recategorize
d fires with d fires with
a known an unknown
source or source or
Fire loss type origin origin Total
---------------- ------------ ------------ ------------
Deaths 87 65 152
Injuries 296 138 434
Property damage $48 million $22 million $70 million
----------------------------------------------------------
To determine the impact these adjustments have on CPSC's estimate of
potential benefits, we used a two-step process. First, we compared
the total cost of fire losses based on national estimates resulting
from our adjustments with the total costs from CPSC's national fire
loss estimates. Then we applied the difference between CPSC's and
our total cost estimates to CPSC's estimate of potential benefit from
the standard.
A general limitation of our analysis was that it did not review the
scientific basis for the effectiveness of the potential standard.
Laboratory testing to show the extent that the standard would prevent
different types of upholstered furniture fires is ongoing, according
to CPSC. Rather, the scope of our work was limited to reviewing
whether the methodology for using existing data was sufficiently
reliable to produce sound fire loss estimates.
(See figure in printed edition.)Appendix II
COMMENTS FROM THE CONSUMER PRODUCT
SAFETY COMMISSION
=========================================================== Appendix I
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
*** End of document. ***