Particulate Matter: EPA Has Started to Address the National	 
Academies' Recommendations on Estimating Health Benefits, but	 
More Progress Is Needed (17-JUL-06, GAO-06-780).		 
                                                                 
A large body of scientific evidence links exposure to particulate
matter--a widespread form of air pollution--to serious health	 
problems, including asthma and premature death. Under the Clean  
Air Act, the Environmental Protection Agency (EPA) periodically  
reviews the appropriate air quality level at which to set	 
national standards to protect the public against the health	 
effects of particulate matter. EPA proposed revisions to these	 
standards in January 2006 and issued a draft regulatory impact	 
analysis of the revisions' expected costs and benefits. The	 
estimated benefits of air pollution regulations have been	 
controversial in the past. A 2002 National Academies report	 
generally supported EPA's approach but made 34 recommendations to
improve how EPA implements its approach. GAO was asked to	 
determine whether and how EPA applied the Academies'		 
recommendations in its estimates of the health benefits expected 
from the January 2006 proposed revisions to the particulate	 
matter standards. GAO examined the draft analysis, met with EPA  
officials, and interviewed members of the National Academies'	 
committee. In providing technical comments on the report, EPA	 
officials said it was fair and balanced and noted the agency's	 
progress in addressing recommendations via research and 	 
development and other analyses. 				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-06-780 					        
    ACCNO:   A56980						        
  TITLE:     Particulate Matter: EPA Has Started to Address the       
National Academies' Recommendations on Estimating Health	 
Benefits, but More Progress Is Needed				 
     DATE:   07/17/2006 
  SUBJECT:   Air pollution					 
	     Air pollution control				 
	     Air quality					 
	     Environmental law					 
	     Environmental policies				 
	     Environmental research				 
	     Health hazards					 
	     Pollutants 					 
	     Public health					 
	     Standards						 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-06-780

     

     * Report to Congressional Requesters
          * July 2006
     * PARTICULATE MATTER
          * EPA Has Started to Address the National Academies'
            Recommendations on Estimating Health Benefits, but More Progress
            Is Needed
     * Contents
          * Results in Brief
          * Background
          * EPA Is in the Process of Addressing Many of the Academies'
            Recommendations
               * EPA Applied, at Least in Part, about Two-thirds of the
                 Recommendations to Its Particulate Matter Health Benefit
                 Analysis in the Proposed Rule
               * EPA Plans to Address Some of the Remaining Recommendations
                 in the Final Rule and Has Research and Development Under Way
                 to Address Others
          * Concluding Observations
          * Agency Comments
     * Scope and Methodology
     * Recommendations Applied or Partially Applied to the Draft Particulate
       Matter Regulatory Impact Analysis
     * Recommendations Not Applied to the Draft Particulate Matter Regulatory
       Impact Analysis
     * GAO Contact and Staff Acknowledgments
     * d06780TOC.pdf
          * Contents

Report to Congressional Requesters

July 2006

PARTICULATE MATTER

EPA Has Started to Address the National Academies' Recommendations on
Estimating Health Benefits, but More Progress Is Needed

Contents

Tables

July 14, 2006Letter

The Honorable James M. Inhofe Chairman Committee on Environment and Public
Works United States Senate

The Honorable George Voinovich Chairman Subcommittee on Clean Air, Climate
Change,    and Nuclear Safety Committee on Environment and Public Works
United States Senate

A large body of scientific evidence links exposure to particulate matter-a
ubiquitous form of air pollution commonly referred to as soot-to serious
health problems, including asthma, chronic bronchitis, heart attack, and
premature death. The many sources releasing particulate matter into the
air include cars, trucks, power plants, industrial processes, forest
fires, and waste incinerators. In 1971, the Environmental Protection
Agency (EPA) first established national air quality standards to protect
the public against the health effects of particulate matter, one of the
six widespread criteria pollutants considered harmful to public health.
Under the Clean Air Act, EPA determines the appropriate level at which to
set national air quality standards, and the states must develop programs
to achieve and maintain compliance with them. Further, EPA must review the
standards every 5 years to determine whether they adequately protect human
health and welfare, given the latest scientific information available, and
revise them if they do not.

In January 2006, after its most recent review of the national air quality
particulate matter standards, EPA proposed revisions to the standards and
issued a draft regulatory impact analysis. Overall, the draft regulatory
impact analysis discussed the scope and magnitude of the particulate
matter problem, the likely benefits of the proposed revisions for public
health and the environment, and the expected costs of implementing the
standards. Regarding public health, the analysis presented estimates of
expected health benefits for the particulate matter revisions in five
major urban areas, including reductions in the number of premature deaths
and emergency room visits for asthma. Among the changes EPA said it plans
to make in its final regulatory impact analysis is providing national
estimates of expected health benefits. EPA is required under a court order
to issue the final rule on particulate matter standards by September
2006.1

In 2000, at the direction of the Senate Appropriations Committee, EPA
asked the National Academies (Academies) to evaluate EPA's methodology for
estimating the health benefits of proposed air pollution regulations.2
According to the National Academies, these estimates have often been
controversial, and the methods EPA has used to prepare them have been
questioned. For example, some observers, such as researchers and industry
groups, have expressed concerns that EPA does not adequately factor
uncertainty into its estimates of health benefits. Some level of
uncertainty is unavoidable, in part because the scientific information
used to develop estimates, such as the inventory of particulate matter
emissions, will never be perfect or complete. However, according to some
observers, EPA's estimates of benefits appear more definitive than they
really are because the agency does not adequately account for uncertainty
in its analyses or in its reporting of health benefit estimates.

The National Academies' 2002 report on this subject generally supported
EPA's approach to estimating health benefits but, nevertheless, made 34
detailed recommendations to improve how EPA implements its approach.3
Overall, these recommendations focus on conducting more rigorous
assessments of uncertainty, increasing the transparency of how EPA
estimates benefits, conducting more detailed analyses of exposure, and
estimating the benefits of each regulatory option under consideration.
Many of the recommendations include qualifying language indicating that it
is reasonable to expect that they can be applied in stages, over time; and
a number of the recommendations are interrelated and, in some cases,
overlapping.

You asked us to determine whether and how EPA applied the National
Academies' recommendations in its estimates of the health benefits
expected from the January 2006 proposed revisions to the particulate
matter standards. To respond to this objective, we reviewed EPA's draft
regulatory impact analysis presenting the costs and benefits of the
proposed rule and met with senior officials from EPA's Office of Air and
Radiation, which was responsible for developing the proposed rule and
analyzing its costs and benefits, and with officials from EPA's Office of
Policy, Economics, and Innovation. We also reviewed EPA's and the Office
of Management and Budget's (OMB) guidance on conducting economic analyses,
prior GAO reports on EPA's regulatory impact analyses, and other relevant
reports. As requested, our work addressed the application of the National
Academies' recommendations to EPA's draft regulatory impact analysis
supporting the 2006 proposed particulate matter rule; thus, we did not
examine how EPA applied the recommendations to other recent air rules. Our
work focused on broadly characterizing EPA's progress toward applying the
recommendations; we did not evaluate the effectiveness and quality of the
scientific and technical actions the agency has taken to apply the
recommendations. See appendix I for a more detailed description of the
scope and methodology of our review. We performed our work from January
2006 to July 2006 in accordance with generally accepted government
auditing standards.

Results in Brief

EPA has begun to change the way it conducts and presents its analyses of
health benefits in response to the National Academies' recommendations,
which focused on conducting more rigorous assessments of uncertainty,
increasing the transparency of how EPA estimates benefits, conducting more
detailed analyses of exposure, and estimating the benefits of each
regulatory option under consideration. EPA applied, at least in part,
about two-thirds of the recommendations to its health benefit analysis.
Specifically, of the 34 recommendations, EPA applied 8 and partially
applied 14. For example, EPA responded to some of the recommendations by
taking steps to conduct a more rigorous assessment of uncertainty by, for
instance, evaluating how benefits might change given alternative
assumptions and discussing sources of uncertainty not included in the
benefit estimates. More specifically, EPA applied an alternative technique
for evaluating one important source of uncertainty in its analysis-the
uncertainty underlying the causal link between exposure to particulate
matter and premature death. Consistent with the National Academies'
recommendation to assess uncertainty by developing ranges of estimates of
benefits and specifying the likelihood of attaining that level of
benefits, EPA systematically gathered expert opinions about this
link-through a process called expert elicitation-and developed ranges
reflecting the experts' confidence in attaining reductions in premature
death expected from the proposed revisions. However, the health benefit
analysis does not similarly assess how the benefit estimates would vary in
light of other key uncertainties as the Academies had recommended.
Consequently, this represents a partial application of the recommendation.
Agency officials told us that ongoing research and development efforts
will allow EPA to gradually achieve more progress in applying this and
other recommendations to future analyses.

For various reasons, EPA did not apply the remaining 12 recommendations to
the analysis, such as the recommendation to evaluate the impact of using
the assumption that the components of particulate matter are equally
toxic. EPA officials viewed most of these recommendations as relevant to
its health benefit analyses, but noted that the agency was not ready to
apply specific recommendations because of, among other things, the need to
overcome technical challenges stemming from limitations in the state of
available science. These officials emphasized the agency's commitment to
continue to respond to the recommendations. For example, EPA did not
believe that the state of scientific knowledge on the relative toxicity of
particulate matter components was sufficiently developed to include it in
the January 2006 regulatory impact analysis, and the agency is sponsoring
research on this issue. In addition, according to a senior EPA official,
insufficient resources have impeded the agency's progress in applying the
recommendations, including the limited availability of skilled staff,
time, and other resources to conduct and oversee the required analyses and
research and development. Finally, EPA officials stated that some of the
recommendations the agency did not apply to the draft analysis, such as
one calling for a summary table describing key analytical information to
enhance transparency, will be applied to the regulatory impact analysis
supporting the final rule. To the extent that EPA continues to make
progress addressing the Academies' recommendations, decision makers and
the public will be able to better evaluate the basis for EPA's air
regulations.

We provided a draft of this report to EPA for review. EPA provided
technical comments that we incorporated, as appropriate. Officials from
EPA's Office of Air and Radiation noted in their technical comments that
the report provides a fair and balanced representation of the agency's
application of the recommendations to the particulate matter regulatory
impact analysis and cited EPA's progress in meeting the National
Academies' recommendations through other analyses of air programs and
through research and development efforts.

Background

EPA is required by the Clean Air Act to conduct reviews of the National
Ambient Air Quality Standards (NAAQS) for the six criteria pollutants,
including particulate matter, every 5 years. The overarching purpose of
such reviews is to determine whether the current standards are sufficient
to protect public health and welfare at large, with an adequate margin of
safety, given the latest scientific information available at the time of
the review. Major steps in the NAAQS process include the following:

o developing a criteria document that synthesizes new research on health
and environmental effects;

o preparing a staff paper that assesses the policy implications of the
scientific information in the criteria document, which also discusses
possible ranges for air quality standards; and

o determining whether and how EPA should revise the NAAQS.

If EPA decides to revise the NAAQS, the agency proposes the changes in the
Federal Register. As part of the federal rule-making process, EPA is to
comply with Executive Order 12866, which directs federal agencies to
analyze the costs and benefits of proposed and final rules expected to
affect the economy by $100 million or more per year.4 In September 2003,
the Office of Management and Budget (OMB) issued its Circular A-4, which
presents guidance and best practices and states that agencies should
analyze the costs and benefits in accordance with the principles of full
disclosure and transparency. Further, in cases such as the particulate
matter rule, where expected economic impacts exceed $1 billion annually,
Circular A-4 also states that agencies should conduct a comprehensive
assessment of key uncertainties in their analyses of costs and benefits,
which EPA also refers to as regulatory impact analyses.5 EPA's January
2006 regulatory impact analysis presents estimates of the costs and
benefits for the proposed particulate matter rule.

The focus of the National Academies' 2002 report was on how EPA estimates
the health benefits of its proposed air regulations. To develop such
estimates, EPA conducts analyses to quantify the expected changes in the
number of deaths and illnesses that are likely to result from proposed
regulations. The regulatory impact analyses also estimate the costs
associated with implementing proposed air regulations, although, under the
Clean Air Act, EPA is not permitted to consider costs in setting
health-based standards for the criteria air pollutants, such as
particulate matter.

Soon after the National Academies issued its report in 2002, EPA staff
identified key recommendations and developed a strategy, in consultation
with OMB, to apply some of the recommendations to benefit analyses for air
pollution regulations under consideration at the time. EPA roughly
approximated the time and resource requirements to respond to the
recommendations, identifying those the agency could address within 2 or 3
years and those that would take longer. According to EPA officials, the
agency focused primarily on the numerous recommendations related to
analyzing uncertainty.

Both the National Academies' report and the OMB guidance emphasize the
need for agencies to account for uncertainties and to maintain
transparency in the course of conducting benefit analyses. Identifying and
accounting for uncertainties in these analyses can help decision makers
evaluate the likelihood that certain regulatory decisions will achieve the
estimated benefits. Transparency is important because it enables the
public and relevant decision makers to see clearly how EPA arrived at its
estimates and conclusions. In prior work on regulatory impact analyses, we
have found shortcomings in EPA's analyses of uncertainty and the
information the agency provides with its estimates of costs and benefits.6

EPA Is in the Process of Addressing Many of the Academies' Recommendations

EPA applied-either wholly or in part-approximately two-thirds of the
Academies' recommendations to its January 2006 regulatory impact analysis
and continues to address the recommendations through ongoing research and
development. The January 2006 regulatory impact analysis demonstrated
progress toward an expanded analysis of uncertainty and consideration of
different assumptions. EPA officials cited time and resource constraints,
as well as the need to mitigate complex technical challenges, as the basis
for not applying other recommendations. According to EPA officials, the
agency did not apply some of the more complex recommendations because it
had not achieved sufficient progress in the research and development
projects under way.

EPA Applied, at Least in Part, about Two-thirds of the Recommendations to
Its Particulate Matter Health Benefit Analysis in the Proposed Rule

The January 2006 regulatory impact analysis on particulate matter
represents a snapshot of an ongoing EPA effort to respond to the National
Academies' recommendations on developing estimates of health benefits for
air pollution regulations. Specifically, the agency applied, at least in
part, approximately two-thirds of the recommendations-8 were applied and
14 were partially applied-by taking steps toward conducting a more
rigorous assessment of uncertainty for proposed air pollution regulations
by, for example, evaluating the different assumptions about the link
between human exposure to particulate matter and health effects and
discussing sources of uncertainty not included in the benefit estimates.
According to EPA officials, the agency focused much of its time and
resources on the recommendations related to uncertainty. In particular,
one overarching recommendation suggests that EPA take steps toward
conducting a formal, comprehensive uncertainty analysis-the systematic
application of mathematical techniques, such as Monte Carlo simulation-and
include the uncertainty analysis in the regulatory impact analysis to
provide a "more realistic depiction of the overall uncertainty" in EPA's
estimates of the benefits.7 A number of the other recommendations
regarding uncertainty are aimed at EPA's developing the information and
methodologies needed to carry out a comprehensive uncertainty analysis.

Overall, the uncertainty recommendations suggest that EPA should determine
(1) which sources of uncertainties have the greatest effect on benefit
estimates and (2) the degree to which the uncertainties affect the
estimates by specifying a range of estimates and the likelihood of
attaining them. In response, EPA devoted significant resources to applying
an alternative technique called expert elicitation in a multiphased pilot
project. The pilot project was designed to systematically obtain expert
advice to begin to better incorporate in its health benefit analysis the
uncertainty underlying the causal link between exposure to particulate
matter and premature death. EPA used the expert elicitation process to
help it more definitively evaluate the uncertainty associated with
estimated reductions in premature death-estimates that composed 85 percent
to 95 percent of EPA's total health benefit estimates for air pollution
regulations in the past 5 years, according to the agency.8 EPA developed a
range of expected reductions in death rates based on expert opinion
systematically gathered in its pilot expert elicitation project and
provided the results of this supplemental analysis in an appendix to the
regulatory impact analysis.9 However, the National Academies had
recommended that EPA merge such supplemental analyses into the main
benefit analysis.

Moreover, the Academies recommended that EPA's main benefit analysis
reflect how the benefit estimates would vary in light of uncertainties. In
addition to the uncertainty underlying the causal link between exposure
and premature death that EPA analyzed, other key uncertainties can
influence the estimates. For example, there is uncertainty about the
effects of the age and health status of people exposed to particulate
matter, the varying composition of particulate matter, and the
measurements of actual exposure to particulate matter. EPA's health
benefit analysis, however, does not account for these key uncertainties by
specifying a range of estimates and the likelihood of attaining them,
similar to estimates derived from the expert elicitation addressing causal
uncertainty. For these reasons, EPA's responses reflect a partial
application of the Academies' recommendation.

In addition, the Academies recommended that EPA both continue to conduct
sensitivity analyses on sources of uncertainty and expand these analyses.
In the particulate matter regulatory impact analysis, EPA included a new
sensitivity analysis regarding assumptions about thresholds, or levels
below which those exposed to particulate matter are not at risk of
experiencing harmful effects. EPA has assumed no threshold level
exists-that is, any exposure poses potential health risks.10 Some experts
have suggested that different thresholds may exist and the National
Academies recommended that EPA determine how changing its assumption-that
no threshold exists-would influence the estimates. The sensitivity
analysis EPA provided in the regulatory impact analysis examined how its
estimates of expected health benefits would change assuming varying
thresholds.

Another recommendation that EPA is researching and partially applied to
the draft regulatory impact analysis concerns alternative assumptions
about cessation lags-the time between reductions in exposure to
particulate matter and the health response. The National Academies made
several recommendations on this topic, including one that EPA incorporate
alternative assumptions about lags into a formal uncertainty analysis to
estimate benefits that account for the likelihood of different lag
durations. In response, EPA has sought advice from its Advisory Council on
Clean Air Compliance Analysis on how to address this recommendation and
has conducted a series of sensitivity analyses related to cessation lags.
EPA is also funding research to explore ways to address lag effects in its
uncertainty analysis. According to an EPA official, specifying the
probability of different lag effects is computationally complex, and the
agency is working to resolve this challenge.

In response to another recommendation by the National Academies, EPA
identified some of the sources of uncertainty that are not reflected in
its benefit estimates. For example, EPA's regulatory impact analysis
disclosed that its benefit estimates do not reflect the uncertainty
associated with future year projections of particulate matter emissions.
EPA presented a qualitative description about emissions uncertainty,
elaborating on technical reasons-such as the limited information about the
effectiveness of particulate matter control programs-why the analysis
likely underestimates future emissions levels. EPA also applied the
Academies' recommendation on the presentation of uncertainty, which
encouraged the agency to present the results of its health benefit
analyses in ways that convey the estimated benefits more realistically by,
for example, placing less emphasis on single estimates and rounding the
numbers. EPA's regulatory impact analysis presented ranges for some of the
benefit estimates. Also, EPA sought to convey the overall uncertainty of
its benefit estimates in a qualitative manner by clearly stating that
decision makers and the public should not place significant weight on the
quantified benefit estimates in the regulatory impact analysis because of
data limitations and uncertainties.

Another example of EPA's response to the National Academies'
recommendations involves exploring the various regulatory choices
available to decision makers. The Academies recommended that EPA estimate
the health benefits representing the full range of regulatory choices
available to decision makers. In the particulate matter analysis, EPA
presented health benefits expected under several regulatory options
targeting fine particulate matter. Citing a lack of data and tools needed
to conduct an accurate analysis, EPA did not estimate the benefits
expected under the proposed regulatory options for coarse particulate
matter but, consistent with the National Academies' recommendation,
presented its rationale for not doing so. Overall, we considered this a
partial application of the recommendation. (See app. II for more detail on
the recommendations that EPA has applied or partially applied to the draft
particulate matter regulatory impact analysis.)

EPA Plans to Address Some of the Remaining Recommendations in the Final
Rule and Has Research and Development Under Way to Address Others

EPA did not apply the remaining 12 recommendations to the analysis for
various reasons. While EPA applied some recommendations-either wholly or
in part-that require additional studies, methodologies, or data to its
particulate matter analysis, the agency had not made sufficient progress
in addressing others and therefore did not apply them to the analysis. EPA
officials viewed most of these recommendations as relevant to its health
benefit analyses and, citing the need for additional research and
development, emphasized the agency's commitment to continue to respond to
the recommendations. According to a senior EPA official, insufficient
resources impeded the agency's progress in applying the recommendations.
This official cited limited availability of skilled staff, time, and other
resources to conduct the required analyses and research and development.
According to EPA, some of the more complex, long-term recommendations
include the following: relying less on simplifying assumptions, such as
the assumption that the various components of particulate matter have
equal toxicity;11 conducting a formal assessment of the uncertainty of
particulate matter emissions; and assessing the expected reduction of any
harmful effects other than air pollution or human health problems.

For example, EPA is in the process of responding to a recommendation
involving the relative toxicity of components of particulate matter, an
emerging area of research that has the potential to influence EPA's
regulatory decisions in the future.12 Specifically, the agency could,
hypothetically, refine national air quality standards to address the
potentially varying health consequences associated with different
components of particulate matter. The National Academies recommended that
EPA strengthen its benefit analyses by evaluating a range of alternative
assumptions regarding relative toxicity and incorporate these assumptions
into sensitivity or uncertainty analyses as more data become available.13
EPA did not believe the state of scientific knowledge on relative toxicity
was sufficiently developed at the time it prepared the draft regulatory

impact analysis to include this kind of analysis.14 However, EPA is
sponsoring research on this issue. For example, EPA is supporting
long-term research on the relative toxicity of particulate matter
components being conducted by EPA's intramural research program, its five
Particulate Matter Research Centers, and the Health Effects Institute, an
organization funded in part by EPA. In addition, an EPA contractor has
begun to investigate methods for conducting a formal analysis that would
consider sources of uncertainty, including relative toxicity and lag
effects. To date, the contractor has created a model to assess whether and
how much these sources of uncertainty may affect benefit estimates in one
urban area.

The National Academies also recommended that EPA incorporate an assessment
of uncertainty into the early stages of its benefit analyses by
characterizing the uncertainty of its emissions estimates on which the
agency is going to base its benefit estimates.15 While the agency is
investigating ways to assess or characterize this uncertainty, EPA did not
conduct a formal uncertainty analysis for particulate matter emissions for
the draft regulatory impact analysis because of data limitations. These
limitations stem largely from the source of emissions data, the National
Emissions Inventory,16 an amalgamation of data from a variety of entities,
including state and local air agencies, tribes, and industry. According to
EPA, these entities use different methods to collect data, which have
different implications for how to characterize the uncertainty.
Furthermore, the uncertainty associated with emissions varies by the
source of emissions. For example, the analytical methods for evaluating
the uncertainty of estimates of emissions from utilities would differ from
those for car and truck emissions because the nature of these emissions
and the data collection methods differ. In sum, to apply this
recommendation, EPA must determine how to characterize the uncertainty of
the estimates for each source of emissions before aggregating the
uncertainty to a national level and then factoring that aggregation into
its benefit estimates. According to EPA officials, the agency needs much
more time to resolve the complex technical challenges of such an analysis.
EPA officials also noted that the final particulate matter analysis will
demonstrate steps toward this recommendation by presenting emissions data
according to the level emitted by the different kinds of sources, such as
utilities, cars, and trucks.

Another recommendation that EPA is researching but did not apply to the
draft regulatory impact analysis concerns whether the proposed revisions
to the particulate matter standards would have important indirect impacts
on human health and the environment. According to an EPA official, the
agency could not rule out the possibility that the revisions could have
indirect impacts on the environment, such as whether reductions to
particulate matter emissions would reduce the amount of particulate matter
deposited in water bodies, thereby decreasing water pollution. EPA has
considered indirect impacts of air pollution regulations on sensitive
water bodies in the past and plans to include a similar analysis in the
final particulate matter rule. An agency official further noted that
ongoing research about environmental impacts could reveal additional
indirect impacts for future analyses.

Other recommendations that EPA did not apply to its benefit estimates in
the regulatory impact analysis concern issues such as transparency and
external review of EPA's benefit estimation process. For example, the
National Academies recommended that EPA clearly summarize the key elements
of the benefit analysis in an executive summary that includes a table that
lists and briefly describes the regulatory options for which EPA estimated
the benefits, the assumptions that had a substantial impact on the benefit
estimates, and the health benefits evaluated. EPA did not, however,
present a summary table as called for by the recommendation or summarize
the benefits in the executive summary. As EPA stated in the particulate
matter analysis, the agency decided not to present the benefit estimates
in the executive summary because they were too uncertain. Specifically,
officials said the agency was not able to resolve some significant data
limitations before issuing the draft regulatory impact analysis in January
2006-a deadline driven by the need to meet the court-ordered issue date
for the final rule in September 2006. According to EPA officials, EPA has
resolved some of these data challenges by, for example, obtaining more
robust data on anticipated strategies for reducing emissions, which will
affect the estimates of benefits. The officials also said that EPA intends
to include in the executive summary of the regulatory impact analysis
supporting the final rule a summary table that describes key analytical
information. EPA officials also acknowledged other presentation
shortcomings, including references to key analytical elements that were
insufficiently specific, that officials attributed to tight time frames
and the demands of working on other regulatory analyses concurrently. They
said they plan to address these shortcomings in the final regulatory
impact analysis.

Regarding external review, the National Academies recommended that EPA
establish an independent review panel, supported by permanent technical
staff, to bolster EPA's quality control measures for its regulatory impact
analyses, such as the one for particulate matter. The National Academies
noted that peer review of EPA's regulatory impact analyses would be
advantageous when the agency designs and conducts its economic analysis.17
EPA has not directly addressed this recommendation. According to the
Director of the Office of Policy Analysis and Review in EPA's Office of
Air and Radiation, establishing and supporting independent committees is
costly, making it important for EPA to take advantage of existing panels
rather than set up new ones. Further, an official in the Office of Air and
Radiation who oversees the development of regulatory impact analyses said
that the cost of reviewing all regulatory impact analyses would be
substantial. In this regard, EPA officials identified peer reviews the
agency received from its existing independent committees, such as the
Clean Air Scientific Advisory Committee and the Advisory Council on Clean
Air Compliance.18 For example, to respond to the Academies'
recommendations about lag effects, EPA sought independent advice on the
assumptions it was developing regarding the time between reduced exposure
to particulate matter and reductions in incidences of health effects.19
Finally, EPA officials noted that although the agency does not have each
regulatory impact analysis peer reviewed, EPA typically does have the
methodologies that will be applied to regulatory impact analyses peer
reviewed. (See app. III for more detail on these recommendations and
others that EPA did not apply to the draft particulate matter regulatory
impact analysis.)

Concluding Observations

While EPA has taken a number of steps to respond to the Academies'
recommendations on estimating health benefits, continued commitment and
dedication of resources will be needed if EPA is to fully implement the
improvements endorsed by the National Academies. In particular, the agency
will need to ensure that it allocates resources to needed research on
emerging issues, such as the relative toxicity of particulate matter
components; assessing which sources of uncertainty have the greatest
influence on benefit estimates; and estimating other benefits, such as
environmental improvements. In addition, it is important for EPA to
continue to improve its uncertainty analysis in accordance with the
Academies' recommendations. The agency's draft regulatory impact analysis
illustrates that estimates of health benefits can be highly uncertain. In
fact, EPA officials viewed these estimates as so uncertain that they chose
to not present them in the executive summary of the regulatory impact
analysis. While EPA officials said they expect to reduce the uncertainties
associated with the health benefit estimates in the final particulate
matter analysis, robust uncertainty analysis will nonetheless be important
for decision makers and the public to understand the likelihood of
attaining the estimated health benefits. According to EPA officials, the
final regulatory impact analysis on particulate matter will reflect
further responsiveness to the Academies' recommendations by, for example,
providing additional sensitivity analysis and improving the transparency
of the regulatory impact analysis by highlighting key data and assumptions
in the executive summary. Moreover, these officials emphasized the
agency's commitment to further enhancing the transparency of the analysis
by presenting clear and accurate references to the supporting technical
documents, which detail the analytical assumptions and describe the data
supporting the estimates. To the extent EPA continues to make progress
addressing the Academies' recommendations, decision makers and the public
will be able to better evaluate the basis for EPA's air regulations.

Agency Comments

We provided a draft of this report to EPA for review. EPA provided
technical comments that we incorporated, as appropriate. Officials from
the Office of Policy Analysis and Review within EPA's Office of Air and
Radiation noted in their technical comments that the report provides a
fair and balanced representation of EPA's efforts to apply the National
Academies' recommendations to the draft particulate matter regulatory
impact analysis. However, these officials also cited progress made in
applying the National Academies' recommendations through analyses of other
air programs and through research and development efforts. We note that
this report does identify, as appropriate, EPA's research and development
efforts for recommendations EPA did not apply to the draft particulate
matter analysis, its plans to apply some additional recommendations to the
final particulate matter regulatory impact analysis, and the agency's
responses to recommendations in prior rule-making analyses of air
programs.

As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from
the report date. At that time, we will send copies to the EPA
Administrator and other interested parties. We will also make copies
available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me at (202) 512-3841 or [email protected] . Contact points for our
Offices of Congressional Relations and Public Affairs may be found on the
last page of this report. Key contributors to this report are listed in
appendix IV.

John B. Stephenson Director, Natural Resources and Environment

Appendix I  Scope and Methodology

We were asked to determine whether and how the Environmental Protection
Agency (EPA) applied the National Academies (Academies) recommendations in
its estimates of the health benefits expected from the January 2006
proposed revisions to the particulate matter national ambient air quality
standards. In response to this objective, we assessed EPA's response to
the Academies' recommendations and present an overview of the agency's
completed, ongoing, and planned actions addressing the recommendations. To
develop this overview, we reviewed EPA's particulate matter regulatory
impact analysis, EPA's economic analysis guidelines, and Office of
Management and Budget (OMB) guidance on regulatory impact analysis. We
also analyzed documentation addressing current and future agency efforts
to address the recommendations, such as project planning memorandums and
technical support documents discussing the application of economic
techniques. In addition, we met with senior officials from EPA's Office of
Air and Radiation, which was responsible for developing the proposed rule
and analyzing its economic effects, and with officials from EPA's Office
of Policy, Economics, and Innovation to discuss the agency's responses to
the recommendations. We interviewed several experts outside EPA, including
(1) the Chair and other members of the National Academies' Committee on
Estimating the Health-Risk-Reduction Benefits of Proposed Air Pollution
Regulations, to clarify the basis for their recommendations; and (2)
economists at Resources for the Future, to discuss the technical issues
underlying the recommendations on uncertainty analysis.

While the 2002 National Academies' report is generally applicable to EPA
air pollution regulations, our review focused on the application of the
recommendations to the proposed revisions to the particulate matter
standards, as requested. Our work focused on broadly characterizing EPA's
progress toward applying the recommendations; we did not evaluate the
effectiveness or quality of the scientific and technical actions the
agency has taken to apply them. To assess whether and how EPA has made
progress in responding to the recommendations, we developed the following
recommendation classification continuum: applied, partially applied, and
not applied. The applied and partially applied categories refer to
completed and initiated actions in EPA's health benefit analysis of
particulate matter that corresponds to components of the National
Academies' recommendations. The not applied category includes
recommendations that EPA did not apply when conducting the analysis for
the January 2006 particulate matter regulatory impact analysis and
identifies those for which ongoing research and development efforts were
not far enough along to apply to the particulate matter analysis. We
performed our work from January 2006 to July 2006 in accordance with
generally accepted government auditing standards.

Appendix II  Recommendations Applied or Partially Applied to the Draft Particulate
Matter Regulatory Impact Analysis 

Table 1 provides a summary of the National Academies' recommendations that
EPA has applied or partially applied to its draft regulatory impact
analysis (RIA) for particulate matter (PM).  This table also provides
GAO's assessment of EPA's progress in applying each recommendation, in
terms of steps EPA has taken thus far to address issues highlighted in the
National Academies' report. The final column characterizes EPA's comments
regarding each recommendation, including, as pertinent, contextual
information, potential impediments to application, and intended next
steps.

Table 1: Recommendations Applied or Partially Applied to the Draft
Particulate Matter Regulatory Impact Analysis

                                        

    National Academies'    Status       GAO assessment         EPA's response     
       recommendation                                      
1  To the extent          Partially EPA estimated the      EPA did not estimate   
possible, EPA should   applied   health benefits for    benefits for coarse    
estimate the benefits            the base case and some particulate matter     
for several regulatory           of the proposed        primarily because of a 
options that represent           standards for fine     lack of data and tools 
the full range of                particulate matter,    needed to conduct an   
choices available to             but did not estimate   accurate analysis. In  
the decision maker.              benefits that          addition, EPA said     
The regulatory options           represented the full   that time and          
should include graded            range of choices       resources limited the  
levels of stringency             available to the       number of regulatory   
requirements and the             decision maker. In     options to be modeled. 
time schedule for                addition, EPA did not  EPA said that given    
achieving reductions             estimate the benefits  these constraints, the 
in emissions or                  for coarse particulate agency selected        
exposures. If options            matter. While EPA      regulatory options     
are eliminated at an             provided an            that represent         
earlier stage, the               explanation as to why  reasonable bounds on   
rationale for doing so           it did not estimate    alternatives.          
should be provided.              benefits for the       
                                    regulatory options for 
                                    coarse particulate     
                                    matter, it did not     
                                    present its rationale  
                                    for not estimating the 
                                    benefits of the full   
                                    range of options for   
                                    fine particulate       
                                    matter.                
2  EPA should modify the  Partially EPA has modified air   EPA characterized its  
air quality models     applied   quality models but has response to this       
used in translating              not yet demonstrated   recommendation as a    
predicted emissions              that these changes     work in progress. EPA  
into predicted levels            reduce resources. EPA  said that the agency   
of ambient air quality           reports faster         has achieved greater   
to reduce resources              modeling runs once it  efficiency in the      
required for air                 has designated a model models, but that the   
quality modeling. This           but has also needed to overall time and       
change is necessary if           conduct more complex   resources devoted to   
EPA is to evaluate               model runs, which      modeling have not      
multiple regulatory              require time.          decreased because of   
alternatives and to                                     the increased          
evaluate each                                           complexity and         
alternative at                                          increased volume of    
reasonable time                                         air quality model      
intervals, such as                                      runs. Citing emissions 
every 5 years.                                          data limitations as an 
Evaluation of the                                       ongoing challenge, EPA 
ambient air quality                                     reported notable       
associated with more                                    improvements to the    
emissions scenarios is                                  emissions inventory    
also essential if the                                   that contributed to    
uncertainty inherent                                    efficiency gains in    
in emissions estimates                                  air quality modeling   
is to be carried                                        runs. Overall, EPA is  
through to the                                          working to balance     
estimation of avoided                                   efforts to streamline  
cases of mortality and                                  models with the        
morbidity.                                              demands for more       
                                                           sophisticated          
                                                           analyses. In addition, 
                                                           EPA cited additional   
                                                           analytical             
                                                           requirements, such as  
                                                           new peer review        
                                                           policies and OMB's     
                                                           expanded data quality  
                                                           guidelines, that add   
                                                           to the complexity of,  
                                                           and therefore time and 
                                                           resources needed for,  
                                                           its models.            
3  EPA should incorporate Partially EPA incorporated       EPA did not            
estimates of future    applied   future trends in       incorporate future     
trends in background             background mortality   morbidity trends       
mortality and                    rates but did not do   because of time        
morbidity for the                so for future          constraints. As time   
major health outcomes,           morbidity trends.      permits, EPA plans to  
such as those that                                      incorporate such       
make up two-thirds of                                   projections in the     
total deaths or lost                                    final analysis.        
life years that are                                     
under consideration.                                    
4  As in all other stages Partially The draft PM RIA       EPA intends to include 
of the benefits        applied   presented EPA's        references in the      
analysis, EPA should             justifications and     final RIA, detailing   
justify and clearly              included clear         the assumptions and    
describe the                     descriptions of a      methods underlying its 
assumptions and                  number of assumptions  health benefits        
methods used to assess           and methods used in    analyses. EPA stated   
exposure, choose                 the benefit analysis.  that rather than       
health outcomes, and             As discussed           restating information  
select studies and               elsewhere, EPA's       from prior RIAs, it    
concentration-response           assessment of key      provides references to 
functions, paying                sources of uncertainty these discussions in   
careful attention to             generally relied on    order to manage the    
assessing and                    qualitative discussion length of the RIA and  
communicating key                and sensitivity        save time.             
sources of                       analysis (see          Furthermore, EPA       
uncertainty.                     recommendation 14).    stated that it is not  
                                    However, the extent to necessary to document  
                                    which EPA provided     assumptions for tools  
                                    justifications and     that have already been 
                                    clear descriptions     peer reviewed, such as 
                                    varied. For example,   BenMAP, the model used 
                                    EPA included detailed  to estimate health     
                                    information about why  benefits.              
                                    it chose studies       
                                    related to the         
                                    concentration-response 
                                    function, but the      
                                    agency did not present 
                                    its justification for  
                                    an assumption used to  
                                    assess exposure.       
5  Because pollution      Partially EPA assessed local     EPA focused on         
modeling rarely        applied   concentrations in      assessing the local    
addresses the                    terms of how different concentrations from    
smaller-scale issue of           source categories,     specific source        
how local                        such as stationary and activities but assumed 
concentrations from              mobile sources of PM,  the same time and      
specific source                  affect total           activity patterns for  
categories interact              exposures. However,    each scenario. EPA     
with human                       EPA has not yet        stated that it does    
time-activity                    assessed how           not intend to conduct  
patterns, EPA should             human-time activity    a detailed analysis of 
examine how different            patterns, such as      micro-environmental    
major source                     lifestyles, affect     issues or human-time   
categories-for                   exposure to PM.        activity patterns.     
example, mobile versus                                  
large stationary                                        
sources-affect total                                    
exposures per unit                                      
emissions.                                              
6  EPA should consider    Applied   EPA considered data    EPA determined, on the 
data from U.S. and               from additional U.S.   basis of advice from   
non-U.S. studies to              studies as part of its the Science Advisory   
extrapolate beyond the           effort to expand the   Board, that it would   
age groups evaluated             age groups in its      only extrapolate data  
and incorporate other            estimates of the       to other ages when it  
relevant outcomes not            health outcomes        found a reasonable     
evaluated in its                 (premature mortality   physiological basis    
current benefit                  and illness). EPA      for doing so. For      
analyses.                        extrapolated beyond    example, EPA used data 
                                    the age groups in some from a study on asthma 
                                    cases, but not all.    in children ages 7 to  
                                                           11 to estimate the     
                                                           reductions in asthma   
                                                           for the entire child   
                                                           age group-ages 6 to    
                                                           18.                    
7  EPA provided little    Applied   EPA referred readers   
information in the               to a prior RIA for     
benefit analyses                 information on the     
reviewed by the                  causal association     
committee on causal              between particulate    
association between              matter and adverse     
particular types of              health outcomes.       
air pollution and                                       
adverse health                                          
outcomes. EPA should                                    
summarize the evidence                                  
for causality to                                        
justify the inclusion                                   
or exclusion of the                                     
health outcomes and to                                  
assess the uncertainty                                  
associated with the                                     
assumption of                                           
causality.                                              
8  EPA should investigate Applied   EPA investigated one   EPA intends to         
and, if necessary,               method-expert          continue its effort to 
develop methods of               elicitation-to         better characterize    
evaluating causal                evaluate how causal    the uncertainty in key 
uncertainty relating             uncertainty affects    health outcomes.       
to key outcomes so               final benefit          
that this uncertainty            estimates. See         
can be represented in            recommendation 14 for  
the final benefit                more details.          
estimates.                                              
9  Although the committee Applied   EPA reviewed its       EPA incorporated new   
believes the use of              selection of the best  studies regarding      
the American Cancer              estimate for long-term estimates of premature 
Society (ACS) study to           effects of air         mortality and          
derive premature                 pollution on mortality justified the          
mortality estimates              and concluded that     concentration-response 
was reasonable, EPA              data from the extended functions derived from 
should thoroughly                analysis of the ACS    these studies. These   
review its selection             study provided the     justifications were    
of the best estimate             best estimates of      included in technical  
for long-term effects            premature mortality.   appendices to the      
of air pollution on              EPA consulted the      draft PM RIA. EPA also 
mortality. Several new           Science Advisory Board consulted with the     
studies have been                to reach its decision  Science Advisory Board 
published since the              to emphasize the ACS   regarding the best     
ACS study, including             data. EPA also         estimates for          
an extended analysis             determined that it was premature mortality.   
of the ACS study, a              not appropriate to     EPA decided not to     
new U.S. cohort study,           derive a weighted mean derive a weighted mean 
and other non-U.S.               estimate from cohort   estimate because the   
studies. EPA should              studies.               use of this estimate   
also consider whether                                   would risk losing the  
the derivation of a                                     variability across     
weighted mean estimate                                  cohort studies.        
from the cohort                                         
studies is appropriate                                  
following review of                                     
the database.                                           
10 To evaluate short-term Applied   EPA used studies that  
effects of air                   integrate distributed  
pollution, EPA should            lag models that        
use                              account for the onset  
concentration-response           of health effects      
functions from studies           occurring more than    
that integrate over              several days after     
several days or weeks            exposure.              
the exposure period                                     
and the time period to                                  
the event (cumulative                                   
or distributed lag                                      
models), rather than                                    
those that restrict                                     
these time periods to                                   
1 or 2 days.                                            
11 Although the           Partially EPA evaluated several  EPA reported some      
assumption of no       applied   different threshold    progress toward        
thresholds in the most           assumptions in a       improving its approach 
recent EPA benefit               sensitivity analysis   to characterize        
analyses was                     but has not yet        uncertainties and to   
appropriate, EPA                 considered these       conduct a formal       
should evaluate                  assumptions in a       uncertainty analysis.  
threshold assumptions            formal uncertainty     (See recommendation 14 
in a consistent and              analysis. The          for more information.) 
transparent framework            sensitivity analysis   
using several                    was transparent-EPA    
alternative                      clearly explained the  
assumptions in the               basis for the          
formal uncertainty               different threshold    
analysis.                        assumptions.           
12 The committee found    Partially EPA sought advice from Consistent with the    
little justification   applied   the Advisory Council   recommendation made by 
for the 5-year time              on Clean Air           the Council, EPA now   
course of exposure and           Compliance (Council)   uses a 20-year lag     
outcome assumed in               and incorporated the   model. The revised lag 
more recent EPA                  advice by adjusting    model involves a       
analyses and                     the agency's           higher percentage of   
recommends that EPA              assumptions regarding  mortality reductions   
more fully account for           lag times. As for the  occurring in the first 
the uncertainty                  uncertainty            year (30 percent) than 
regarding lags in                surrounding lags in    the previous EPA lag   
health effects by                health effects, EPA    model (20 percent).    
incorporating a range            conducted a series of  EPA told us that it    
of assumptions and               sensitivity analyses   will not include       
probabilities about              on the temporal        probability-based      
the temporal                     relationship but did   distributions in the   
relationship.                    not assess underlying  final RIA because of   
                                    probabilities.         computational          
                                                           complexities.          
13 EPA is encouraged to   Partially EPA included some      EPA intends to present 
estimate and report    applied   health benefit         in the final RIA a     
benefits by age, sex,            estimates for specific detailed summary table 
and other demographic            age groups in an       with the age breakdown 
factors. The committee           appendix of the draft  of its sample          
recognizes, however,             PM RIA. However, the   population. EPA will   
that evaluating the              agency did not report  not, however, estimate 
differences for                  the primary benefit    and report benefits by 
various subgroups adds           estimates according to other demographic      
complexity and                   age groups or other    factors because of     
uncertainty to the               demographic factors.   time and resource      
analysis and that                                       constraints. EPA also  
caution must be                                         cited data             
exercised in the                                        limitations, noting    
interpretation of such                                  that it lacks data     
results.                                                accounting for local   
                                                           variations in          
                                                           demographic factors.   
                                                           The agency intends to  
                                                           overcome this          
                                                           limitation using       
                                                           tract-level data,      
                                                           which captures details 
                                                           about local level      
                                                           conditions, from the   
                                                           U.S. Census data in    
                                                           future analyses.       
14 EPA should begin to    Partially EPA has taken some     EPA reported some      
move the assessment of applied   steps toward the       progress toward        
uncertainties from its           formal uncertainty     improving its approach 
ancillary analyses to            analysis called for by to characterize        
its primary analyses.            this recommendation,   uncertainties with     
This shift will                  but the primary        particular emphasis on 
require the                      analysis in the draft  one source of          
specification of a               PM RIA generally       uncertainty-premature  
probability                      addresses uncertainty  death linked to PM     
distribution for each            in a qualitative       exposure.              
uncertainty source               manner. Overall, the   Specifically, EPA used 
that is added to the             numerical benefit      expert elicitation to  
primary analysis and,            estimates do not       begin to specify a     
as necessary, the                capture the key        distribution for the   
specification of joint           sources of             uncertainty in         
distributions for the            uncertainty. The       concentration levels   
uncertainty sources              agency generally       of PM linked to        
that are not                     relied on sensitivity  premature death. The   
independent of each              analysis to assess     agency cited technical 
other. Expert                    some uncertain factors challenges, such as    
judgment, as well as             one at a time rather   lack of data or        
data, will be required           than using more        reliable methods, and  
to specify these                 comprehensive          resource constraints,  
distributions.                   techniques for         including limitations  
Although the effect on           assessing probability  to its progress to     
the mean of the                  distributions of       fully characterize     
resulting probability            multiple variables. In uncertainty. Moreover, 
distribution might               addition, EPA used     EPA stated that its    
increase, decrease, or           expert elicitation to  focus on the expert    
remain the same, the             help assess            elicitation technique  
effect on the spread             uncertainty relating   limited the time and   
of the distribution              to the concentrations  resources necessary to 
will be a predictable            of PM linked to        address other aspects  
widening and,                    premature death and    of uncertainty in      
therefore, a more                the dollar value of    premature death as     
realistic depiction of           risk reductions        well as illnesses      
the overall                      associated with        linked to PM exposure. 
uncertainty in the               reductions in PM.                             
analysis.                        However, these results EPA applied another    
                                    were presented in an   formal method for      
                                    appendix and not in    assessing              
                                    the primary analysis   uncertainty-Monte      
                                    in the draft RIA. In   Carlo analysis-to a    
                                    addition, the health   previous regulatory    
                                    benefit analysis did   impact analysis but,   
                                    not present a          according to EPA, did  
                                    quantitative           not have time to       
                                    assessment of how the  incorporate this work  
                                    benefit estimates      in the PM RIA. Time    
                                    would vary in light of permitting, the agency 
                                    other key              plans to present the   
                                    uncertainties.         results of a Monte     
                                                           Carlo analysis in the  
                                                           final PM RIA.          
15 Because the            Applied   EPA distinguished      
incorporation of                 between data-derived   
expert judgment when             estimates and those    
data are unavailable             from expert judgment.  
will influence the               For example, in the    
estimates of health              appendix, EPA clearly  
benefits as well as              distinguished between  
the uncertainty                  data derived from      
analyses, the                    experts and the data   
committee also                   based on an empirical  
recommends that EPA              study. In addition,    
clearly distinguish              EPA discusses the      
between data-derived             basis for assumptions, 
estimates of some                which require          
components, such as              analytical judgment,   
the                              either directly in the 
concentration-response           draft PM RIA or by     
function, and expert             reference to           
opinions about other             supporting documents.  
components that are                                     
lacking in scientific                                   
data, such as the                                       
degree of compliance                                    
with a particular                                       
regulation 30 years                                     
into the future. In                                     
this way, policymakers                                  
will better understand                                  
how existing data and                                   
expert judgment                                         
combine to produce                                      
estimates and where                                     
new data would be most                                  
valuable.                                               
16 As EPA begins the      Partially EPA expanded the       EPA has begun to       
transition to          applied   sensitivity analysis   explore how to expand  
incorporate additional           by, for example,       sensitivity analysis   
sources of uncertainty           considering how        and consider sources   
into its primary                 benefit estimates      of uncertainty         
health benefits                  change according to    jointly. EPA noted     
analyses, it should              different threshold    that in recent air     
continue the                     assumptions, such as   pollution analyses,    
sensitivity analyses             cut                    the agency considered  
it has traditionally             points-concentrations  sources of uncertainty 
conducted. These                 of particulate matter  jointly for the last   
analyses should be               below which there      stage of the benefit   
expanded, however, to            would be no benefit to estimation             
consider sources of              further reductions.    process-valuing the    
uncertainty jointly              This sensitivity       benefits in dollar     
rather than singly.              analysis examined how  terms-by doing a       
                                    different assumed cut  probabilistic          
                                    points would change    assessment of the      
                                    the estimates of       value of a statistical 
                                    avoided cases of       life.                  
                                    death. EPA did not,                           
                                    however, conduct a     In addition, EPA's     
                                    sensitivity analysis   policy and economics   
                                    to consider the        division has begun a   
                                    sources of uncertainty long-term project to   
                                    jointly rather than    expand uncertainty     
                                    singly.                analysis. EPA reports  
                                                           that one technique     
                                                           that considered how    
                                                           much different sources 
                                                           of uncertainty affect  
                                                           the estimates is not   
                                                           ready to be applied to 
                                                           a rule-making          
                                                           analysis.              
17 In presenting the      Partially EPA did not present a  EPA attributed         
probability            applied   probability            incomplete or lacking  
distribution for each            distribution for the   references in the      
health benefit                   primary benefit        draft PM RIA to time   
produced by a primary            analysis but included  constraints, noting    
analysis, EPA should             some discussion of     that it would present  
emphasize even more              sources of uncertainty clear references in    
than it has in the               not incorporated into  the final version. EPA 
past the sources of              the benefit estimates. also clarified that it 
uncertainty that                 EPA also referred the  relies on references   
remain unaccounted for           reader to a previous   to other documents,    
in the primary                   analysis involving     rather than repeating  
analysis. These                  particulate matter for information that has   
uncertainties should             detailed discussions   not changed, in order  
continue to be                   on sources of          to keep the            
described as                     uncertainty. The       presentation to a      
completely and                   citations are          manageable size.       
realistically as                 incomplete, however,                          
possible.                        leaving readers to     (See also              
                                    search within the      recommendation 14 for  
                                    voluminous document    EPA's basis for not    
                                    for the relevant       including probability  
                                    information.           distributions.)        
18 EPA should consider    Applied   According to EPA, the  EPA determined that    
providing a                      agency considered      this information would 
preliminary analysis             providing the          not provide meaningful 
that estimates, in               preliminary analysis   information to the     
current populations,             described in this      draft PM benefit       
the health benefits              recommendation.        analysis. Aside from   
resulting from                                          its questions about    
hypothetical changes                                    the technical          
to current levels of                                    feasibility of         
emissions. These                                        responding to this     
preliminary analyses                                    recommendation, EPA    
would help EPA develop                                  expressed doubts that  
an idea of the lower                                    doing this analysis    
bound on the                                            for current            
uncertainty of future                                   populations would      
consequences and would                                  establish a lower      
have fewer                                              bound on the           
uncertainties than                                      uncertainty of future  
analyses of the                                         consequences. One EPA  
impacts of proposed                                     official concluded     
regulatory actions on                                   that the analysis      
future exposures and                                    could introduce more   
health outcomes.                                        uncertainty into its   
                                                           benefit estimates      
                                                           because benefit        
                                                           projections will       
                                                           depend on controls     
                                                           implemented in the     
                                                           future. Finally, EPA   
                                                           clarified that, as     
                                                           provided by Circular   
                                                           A-4, it estimates      
                                                           benefits for a future  
                                                           year when the          
                                                           regulatory revisions   
                                                           become effective, not  
                                                           the current year,      
                                                           which would not        
                                                           account for changes in 
                                                           variables other than   
                                                           emissions.             
19 EPA should continue to Applied   EPA followed some of   
strive to present the            the National           
results of its health            Academies' suggestions 
benefit analyses in              to present the data in 
ways that avoid                  a way that avoids      
conveying an                     conveying an           
unwarranted degree of            unwarranted level of   
certainty. Such ways             certainty. EPA rounded 
include rounding to              the estimates to fewer 
fewer significant                significant digits.    
digits, increasing the           EPA increased the      
use of graphs,                   emphasis on ranges by  
presenting projected             presenting some data   
baselines along with             in ranges of benefit   
projected health                 and cost estimates.    
benefits, and placing            EPA summarized the     
less emphasis on                 primary benefit        
single numbers (for              estimates in a table.  
example, the mean of                                    
the probability                                         
distribution for a                                      
health benefit) and                                     
greater emphasis on                                     
ranges (for example,                                    
the range between 5th                                   
and 95th percentiles                                    
of the distribution).                                   
20 EPA should perform     Partially The draft PM RIA       Overall, EPA cited     
similar detailed       applied   demonstrated some      technical challenges,  
analyses of                      steps toward analysis  including data         
uncertainty in the               of cost uncertainty    limitations, as the    
valuation of health              but did not present an primary reason for not 
benefits and in the              uncertainty analysis   applying a formal      
regulatory cost                  of the benefit         uncertainty analysis   
analyses that the                valuation-the stage    to the cost estimates  
committee recommends             when it assigns a      in the PM RIA.         
for the health benefit           dollar value to the    According to EPA, the  
analyses.                        benefit estimates. In  number and breadth of  
                                    terms of cost          PM sources considered  
                                    uncertainty, EPA       under the National     
                                    presented a            Ambient Air Quality    
                                    qualitative discussion Standards rule poses a 
                                    of the uncertainties   challenge to           
                                    about costs and the    development of a       
                                    expected impact on the characterization of    
                                    cost estimates but did all costs and their    
                                    not perform a formal   uncertainties.         
                                    uncertainty analysis   Moreover, EPA noted    
                                    for the costs.         that the collection    
                                                           process for cost       
                                                           information is not     
                                                           systematic, and there  
                                                           is a limited amount of 
                                                           information about the  
                                                           cost of implementing   
                                                           some types of          
                                                           controls. Even when    
                                                           costs are known, it    
                                                           may be difficult to    
                                                           specify the underlying 
                                                           probability            
                                                           distributions.         
                                                                                  
                                                           EPA told us that it    
                                                           will incorporate more  
                                                           refined information,   
                                                           such as emissions      
                                                           control analysis,      
                                                           affecting cost         
                                                           estimates in the final 
                                                           RIA. EPA expects the   
                                                           final RIA to include   
                                                           more reasonable        
                                                           cutoffs such that cost 
                                                           estimates will not be  
                                                           based on ineffective   
                                                           controls and excessive 
                                                           costs (i.e., controls  
                                                           that are not likely to 
                                                           be used).              
                                                                                  
                                                           In terms of health     
                                                           valuation uncertainty, 
                                                           EPA has made progress  
                                                           toward a formal        
                                                           uncertainty analysis   
                                                           by doing a             
                                                           probabilistic          
                                                           assessment of the      
                                                           value of a statistical 
                                                           life. EPA conducted    
                                                           this analysis in prior 
                                                           RIAs but did not have  
                                                           enough time to         
                                                           complete model runs in 
                                                           time for the draft     
                                                           RIA. EPA plans to      
                                                           incorporate an         
                                                           uncertainty analysis   
                                                           related to the         
                                                           mortality valuation in 
                                                           the final PM RIA.      
                                                                                  
                                                           EPA has focused its    
                                                           expansion of           
                                                           uncertainty analysis   
                                                           more on benefits than  
                                                           on costs, in part      
                                                           because the benefit    
                                                           estimates for air      
                                                           rules tend to be much  
                                                           larger. EPA identified 
                                                           efforts to analyze     
                                                           cost uncertainty       
                                                           factors, including     
                                                           sensitivity analyses   
                                                           in previous RIAs and   
                                                           plans for              
                                                           accountability         
                                                           analysis comparing     
                                                           estimated costs to     
                                                           actual costs. Finally, 
                                                           EPA pointed out that   
                                                           the work it conducts   
                                                           to understand the      
                                                           benefits also carries  
                                                           over to the cost       
                                                           estimates.             
21 EPA should provide     Partially EPA's detailed         Although EPA used the  
health benefit         applied   breakdown of the       quality-adjusted life  
estimates in ways that           benefit estimates and  years approach in a    
will support multiple            the reference to its   prior RIA to aggregate 
kinds of analysis,               benefit model allows   benefits, EPA did not  
including various                others to apply        use this approach in   
approaches to                    various approaches to  the draft PM RIA       
mortality valuation              mortality valuation,   because of time        
and aggregation of               such as alternative    constraints. EPA told  
benefits using                   estimates of the value us that it plans to    
quality-adjusted life            of a statistical life. aggregate the benefits 
years.                           EPA did not, however,  using quality-adjusted 
                                    aggregate benefits     life years in the      
                                    using quality-adjusted final PM RIA.          
                                    life years.            
22 Each analysis should   Partially EPA included some      EPA intends to present 
provide results        applied   health benefit         in the final RIA a     
according to                     estimates for specific detailed summary table 
demographic or other             age groups in an       with the age breakdown 
subgroups when the               appendix, but the      of its sample          
expected changes in              agency did not present population. The agency 
pollution and, thus,             the primary benefit    noted that it will not 
the health benefits              estimates according to present the benefits   
are not distributed              age groups or other    according to other     
uniformly across the             demographic factors.   demographic factors,   
population. This                 (See also              such as race and       
information would aid            recommendation 13.)    income, because of     
decision makers in                                      political              
situations in which                                     sensitivities and data 
equity issues might be                                  limitations. EPA       
involved.                                               clarified that it      
                                                           lacks the data on      
                                                           local conditions but   
                                                           that it intends to     
                                                           overcome this          
                                                           limitation using       
                                                           tract-level data-which 
                                                           captures details about 
                                                           local conditions-from  
                                                           the U.S. Census data   
                                                           in future analyses.    

Source: GAO analysis of National Academies and EPA information.

Appendix III  Recommendations Not Applied to the Draft Particulate Matter Regulatory
Impact Analysis 

Table 2 provides a summary of the National Academies' recommendations that
EPA has not applied to its draft regulatory impact analysis (RIA) for
particulate matter (PM). This table provides GAO's assessment of EPA's
progress to date regarding recommendations that required additional
research and development, were deemed as not relevant to the PM National
Ambient Air Quality Standards (NAAQS) by the agency, or were not included
in the draft PM RIA due to time and resource constraints. The final column
characterizes EPA's comments regarding each recommendation, including
contextual information, potential impediments to application,
justification for not addressing the recommendation, and intended next
steps, if applicable.

Table 2: Recommendations Not Applied to the Draft Particulate Matter
Regulatory Impact Analysis

                                        

        National          Status       GAO assessment       EPA's response       
       Academies'                                      
     recommendation                                    
  1  The uncertainty Not              EPA has not      EPA stated that the       
     in emissions    applied-research quantified the   application of this       
     estimates       and development  uncertainty      recommendation-conducting 
     should be       under way        related to       a formal analysis of      
     quantified and                   emissions        emissions                 
     carried through                  because of       uncertainty-requires      
     the health                       limited data and long-term research and    
     benefit                          computational    development. EPA reports  
     analysis to the                  complexities and that it discussed the     
     calculation of                   has therefore    possibility of conducting 
     avoided cases                    not yet carried  a quantitative            
     of mortality                     such uncertainty uncertainty analysis for  
     and morbidity.                   through in the   emissions in its          
                                      health benefit   particulate matter        
                                      analysis.        analysis, but the fixed   
                                                       timeline prevented EPA    
                                                       from doing this work. The 
                                                       primary challenge stems   
                                                       from the nature of the    
                                                       emissions inventory-data  
                                                       are collected from a      
                                                       plethora of entities,     
                                                       complicating the agency's 
                                                       ability to evaluate       
                                                       uncertainty.              
                                                                                 
                                                       EPA told us that,         
                                                       currently, the only way   
                                                       to assess emissions       
                                                       uncertainty is through    
                                                       qualitative means. EPA    
                                                       also stated that its      
                                                       final particulate matter  
                                                       analysis will demonstrate 
                                                       steps toward this         
                                                       recommendation because it 
                                                       will present a            
                                                       sensitivity analysis of   
                                                       the emissions data and    
                                                       will present emissions    
                                                       data according to the     
                                                       level emitted by the      
                                                       different kinds of        
                                                       sources, such as          
                                                       utilities, cars, and      
                                                       trucks.                   
  2  Because a       Not              While EPA did    EPA stated that in past   
     regulation to   applied-research not provide      rules, the agency has     
     improve air     and development  information in   looked at indirect        
     quality may     under way        the draft PM RIA impacts in terms of       
     affect pathways                  to show that it  deposition of nitrogen    
     other than air,                  considered       and sulfates to sensitive 
     EPA should                       indirect impacts water bodies. EPA plans   
     determine                        involving        to incorporate this       
     whether there                    pathways other   analysis in the final PM  
     are likely to                    than air, EPA's  RIA. EPA said it has not  
     be any                           ongoing research yet identified any other  
     important                        on the           indirect impacts. While   
     indirect                         environmental    an EPA official suggested 
     impacts of a                     impacts may      that this recommendation  
     regulation on                    identify         did not seem relevant to  
     human health                     important        the NAAQS analysis in     
     and the                          indirect         terms of human health     
     environment. If                  impacts.         impacts-EPA could not     
     any such                                          determine how human       
     impacts are                                       health would be affected  
     identified, EPA                                   by exposure to PM from    
     should include                                    pathways other than       
     in the analysis                                   air-the agency is         
     a plan to                                         conducting research to    
     assess them                                       identify important        
     more                                              indirect impacts on the   
     completely.                                       environment. EPA          
                                                       characterized             
                                                       environmental impacts as  
                                                       an area of research.      
  3  EPA has         Not              Although EPA     EPA stated that it does   
     typically made  applied-research assumed          not have sufficient       
     the assumption  and development  equivalent       information to            
     of equivalent   under way        toxicity in the  distinguish between       
     potency across                   PM RIA and did   particle components for   
     particle types                   not include      the final rule. EPA is    
     because of                       related          funding long-term         
     insufficient                     sensitivity or   research on relative      
     scientific                       uncertainty      toxicity, including       
     information. As                  analyses, EPA is technical studies to      
     more data                        sponsoring       understand any            
     become                           research         differential toxicities   
     available, EPA                   directed at      as well as economic       
     should                           incorporating    analyses to explore ways  
     strengthen its                   its findings on  to characterize the       
     benefit                          relative         uncertainty in benefit    
     analyses by                      toxicity into    estimates. For example,   
     evaluating a                     future analyses. an EPA contractor         
     range of                                          conducted a sensitivity   
     alternative                                       analysis of relative      
     assumptions                                       toxicity of particle      
     regarding                                         components, including     
     relative                                          carbons, nitrates,        
     particle                                          crustal material, and     
     toxicity and                                      sulfates. To date, the    
     incorporate                                       contractor has created a  
     these                                             model to assess whether   
     assumptions in                                    and how much these        
     sensitivity or                                    sources of uncertainty    
     uncertainty                                       may affect benefit        
     analyses.                                         estimates in one urban    
                                                       area. EPA is also         
                                                       supporting research to    
                                                       explore relative toxicity 
                                                       through its intramural    
                                                       research program, its     
                                                       five Particulate Matter   
                                                       Research Centers, and the 
                                                       Health Effects Institute, 
                                                       an organization funded in 
                                                       part by EPA. EPA's        
                                                       science grant program     
                                                       recently awarded $40      
                                                       million to the five       
                                                       Particulate Matter        
                                                       Research Centers.         
  4  As it           Not              In the draft PM  An EPA contractor is      
     incorporates    applied-research RIA, although    researching techniques    
     additional      and development  EPA presented a  for influence analysis,   
     sources of      under way        qualitative      an expanded form of       
     uncertainty                      discussion about uncertainty analysis that 
     into its                         the importance   would determine which     
     primary health                   of its           uncertainty sources have  
     benefit                          assumptions that the greatest influence on 
     analyses, EPA                    impact           the benefit estimates.    
     should consider                  uncertainty, it  The influence analysis    
     conducting                       did not consider work targeted three       
     analyses to                      which sources of sources of uncertainty,   
     determine which                  uncertainty have including the             
     uncertainty                      the greatest     concentration-response    
     sources have                     influence on the function (which involves  
     the greatest                     mean and spread  threshold and slope); lag 
     influence on                     of the           effects; and relative     
     the mean and                     probability      toxicity. The contractor  
     spread of the                    distribution.    compiled a draft report   
     probability                      EPA is           discussing techniques to  
     distribution.                    sponsoring       conduct this analysis and 
     The need for                     research to      incorporate the           
     these                            incorporate      uncertainty analysis in   
     sensitivity                      influence        benefit estimates.        
     analyses will                    analysis in                                
     be particularly                  future analyses. EPA and the contractor    
     great for                        (See             stated that because these 
     distributions                    recommendation   techniques are in the     
     that are based                   14; EPA did not  exploratory stage, it is  
     on expert                        specify          premature to apply this   
     judgment. The                    probability      work to a specific        
     uncertainty                      distributions    rule-making analysis. For 
     sources that                     for uncertainty  example, EPA cited the    
     have the                         in its primary   need to determine how     
     greatest                         analysis).       much uncertainty is       
     consequences                                      explained by differential 
     for decision                                      toxicity or by different  
     making,                                           thresholds. Finally, EPA  
     including those                                   stated that because it    
     that have the                                     focused its resources on  
     greatest impact                                   the expert elicitation    
     on the spread                                     work (see recommendation  
     of the                                            14), the influence        
     distribution,                                     analysis received fewer   
     should be given                                   resources and has not     
     high priority                                     advanced as quickly.      
     for additional                                    
     research.                                         
  5  EPA should      Not applied      EPA did not      EPA cited limited time    
     estimate the                     estimate         and resources to estimate 
     benefits over                    benefits         benefits for years other  
     the regulatory                   covering the     than 2015, but plans to   
     time period,                     implementation   also include estimates    
     including both                   period and       for 2020, in the final    
     the                              expression       RIA. In addition, EPA did 
     implementation                   period of all    not think that this       
     period and the                   important health recommendation is         
     expression                       effects. EPA     particularly meaningful   
     period of all                    estimated the    to the NAAQS analysis     
     important                        benefits         because the only variable 
     health effects.                  expected in one  change over the course of 
     Because                          year only-the    the PM implementation     
     calculating                      base case year,  period is population. EPA 
     benefits for                     2015. The year   concluded that there      
     every future                     2015 is the      would not be much         
     year is                          first attainment difference between the    
     resource                         date when states benefit estimates given   
     intensive and                    should be in     in 5-year increments      
     unlikely to                      compliance with  (i.e., 2010 and 2015).    
     show true                        the new          
     increases in                     standards. The   
     precision,                       Clean Air Act    
     calculations                     allows for up to 
     can be made,                     a 5-year         
     for example,                     extension for    
     every fifth                      states that      
     year with                        cannot meet the  
     simple                           standards by the 
     interpolation                    attainment year. 
     techniques                       Therefore, the   
     applied to                       implementation   
     estimate                         period for some  
     benefits for                     states may       
     intervening                      extend to 2020.  
     years.                                            
  6  The components  Not applied      EPA presented a  EPA stated that the kind  
     of emissions                     qualitative      of information called for 
     estimates (such                  discussion about by this                   
     as number of                     emissions, but   recommendation-key input  
     vehicles in a                    did not address  data, assumptions, and    
     class, average                   components of    intermediate modeling     
     miles traveled                   emissions        outcomes-would be useful  
     per vehicle,                     estimates or     in future rule-making     
     and emissions                    provide          analyses, but that the    
     per mile)                        information to   agency needs to review a  
     should be                        allow readers to large amount of data to   
     presented with                   understand which determine which elements  
     and without                      components of    would be most helpful.    
     implementation                   emissions        EPA questioned the value  
     of the                           estimates drive  of incorporating all of   
     regulation at                    the reductions   the recommended           
     the national                     associated with  information in regulatory 
     level. This                      regulation, such impact analyses, noting   
     will help                        as activity      that it works to present  
     readers judge                    level or         enough information to     
     how reasonable                   emissions        readers while maintaining 
     these                            intensity. EPA   a document of manageable  
     predictions are                  did not present  length.                   
     and will                         historical                                 
     suggest which                    trends.          EPA stated that it needs  
     components of                                     time to design data       
     emissions                                         reporting strategies that 
     estimates drive                                   would be                  
     the emissions                                                               
     reductions                                        appropriate for the       
     associated with                                   different scales and      
     the regulation.                                   scopes of the regulatory  
     Historical                                        impact analyses. EPA      
     trends in these                                   plans to explore this     
     components                                        recommendation more       
     should also be                                    thoroughly as part of its 
     presented.                                        comprehensive economic    
                                                       analysis of the Clean Air 
                                                       Act.                      
                                                                                 
                                                       Finally, EPA officials    
                                                       suggested that the final  
                                                       PM RIA will respond to    
                                                       this recommendation in    
                                                       part by providing         
                                                       information about how     
                                                       emissions reductions      
                                                       might vary across PM      
                                                       sources in order to show  
                                                       the primary drivers of    
                                                       emissions reductions in   
                                                       the final RIA. EPA also   
                                                       plans to compare current  
                                                       data to the historical    
                                                       trends predicted in the   
                                                       past and show that EPA's  
                                                       current predictions are   
                                                       for the future.           
  7  EPA should      Not applied      EPA did not      EPA said its models are   
     quantify                         quantify the     not configured to         
     uncertainties                    uncertainties    quantify these sources of 
     with regard to                   related to       uncertainty. EPA would    
     future                           future           need to modify the model  
     population                       population       in order to do this       
     distributions                    distributions    analysis. Agency          
     and background                   and background   officials noted that the  
     disease rates.                   disease rates.   agency tries to be        
     EPA should also                  EPA did not      selective when            
     summarize what                   summarize what   determining which sources 
     is known about                   is known about   of uncertainty to assess  
     the potential                    the potential    because the cost of doing 
     importance of                    importance of    this work might outweigh  
     disease                          disease          the value added from the  
     interactions                     interactions and information.              
     and competing                    competing risks                            
     risks affecting                  affecting health In addition, EPA          
     the health                       outcomes of      officials said the agency 
     outcomes of                      primary          does not have all of the  
     primary                          interest.        data necessary to         
     interest and                                      reconfigure the models to 
     discuss the                                       quantify key sources of   
     possible biases                                   uncertainty. EPA          
     that might be                                     disagrees with the        
     introduced in                                     National Academies'       
     the final                                         comment that lack of      
     analysis by                                       information should not    
     changes in                                        preclude the              
     those factors.                                    quantification of         
                                                       uncertainty. EPA believes 
                                                       that using the techniques 
                                                       to quantify uncertainty   
                                                       without empirical data    
                                                       would generate results    
                                                       that could be more        
                                                       misleading than the       
                                                       results that do not       
                                                       account for uncertainty.  
                                                       EPA plans to add relevant 
                                                       uncertainty               
                                                       characterizations as it   
                                                       obtains data.             
  8  The lack of     Not applied      The model EPA    EPA has begun work on     
     clear                            used to estimate categorizing severity of  
     categorization                   benefits,        some health outcomes,     
     of severity of                   BenMAP, did not  including chronic         
     certain health                   account for the  bronchitis and asthma     
     outcomes in                      potential        incidence. EPA is         
     benefits                         variations in    continuing to refine its  
     analyses has                     severity of      BenMAP model to better    
     implications                     illnesses and    quantify and monetize     
     for the                          prevalence and   health outcomes.          
     quantification                   expected         
     and the                          incidence of     
     valuation of                     health effects.  
     these outcomes.                                   
     Although EPA                                      
     has made some                                     
     attempt to                                        
     recognize this                                    
     issue, it                                         
     should continue                                   
     to develop and                                    
     improve methods                                   
     used to                                           
     reconcile                                         
     differences                                       
     between the                                       
     severity of                                       
     disease                                           
     described in                                      
     air pollution                                     
     epidemiology                                      
     and that                                          
     commonly used                                     
     to develop                                        
     estimates of                                      
     background                                        
     disease                                           
     prevalence and                                    
     incidence.                                        
  9  EPA should give Not applied      EPA did not      EPA acknowledged the      
     more emphasis                    place more       importance of             
     to the                           emphasis on the  morbidity-i.e.,           
     assessment,                      assessment,      illness-benefits and is   
     presentation,                    presentation,    working to expand that    
     and                              and              analysis. For example,    
     communication                    communication of EPA told us that it plans 
     of changes in                    changes in       to include sensitivity    
     morbidity.                       morbidity. Prior analysis of changes in    
     Although often                   RIAs have        illnesses in the final PM 
     difficult to                     quantified       RIA. EPA also stated that 
     quantify, these                  expected changes the final PM RIA will     
     factors may                      in morbidity,    reflect updates to its    
     begin to play a                  such as          model-EPA is working to   
     more dominant                    reductions in    include projections that  
     role in benefit                  asthma and       will allow the agency to  
     analysis if the                  chronic          evaluate expected changes 
     value assigned                   bronchitis. The  in illnesses such as      
     to mortality                     draft PM RIA did asthma and chronic        
     decreases.                       not include      bronchitis.               
                                      additional       
                                      morbidity        
                                      information.     
                                      Moreover, the    
                                      main benefit     
                                      estimates in the 
                                      draft PM RIA did 
                                      not include      
                                      morbidity        
                                      estimates.       
  10 There is a      Not applied      EPA did not      EPA characterized this    
     common                           balance the      recommendation as not     
     misperception                    costs and        germane to the            
     that a high                      benefits-the     particulate matter        
     degree of                        Clean Air Act    regulatory impact         
     certainty is                     prohibits EPA    analysis because the      
     required for                     from basing      Clean Air Act prohibits   
     regulatory                       revisions to the the agency from           
     actions to take                  NAAQS on costs.  considering the costs     
     place to                         We note that in  when revising NAAQS.      
     protect public                   response to      
     health. As a                     other            
     result, primary                  recommendations, 
     health benefit                   EPA has taken    
     analyses that                    steps toward     
     more fully and                   assessing        
     accurately                       current levels   
     portray the                      of               
     uncertainties                    uncertainty-see  
     might not be                     appendix II,     
     considered                       recommendations  
     useful. It is                    14 and 20.       
     unrealistic for                                   
     EPA to defer                                      
     decisions until                                   
     it can make                                       
     them on the                                       
     basis of                                          
     perfect                                           
     science. A                                        
     careful and                                       
     deliberate                                        
     balancing of                                      
     the benefits                                      
     and costs is                                      
     required, and                                     
     this balancing                                    
     must be                                           
     informed by a                                     
     fair assessment                                   
     of the current                                    
     levels of                                         
     uncertainty and                                   
     a realistic                                       
     evaluation of                                     
     the likely                                        
     reductions in                                     
     uncertainty                                       
     attainable                                        
     through further                                   
     research.                                         
  11 EPA should      Not applied      EPA did not      Due to data and time      
     provide a                        provide the      constraints, EPA did not  
     summary of the                   summary table in summarize its conclusions 
     analysis                         the executive    in the executive summary  
     containing                       summary as       of the PM RIA.            
     information as                   outlined in the  
     outlined in the                  National         
     National                         Academies'       
     Academies'                       report,          
     report (table                    including a      
     6-1). This                       description of   
     information                      regulatory       
     would allow the                  options,         
     reader to                        boundaries of    
     evaluate the                     analysis,        
     study design                     regulatory       
     and verify                       baseline, and    
     estimates                        assumptions that 
     obtained in the                  have a           
     analysis.                        significant      
                                      impact on        
                                      results of       
                                      analysis.        
  12 To enhance the  Not applied      EPA did not      EPA stated that in        
     quality of                       convene a        addition to the advice it 
     future                           standing group   sought from the National  
     regulatory                       of experts to    Academies, the agency     
     benefit                          guide the        continues to seek input   
     analyses, a                      agency's initial from independent          
     standing,                        work on the      committees, including the 
     independent                      draft PM RIA. In Clean Air Scientific      
     technical                        the course of    Advisory Committee, the   
     review panel                     developing       Advisory Council on Clean 
     should advise                    economic         Air Compliance Analysis,  
     EPA in the                       methodologies,   and subcommittees         
     initial stages                   EPA sought and   chartered by the Science  
     of its benefit                   considered       Advisory Board's          
     analysis. This                   information from Environmental Economics   
     panel should                     existing         Advisory Committee. The   
     have expertise                   advisory         committees advised EPA on 
     in regulatory                    committees on    data, methods, and        
     options                          the soundness of modeling choices          
     analysis,                        certain          applicable to various     
     emissions and                    assumptions,     economic analyses,        
     exposure                         such as          including the draft PM    
     assessment,                      cessation lags.  RIA.                      
     toxicology,                                                                 
     epidemiology,                                     EPA stated that the costs 
     risk analysis,                                    involved in convening an  
     biostatistics,                                    entirely new panel could  
     and economics                                     be prohibitively          
     and should be                                     expensive, due to         
     appointed with                                    organizing costs and      
     strict                                            travel expenses and the   
     attention to                                      scope of the Academies'   
     avoiding                                          recommendation. As a      
     conflict of                                       result, EPA said that it  
     interest,                                         has attempted to take     
     balancing                                         advantage of existing     
     biases and                                        groups, such as the       
     ensuring broad                                    Advisory Council on Clean 
     representation.                                   Air Compliance Analysis,  
     This panel                                        rather than arrange for a 
     should be                                         new panel.                
     supported by                                                                
     permanent                                         Furthermore, EPA noted    
     technical staff                                   that new methodologies    
     to ensure                                         and assumptions used in   
     consistency of                                    the PM RIA were           
     reviews over                                      peer-reviewed and that    
     time. EPA                                         previously used methods   
     should follow                                     used in the RIA had       
     the panel's                                       already been reviewed and 
     guidance on the                                   validated in a prior      
     need for peer                                     context.                  
     review.                                           

Source: GAO analysis of National Academies and EPA information.

Appendix IV  GAO Contact and Staff Acknowledgments

John B. Stephenson, (202) 512-3841 or [email protected]

In addition to the contact named above, Christine Fishkin, Assistant
Director; Kate Cardamone; Nancy Crothers; Cindy Gilbert; Tim Guinane;
Jessica Lemke; and Meaghan K. Marshall made key contributions to this
report. Timothy Bober, Marcia Crosse, and Karen Keegan also made important
contributions.

(360660)

www.gao.gov/cgi-bin/getrpt? GAO-06-780 .

To view the full product, including the scope

and methodology, click on the link above.

For more information, contact John B. Stephenson at (202) 512-3841 or
[email protected].

Highlights of GAO-06-780 , a report to congressional requesters

July 2006

PARTICULATE MATTER

EPA Has Started to Address the National Academies' Recommendations on
Estimating Health Benefits, but More Progress Is Needed

A large body of scientific evidence links exposure to particulate matter-a
widespread form of air pollution-to serious health problems, including
asthma and premature death. Under the Clean Air Act, the Environmental
Protection Agency (EPA) periodically reviews the appropriate air quality
level at which to set national standards to protect the public against the
health effects of particulate matter. EPA proposed revisions to these
standards in January 2006 and issued a draft regulatory impact analysis of
the revisions' expected costs and benefits.

The estimated benefits of air pollution regulations have been
controversial in the past. A 2002 National Academies report generally
supported EPA's approach but made 34 recommendations to improve how EPA
implements its approach. GAO was asked to determine whether and how EPA
applied the Academies' recommendations in its estimates of the health
benefits expected from the January 2006 proposed revisions to the
particulate matter standards. GAO examined the draft analysis, met with
EPA officials, and interviewed members of the National Academies'
committee. In providing technical comments on the report, EPA officials
said it was fair and balanced and noted the agency's progress in
addressing recommendations via research and development and other
analyses.

EPA has begun to change the way it conducts and presents its analyses of
health benefits in response to recommendations from the National
Academies. Specifically, EPA applied, at least in part, 22-or about
two-thirds-of the Academies' recommendations to its health benefit
analysis of proposed revisions to particulate matter standards. For
example, in response to some of the recommendations, EPA took steps toward
conducting a more rigorous assessment of uncertainty by, for instance,
evaluating how benefits could change under different assumptions and
discussing sources of uncertainty not included in the benefit estimates.
In one case, EPA applied an alternative technique, called expert
elicitation, for evaluating uncertainty by systematically gathering expert
opinion about the uncertainty underlying the causal link between exposure
to particulate matter and premature death. Consistent with the National
Academies' recommendation to assess uncertainty by developing ranges of
estimates and specifying the likelihood of attaining them, EPA used expert
elicitation to develop ranges of reductions in premature death expected
from the proposed revisions. EPA officials said that ongoing research and
development efforts will allow the agency to gradually achieve more
progress in applying the recommendations. We note that robust uncertainty
analysis is important because estimates of health benefits can be highly
uncertain, as the draft regulatory impact analysis for particulate matter
illustrates. EPA viewed the estimates in this analysis as so uncertain
that it chose not to present them in the executive summary.

For various reasons, EPA has not applied the remaining 12 recommendations
to the analysis, such as the recommendation to evaluate the impact of
using the simplifying assumption that each component of particulate matter
is equally toxic. EPA officials viewed most of these recommendations as
relevant to its health benefit analyses and, citing the need for
additional research and development, emphasized the agency's commitment to
continue to respond to the recommendations. For example, EPA did not
believe that the state of scientific knowledge on the relative toxicity of
particulate matter components was sufficiently developed to include in the
January 2006 regulatory impact analysis, and the agency is currently
sponsoring research on this issue. In addition, a senior EPA official said
that insufficient resources impeded the agency's progress in applying the
recommendations, citing, in particular, the limited availability of
skilled staff, time, and other resources to conduct the required analyses
and research and development. EPA officials also said that some of the
recommendations the agency did not apply to the draft analysis, such as
one calling for a summary table describing key analytical information to
enhance transparency, will be applied to the analysis supporting the final
rule. To the extent that EPA continues to make progress addressing the
Academies' recommendations, decision makers and the public will be better
able to evaluate the basis for EPA's air regulations.
*** End of document. ***