Public Health and Hospital Emergency Preparedness Programs:
Evolution of Performance Measurement Systems to Measure Progress
(23-MAR-07, GAO-07-485R).
The September 11, 2001, terrorist attacks, the anthrax incidents
during the fall of 2001, Hurricane Katrina, and concerns about
the possibility of an influenza pandemic have raised public
awareness and concerns about the nation's public health and
medical systems' ability to respond to bioterrorist events and
other public health emergencies. From 2002 to 2006, the Congress
appropriated about $6.1 billion to the Department of Health and
Human Services (HHS) to support activities to strengthen state
and local governments' emergency preparedness capabilities under
the Public Health Security and Bioterrorism Preparedness and
Response Act of 2002 (Preparedness and Response Act). HHS has
distributed funds annually to 62 recipients, including all 50
states and 4 large municipalities, through cooperative agreements
under two programs--the Centers for Disease Control and
Prevention's (CDC) Public Health Emergency Preparedness Program,
and the Health Resources and Services Administration's (HRSA)
National Bioterrorism Hospital Preparedness Program. The common
goal of CDC's and HRSA's preparedness programs is to improve
state and local preparedness to respond to bioterrorism and other
large-scale public health emergencies, such as natural disasters
or outbreaks of infectious disease. Annually, both CDC and HRSA
develop and issue program guidance for recipients that describes
activities necessary to improve their ability to respond to
bioterrorism and other public health emergencies and sets out
requirements for measuring their performance. Each recipient is
required to submit periodic reports that track progress in
improving their preparedness. As a result of the nation's
ineffective response to Hurricane Katrina and the need to prepare
for a possible influenza pandemic, members of the Congress have
raised questions about CDC's and HRSA's efforts to monitor the
progress of their preparedness programs. Because of these
questions, we are reporting on (1) how CDC's and HRSA's
performance measurement systems have evolved and (2) how CDC and
HRSA are using these systems to measure the progress of their
preparedness programs.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-485R
ACCNO: A67187
TITLE: Public Health and Hospital Emergency Preparedness
Programs: Evolution of Performance Measurement Systems to Measure
Progress
DATE: 03/23/2007
SUBJECT: Bioterrorism
Cooperative agreements
Emergency preparedness
Emergency preparedness programs
Emergency response funds
Homeland security
Local governments
Performance measures
Program evaluation
Public health
Reporting requirements
Strategic planning
CDC Public Health Preparedness and
Response for Bioterrorism Program
National Bioterrorism Hospital
Preparedness Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-485R
* [1]PDF6-Ordering Information.pdf
* [2]Order by Mail or Phone
March 23, 2007
The Honorable Bennie G. Thompson
Chairman
Committee on Homeland Security
House of Representatives
The Honorable Judd Gregg
Ranking Minority Member
Committee on the Budget
United States Senate
The Honorable Charles E. Grassley
Ranking Minority Member
Committee on Finance
United States Senate
The Honorable Henry A. Waxman
Chairman
Committee on Oversight and Government Reform
House of Representatives
The Honorable Edward J. Markey
House of Representatives
Subject: Public Health and Hospital Emergency Preparedness Programs:
Evolution of Performance Measurement Systems to Measure Progress
The September 11, 2001, terrorist attacks, the anthrax incidents during
the fall of 2001, Hurricane Katrina, and concerns about the possibility of
an influenza pandemic have raised public awareness and concerns about the
nation's public health and medical systems' ability to respond to
bioterrorist events and other public health emergencies. From 2002 to
2006, the Congress appropriated about $6.1 billion to the Department of
Health and Human Services (HHS) to support activities to strengthen state
and local governments' emergency preparedness capabilities under the
Public Health Security and Bioterrorism Preparedness and Response Act of
2002 (Preparedness and Response Act).^1 HHS has distributed funds annually
to 62 recipients, including all 50 states and 4 large municipalities,
through cooperative agreements under two programs--the Centers for Disease
Control and Prevention's (CDC) Public Health Emergency Preparedness
Program,^2 and the Health Resources and Services Administration's (HRSA)
National Bioterrorism Hospital Preparedness Program. The common goal of
CDC's and HRSA's preparedness programs is to improve state and local
preparedness to respond to bioterrorism and other large-scale public
health emergencies, such as natural disasters or outbreaks of infectious
disease.
^1Pub. L. No. 107-188, 116 Stat. 594. The Pandemic and All-Hazards
Preparedness Act, Pub. L. No. 109-417, 120 Stat. 2831 enacted December 19,
2006, reauthorized and amended the Preparedness and Response Act and
authorized appropriations for HHS's Centers for Disease Control and
Prevention's and Health Resources and Services Administration's public
health and hospital preparedness programs through 2011.
To guide efforts by federal, state, and local departments and agencies to
prepare and respond to terrorism and other major emergencies, the federal
government has developed a number of national strategies, including a
National Strategy for Homeland Security, which was issued in July 2002.^3
Among other things, the National Strategy for Homeland Security requires
federal government departments and agencies to create performance measures
to evaluate progress in achieving homeland security initiatives, including
national preparedness and emergency response, and to allocate future
resources. Annually, both CDC and HRSA develop and issue program guidance
for recipients that describes activities necessary to improve their
ability to respond to bioterrorism and other public health emergencies and
sets out requirements for measuring their performance. Each recipient is
required to submit periodic reports that track progress in improving their
preparedness.
As a result of the nation's ineffective response to Hurricane Katrina and
the need to prepare for a possible influenza pandemic, members of the
Congress have raised questions about CDC's and HRSA's efforts to monitor
the progress of their preparedness programs. Because of these questions,
we are reporting on (1) how CDC's and HRSA's performance measurement
systems have evolved and (2) how CDC and HRSA are using these systems to
measure the progress of their preparedness programs. Enclosure I contains
the information we provided to your staff at our February 28, 2007,
briefing.
To do our work, we reviewed and analyzed federal government documents
related to national security and emergency preparedness. We also obtained
reports and interviewed officials from federal agencies^4 that had
evaluated CDC's and HRSA's public health and hospital preparedness
programs, professional associations involved in emergency preparedness,
and policy research organizations that had published assessments or
evaluations of public health and hospital preparedness programs. We
analyzed CDC and HRSA documents and interviewed officials to determine how
they have developed and implemented performance management systems for
their cooperative agreement programs, including recipient reporting
requirements, and systems for collecting data from recipients.
Additionally, we analyzed other CDC and HRSA documents to identify
procedures in place for management review of program progress and for
providing feedback and suggestions for program improvements to recipients.
We did not evaluate the actual performance measures adopted by CDC or HRSA
or examine the accuracy or completeness of recipients' self-reported data
as contained in the progress reports they are required to submit to CDC or
HRSA. See enclosure II for detailed information on our scope and
methodology. We conducted our work from June 2006 through March 2007 in
accordance with generally accepted government auditing standards.
^2CDC's program was formerly known as the Public Health Preparedness and
Response for Bioterrorism Program.
^3These strategies also include the National Strategy for Pandemic
Influenza and the National Security Strategy.
^4The federal agencies include HHS's Office of Inspector General (OIG),
HHS's Agency for Healthcare Research and Quality (AHRQ), and the
Congressional Research Service (CRS).
Results in Brief
Since 2002, CDC's and HRSA's performance measurements have evolved from
measuring capacity to assessing capability. Early in their programs, both
agencies used markers or values that they called benchmarks to measure
capacity-building efforts, such as purchasing equipment and supplies and
acquiring personnel.^5 These benchmarks were developed from activities
authorized in the Preparedness and Response Act. In 2002, CDC established
14 benchmarks, such as requiring each recipient to designate an executive
director of the bioterrorism and response program, establish a
bioterrorism advisory committee, and develop a statewide response plan.
From 2003 to 2005, CDC further developed its performance measurements by
obtaining input from stakeholders to make a transition from using
benchmarks focused on capacities to using performance measures focused on
capabilities, such as whether personnel have been trained and can
appropriately use equipment. In 2006, CDC continued to work with
stakeholders to refine its performance measures. At the beginning of its
program in 2002, HRSA established 5 benchmarks, such as requiring each
recipient to designate a coordinator for bioterrorism planning, establish
a hospital preparedness committee, and develop a plan for hospitals to
respond to a potential epidemic. From 2003 to 2005, HRSA modified existing
benchmarks and added new ones, such as training benchmarks, based on the
existing legislation and input from stakeholders. In 2006, HRSA convened
an expert panel to propose a set of performance measures focused on
capabilities. CDC and HRSA officials told us they will continue to face
challenges as their performance measures evolve, such as gaining consensus
among stakeholders in light of minimal scientific data about public health
and hospital emergency preparedness.
CDC and HRSA use data from recipients' reports and site visits to monitor
recipients' progress in improving their ability to respond to bioterrorism
events and other public health emergencies. CDC and HRSA project officers
use performance measurement data from recipients' required progress
reports, along with site visits, to monitor progress and provide feedback
about whether individual recipients have accomplished activities related
to their ability to respond to bioterrorism events and other public health
emergencies. Currently, there are no standard analyses or reports that
enable CDC and HRSA to compare data across recipients to measure
collective progress, compare progress across recipients' programs, or
provide consistent feedback to recipients. However, in mid to late 2006
both agencies began developing formal data analysis programs that are
intended to validate recipient-reported data and assist in generating
standardized reports. According to CDC officials, CDC plans to finish
validation projects by August 2007 and then develop routine reports
summarizing individual recipient and national progress. In addition, CDC
plans to issue a report by the end of 2007 providing a "snapshot" of the
progress recipients have made in building emergency readiness capacity and
addressing how CDC will measure capability in the future. However, because
of the expected move of HRSA's program to a different HHS office in 2007,
its schedule for finishing data validation was tentative at the time we
briefed your staff. Furthermore, due to the expected move, HRSA officials
said at that time that decisions about whether to issue a report in 2007
on recipients' progress also had not been made.
^5According to CDC officials, acquisition of personnel was necessary in
order to develop and implement the activities authorized in the
Preparedness and Response Act.
^6These strategies also include the National Strategy for Pandemic
Influenza and the National Security Strategy.
^7Homeland Security Presidential Directives record and communicate
presidential decisions about homeland security policies of the United
States.
^8According to CDC officials, acquisition of personnel was necessary in
order to develop and implement emergency preparedness activities
authorized by the Preparedness and Response Act.
Agency Comments
We requested comments on a draft of this report from HHS. The department
provided written comments that are reprinted in enclosure III.
In commenting on this draft, HHS provided additional information about the
transfer on March 5, 2007, of the National Bioterrorism Hospital
Preparedness Program from HRSA to the new HHS Office of the Assistant
Secretary for Preparedness and Response. According to HHS, it has made a
number of changes that it believes will improve its ability to monitor
performance at the individual recipient level and for the program overall.
HHS is also planning to conduct an analysis of the performance data for
existing recipients for fiscal years 2002-2006 in order to develop a more
complete picture of levels of preparedness from all National Bioterrorism
Hospital Preparedness Program recipients.
Many of the initiatives outlined in HHS' comments were begun after our
briefings to your staff on February 28, 2007, and are still being
implemented; we are unable to comment on their effectiveness. As we
continue to evaluate emergency preparedness programs we will review the
results of their continued efforts to develop measurable evidence-based
benchmarks and objective standards and their ability to compare data
across recipients to measure collective progress, compare progress across
recipients' programs, or provide consistent feedback to recipients.
- - - - -
As arranged with your offices, unless you release its content earlier, we
plan no further distribution of this report until 30 days after its
issuance date. At that time, we will send copies of this report to the
Secretary of HHS and other interested parties. We will also make copies
available to others on request. In addition, the report will be available
at no charge on the GAO Web site at http://www.gao.gov . Contact points
for our Office of Congressional Relations and Public Affairs may be found
on the last page of this report.
If you and your staff have any questions or need additional information,
please contact me at (202) 512-7101, or bascettac@gao.gov . Contact
points for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. GAO staff members who made major
contributions to this report are listed in enclosure IV.
Cynthia A. Bascetta
Director, Health Care
Enclosures - 4
Enclosure I
Information Presented in Briefing on February 28, 2007
The information in this enclosure is taken directly from the slides used
in the briefing presented to the staffs of the Honorable Judd Gregg,
Ranking Minority Member, Senate Committee on the Budget; the Honorable
Charles E. Grassley, Ranking Minority Member, Senate Committee on Finance;
the Honorable Bennie G. Thompson, Chairman, House Committee on Homeland
Security; the Honorable Henry A. Waxman, Chairman, House Committee on
Oversight and Government Reform; and the Honorable Edward J. Markey, House
of Representatives on February 28, 2007.
Introduction (slides 3 through 6)
The September 11, 2001, terrorist attacks, the anthrax incidents,
Hurricane Katrina, and concerns about the possibility of an influenza
pandemic have raised public awareness and concerns about the nation's
public health and medical systems' ability to respond to bioterrorist
events and other public health emergencies. In November 2002, the Congress
passed legislation creating the Department of Homeland Security (DHS),
giving it the overall responsibility for managing emergency preparedness.
The Department of Health and Human Services (HHS) is designated as the
primary agency for implementing activities relating to public health and
hospital emergency preparedness.
From 2002 to 2006, the Congress appropriated about $6.1 billion to support
activities under the Public Health Security and Bioterrorism Preparedness
and Response Act of 2002 (Preparedness and Response Act) to strengthen
state and local governments' emergency readiness capabilities. HHS has
distributed these funds annually to 62 recipients, including all 50 states
and 4 large municipalities, through cooperative agreements under two
programs:
o Centers for Disease Control and Prevention's (CDC) Public Health
Emergency Preparedness Program (formerly the Public Health
Preparedness and Response for Bioterrorism Program), and
o Health Resources and Services Administration's (HRSA) National
Bioterrorism Hospital Preparedness Program.
In addition to bioterrorism, these programs also address other
large-scale public health emergencies, such as natural disasters
or outbreaks of infectious disease. This "all-hazards" approach
recognizes that some aspects of response to bioterrorism, such as
providing emergency medical services and managing mass casualties,
can be the same as for response to other public health
emergencies.
Public Law 109-417, the Pandemic and All-Hazards Preparedness Act,
enacted December 19, 2006, amended the Preparedness and Response
Act and authorizes appropriations for CDC's and HRSA's public
health and hospital preparedness programs through 2011. The
legislation also creates a new Assistant Secretary for
Preparedness and Response in HHS and transfers responsibility for
HRSA's hospital preparedness program to this position. The program
is expected to move some time in 2007. To guide preparedness and
response for terrorism and other major emergencies, the federal
government developed a number of national strategies, including a
National Strategy for Homeland Security issued in July 2002.^6
This national strategy requires federal government departments and
agencies to create performance measures to evaluate progress in
achieving homeland security initiatives, including national
preparedness and emergency response, and to allocate future
resources.
Purpose and Questions (slide 7)
As a result of the nation's ineffective response to Hurricane
Katrina and the need to prepare for a possible influenza pandemic,
members of the Congress have raised questions about CDC's and
HRSA's efforts to monitor the progress of their preparedness
programs.
To assess CDC's and HRSA's systems to monitor these programs, we
reviewed the following questions:
1. How have CDC's and HRSA's performance measurement
systems evolved?
2. How are CDC and HRSA using these systems to
measure the progress of their preparedness programs?
Scope and Methodology (slides 8 through 10)
To do our work, we interviewed officials from
o HHS's Office of Public Health Emergency Preparedness (OPHEP),
Office of the Assistant Secretary for Planning and Evaluation,
Office of the Inspector General (OIG), and Agency for Healthcare
Research and Quality (AHRQ);
o CDC's Coordinating Office for Terrorism Preparedness and
Emergency Response;
o HRSA's National Bioterrorism Hospital Preparedness Program;
o Congressional Research Service; and
o professional associations involved in emergency preparedness and
policy research organizations that had published assessments or
evaluations of public health and hospital preparedness programs.
We also reviewed and analyzed documents from
o The Executive Office of the President, including the National
Strategy for Homeland Security and Homeland Security Presidential
Directives;
o DHS, including the National Response Plan, the Interim National
Preparedness Goal, and the draft Target Capabilities List;
o HHS's OIG and AHRQ;
o Congressional Research Service;
o Office of Management and Budget, including Program Assessment
Rating Tool reviews;
o CDC and HRSA on the development of performance management
systems and recipients' annual applications and progress reports;
and
o professional associations and policy research organizations.
We did not evaluate the actual performance measures adopted by CDC
or HRSA or examine the accuracy or completeness of recipients'
self-reported data as contained in the progress reports they are
required to submit to CDC or HRSA. Our review was conducted from
June 2006 through March 2007 in accordance with generally accepted
government auditing standards.
Background (slides 11 through 17)
CDC's and HRSA's Preparedness Programs
The common goal of CDC's and HRSA's preparedness programs is to
improve state and local preparedness to respond to bioterrorism
and other public health emergencies.
o CDC's program focuses on public health preparedness.
o HRSA's program focuses on hospital preparedness.
CDC and HRSA annually distribute program funds to recipients.
These funds are used to improve their ability to respond to
bioterrorism and other public health emergencies, such as training
volunteers to provide mass vaccinations or antibiotics in the
event of a public health emergency.
CDC and HRSA also develop program guidance for recipients that
describes activities necessary to improve preparedness and sets
out requirements for measuring recipients' performance.
CDC's Preparedness Program
CDC distributes funds under its cooperative agreements on an
annual basis. Each recipient
o must apply annually for these funds;
o receives a base amount, plus an amount based on its proportional
share of the national population; and
o has flexibility in how to distribute the funds to local public
health agencies based on the workplan submitted to CDC with the
recipient's application.
Each recipient must submit reports that track progress in
improving its ability to respond to bioterrorism and other public
health emergencies. These have included quarterly, midyear, and
annual reports.
HRSA's Preparedness Program
HRSA distributes funds under its cooperative agreements on an
annual basis. Each recipient
o receives a base amount, plus an amount based on its proportional
share of the national population; and
o must allocate at least 75 percent of its funds to hospitals or
other health care entities.
o Recipients distribute most of the funds to
hospitals, with a small portion going to other
entities such as community health centers, emergency
medical services, and poison control centers.
o Recipients may use the remaining funds to support
their administrative costs and needs assessments.
Each recipient must submit midyear and annual reports that track
progress in improving its ability to respond to bioterrorism and
other public health emergencies.
Prior Reviews of CDC's and HRSA's Preparedness Programs
Several government and private studies, including those conducted
by GAO, HHS's OIG, and Rand, have noted weaknesses in CDC's and
HRSA's preparedness programs.
o In February 2004, we reported (GAO-04-360R) that although the
states' progress fell short of 2002 goals and much remained to be
accomplished, these programs enabled states to make needed
improvements in public health and health care capabilities
critical for preparedness.
o Since December 2002, HHS's OIG has issued seven evaluation and
inspection reports on program results. It found that all of the
studied recipients had prepared bioterrorism responses and were
working to strengthen their infrastructure, but barriers to
preparedness remained, including problems with staffing, funding,
and communication and the need for standards and guidance.
o Since 2001, Rand has conducted many studies related to
preparedness for public health emergencies. Rand studied how
public health preparedness is transforming public health agencies
and found
o the preparedness mission has raised challenges in
terms of accountability among local health
jurisdictions;
o it is difficult to assess preparedness because
measures to define and assess preparedness, and a
strong evidence base to support those measures are
lacking; and
o it is difficult to measure preparedness because it
involves measuring the capacity to deal with
situations that rarely happen.
Under a contract with HHS, Rand currently is convening expert
panels and performing literature searches to help define
preparedness.
Presidential Directive 8--National Preparedness
Homeland Security Presidential Directive 8 provides some guidance
on implementing the National Strategy for Homeland Security.
Consistent with the directive, DHS developed the Interim National
Preparedness Goal and the draft Target Capabilities List and
issued them in 2005.^7
o The Interim National Preparedness Goal establishes preparedness
priorities, targets, and standards for preparedness assessments
and strategies to align efforts of federal, state, local, tribal,
private-sector, and nongovernmental entities.
o The draft Target Capabilities List identifies 37 capabilities
that federal, state, local, tribal, private-sector, and
nongovernmental entities need in order to prevent, protect
against, respond to, and recover from a major event to minimize
the impact on lives, property, and the economy.
CDC's and HRSA's preparedness programs provide both funds and
guidance to state and local entities and hospitals to help them
develop these capabilities and meet these preparedness priorities.
Performance Measurement Systems
Early in a program, performance measurement systems can focus on
measuring capacity, such as equipment and supplies purchased and
personnel hired.
As programs mature and more data and scientific evidence are
available, performance measurement systems can focus more on
measuring capabilities, such as whether personnel are trained and
can appropriately use equipment and supplies. Measurements can
include
o type or level of program activities conducted (process),
o direct products and services delivered (outputs), or
o results of those products and services (outcomes).
Finding 1: CDC and HRSA Performance Measures Evolved from
Measuring Capacity to Assessing Capability (slides 18 through 29)
In 2002, CDC's and HRSA's efforts focused on measuring capacity,
such as the type of staff hired and equipment needed to respond to
a bioterrorism attack. To do this, CDC and HRSA identified markers
or values against which recipients were expected to measure their
performance. These initial markers or values, which they called
benchmarks, were developed from emergency preparedness activities
authorized in the Preparedness and Response Act.
From 2003 to 2006, CDC and HRSA changed their approach from using
benchmarks that measure capacity to using performance measures
that focus on whether a program has met standards assessing
capabilities.
In 2004, CDC and HRSA increased their coordination and in 2005
began to coordinate with DHS to align their preparedness programs
with the Interim National Preparedness Goal and draft Target
Capabilities List.
2002--CDC's Initial Measurements Based on Legislation
In 2002, CDC initially established its performance measurement
systems using benchmarks based on emergency preparedness
activities authorized in the Preparedness and Response Act.
CDC officials said these initial benchmarks measured program
capacity-building efforts such as purchasing equipment and
supplies and acquiring personnel.^8
CDC established 14 critical benchmarks, such as requiring each
recipient to designate an executive director of the bioterrorism
and response program, establish a bioterrorism advisory committee,
and develop a statewide response plan.
2003 to 2005--CDC's Transition from Measuring Capacity to
Assessing Capability
From 2003 to 2005, CDC began to include the participation and
input of stakeholders--other federal agencies, recipients of
program funds, public health professional association officials,
and industry experts--as it further developed its performance
measurements. This input resulted in modifications of the
benchmarks and the transition from benchmarks to performance
measures that address capabilities.
o In 2003, an initial draft of over 100 proposed measures was
developed from input by CDC internal subject matter experts. An
external workgroup, including professional association
representatives, reviewed and assessed the proposed measures. Some
of the measures focused on new areas, such as exercising,
drilling, and training.
o In 2004, CDC convened a second CDC internal expert panel to
conduct a literature search to identify evidence-based criteria to
support the performance measures. The panel consolidated the over
100 performance measures into 47 interim performance measures.
Subsequent field-testing eliminated one proposed measure.
o In late 2004, CDC held teleconferences with selected recipients
and professional association representatives to discuss these
interim performance measures. This process reduced the number of
performance measures to 34.
o In 2005, CDC introduced the 34 performance measures in the 2005
cooperative agreement guidance and field tested the new measures
in five locations.
Example of the transition of a CDC benchmark into a performance
measure that addresses capabilities:
o 2002 benchmark: Recipients were required to develop a system to
receive and evaluate urgent disease reports on a 24-hour-per-day,
7-day-per-week basis.
o 2003/2004 benchmark: Recipients were required to complete
development of and maintain a system to receive and evaluate
urgent disease reports.
o 2005 performance measure: Recipients were required to meet a
target time of 15 minutes for a knowledgeable public health
professional to respond to a call or a communication that appears
to be of urgent public health consequence.
2005 to 2006--CDC's Refinement of Capability Assessment
In late 2005, CDC met with representatives from professional
organizations and state and local public health laboratories and
health departments to review and refine the performance measures.
In 2006, CDC held further meetings with seven recipients and other
stakeholders to discuss data collection efforts for performance
measures and found that gathering some of the data would not be
feasible. As a result, CDC further reduced the number of
performance measures from 34 to 23.
CDC's 2006 guidance with the 23 performance measures was issued in
June 2006. Recipients were expected to comply with this guidance
when implementing their 2006 programs, during the period from
August 31, 2006, to August 30, 2007.
2002--HRSA's Initial Measurements Based on Legislation
In 2002, HRSA initially established its performance measurement
systems using benchmarks based on emergency preparedness
activities authorized in the Preparedness and Response Act.
HRSA officials said these initial benchmarks measured program
capacity-building efforts such as purchasing equipment and
supplies and acquiring personnel.
HRSA established five critical benchmarks, such as requiring each
recipient to designate a coordinator for bioterrorism planning,
establish a hospital preparedness committee, and develop a plan
for hospitals to respond to a potential epidemic.
2003 to 2005--HRSA's Benchmarks Modified and Expanded
From 2003 to 2005, HRSA, like CDC, began to include the
participation and input of stakeholders--federal agencies,
cooperative agreement recipients, public health professional
association officials, and industry experts--as it further
developed its performance measurements. This input resulted in
modifications of the benchmarks.
o In 2003, HRSA added new benchmarks based on the existing
legislation and meetings and discussions with stakeholders. The
benchmarks focused on such things as exercising, drilling, and
training.
o In 2004, each of HRSA's benchmarks was divided into
HRSA-identified "sentinel indicators," which are smaller component
tasks that are intended to accomplish the larger benchmark
activity. For example, for the benchmark "Surge Capacity: Beds,"
one of the sentinel indicators is the number of additional
hospital beds for which a recipient could make patient care
available within 24 hours.
o In 2005, HRSA increased the number of sentinel indicators from
21 to 72 at HHS's request. For example, HHS asked for additional
measures to identify bed capacity for trauma and burn victims.
2006--HRSA's Transition from Measuring Capacity to Assessing
Capability
In early 2006, HRSA convened an expert panel that proposed a set
of performance measures, which were then disseminated to
stakeholders such as recipients, professional associations,
industry experts, and federal agencies for feedback.
This input resulted in adoption of 6 performance measures and 17
program measures (HRSA defined program measures as a mixture of
program activities and process and outcome measures) that focus on
capabilities.
HRSA also maintained reporting requirements for 17 of its 72
sentinel indicators.
HRSA's 2006 performance and program measures and sentinel
indicators were not issued with its guidance in July 2006 because
HRSA officials were awaiting final approval by HHS. These measures
were issued in December 2006. However, according to HRSA
officials, recipients were aware of the expectations contained in
the guidance because they helped develop them. As such, it was
HRSA's expectation that recipients would comply with them when
implementing their 2006 programs, during the period from September
1, 2006, to August 31, 2007.
Increased Coordination between CDC and HRSA; Coordination
Initiated with DHS
In 2004, CDC and HRSA increased their coordination and in 2005
began to coordinate with DHS to align their preparedness programs
with the Interim National Preparedness Goal and draft Target
Capabilities List. For example,
o CDC and HRSA project officers shared information in monthly
conference calls.
o CDC subject matter experts assisted HRSA's recipients.
o CDC, HRSA, and DHS created a Joint Advisory Committee in 2005 to
create common terminology for their respective programs and
improve commonality in their guidance.
o CDC and HRSA officials stated that in 2005 they had more closely
aligned their performance measurements with the draft Target
Capabilities List and the Interim National Preparedness Goal.
Figure 1 provides an example of how CDC and HRSA have aligned
their performance measurements with DHS's draft Target
Capabilities List and the Interim National Preparedness Goal.
Figure 1: Alignment of CDC and HRSA Performance Measures with
DHS's Draft Target Capabilities List and the Interim National
Preparedness Goal
CDC's and HRSA's Challenges
According to CDC and HRSA officials, they will continue to face
challenges as their performance measures evolve, because gaining
consensus among the various stakeholders--federal agencies, state
and local governments, and professional associations--is
difficult. These difficulties arise because
o minimal scientific data exist in this new area of public health
and hospital emergency preparedness to guide performance
measurement systems; and
o scientists, subject matter experts, and program officials can
disagree as to what could and should be measured.
Finding 2: CDC and HRSA Use Data from Recipients' Reports and Site
Visits to Measure Progress (slides 30 through 36)
CDC and HRSA project officers use performance measurement data
from recipients' required reports, along with site visits, to
monitor progress and provide feedback about whether individual
recipients meet goals and accomplish activities related to their
ability to respond to bioterrorism events and other public health
emergencies.
Both CDC and HRSA are making improvements to address the need for
formal data analysis programs based on validated data and
standardized procedures.
Report and Site Visit Data
CDC and HRSA project officers are responsible for monitoring
individual recipients' progress, providing technical assistance,
and giving feedback on their emergency preparedness activities.
Experts in areas such as epidemiology, laboratory testing, and
surveillance assist project officers in providing technical
assistance.
o Project officers analyze and monitor individual recipients'
progress from the information gathered through recipients'
progress reports, phone calls, and e-mails and by conducting site
visits.
o Project officers use the information and their analyses of it to
(1) provide recipients with technical assistance and feedback on
their ability to respond to bioterrorism and other public health
emergencies, (2) determine issues to discuss during future site
visits, and (3) assist recipients in developing future cooperative
agreement applications.
o Project officers also collaborate with recipients to identify
their specific needs for improving their emergency preparedness.
For example, prior to site visits CDC project officers ask
recipients what type of technical assistance they need and then
include appropriate subject matter experts on the site visit.
Providing Feedback
Both CDC and HRSA have various methods for providing feedback on
progress to recipients:
o Project officers determine the type and amount of feedback to
provide each recipient on their progress.
o CDC and HRSA periodically provide recipients with information
about promising practices and lessons learned on improving their
ability to respond to bioterrorism and other public health
emergencies.
o CDC and HRSA both hold annual conferences with all recipients to
provide training, and other information such as changes to program
guidance.
Standard Analysis and Reports Currently Lacking
CDC and HRSA officials told us that project officers lack standard
protocols, checklists, or procedures for analyzing recipients'
reports that include both qualitative and quantitative data.
Consequently, each project officer develops his or her own methods
or procedures for analyzing and measuring recipients' progress.
CDC and HRSA project officers have not generated standardized
reports summarizing individual or collective recipients' progress
and activities.
Ongoing Improvements
However, both CDC and HRSA are making improvements in measuring
progress:
o In mid to late 2006, both CDC and HRSA began developing formal
data analysis programs. They plan to generate standardized reports
for management and other stakeholders as needed.
o CDC and HRSA plan to put procedures in place to validate the
accuracy, reasonableness, and completeness of selected data that
recipients self-report.
o Officials said validation is needed to
o ensure that reports based on recipients' data
provide accurate information;
o determine whether all recipients are comparably
reporting the status of their preparedness; and
o allow managers to make informed decisions to
improve the individual recipients' cooperative
agreements and, ultimately, the nation's
preparedness.
Once the data validation projects are completed, CDC officials
plan to develop routine reports with specific recipient
information and reports that provide national summaries. CDC
officials plan to finish the validation projects by August 2007.
CDC officials said that in the interim they would continue to use
many of the measurements from 2005 and 2006 to trace recipients'
progress.
HRSA's time frame to finish validation is tentative due to the
hospital preparedness program's expected move to another office
within HHS in 2007.
As programs mature and more data become available, performance
measures will continue to evolve to better measure outcomes.
Because the process is iterative, the system allows for continuous
improvements.
Plans for Making Preparedness Information Public
CDC plans to issue a report by the end of 2007 providing a
"snapshot" of the progress recipients have made in building
emergency readiness capacity and addressing how CDC will measure
capability in the future.
HRSA officials said that decisions about whether to issue a report
in 2007 on recipients' progress had not been made due to the
hospital preparedness program's expected move to another office
within HHS.
Beginning in 2009, and every 4 years thereafter, the Pandemic and
All-Hazards Preparedness Act requires that HHS report to the
Congress on the status of public health emergency preparedness and
response.
o This includes a National Health Security Strategy and an
implementation plan that includes an evaluation of progress made
toward preparedness based on evidence-based benchmarks and
objective standards that measure levels of preparedness.
o The Act is generally silent on the type of information that is
to be included in this evaluation other than an aggregate and
recipient-specific breakdown of funding.
Scope and Methodology
To determine how the Centers for Disease Control and Prevention's
(CDC) and Health Resources and Services Administration's (HRSA)
performance measurement systems have evolved, we reviewed and
analyzed federal government documents related to national security
and emergency preparedness, including the Executive Office of the
President's National Strategy for Homeland Security and several
Homeland Security Presidential Directives, and the Department of
Homeland Security's (DHS) National Response Plan, Interim National
Preparedness Goal, and draft Target Capabilities List. We
interviewed officials from CDC and HRSA to identify and document
how they have developed and implemented performance management
systems for their cooperative agreement programs, including
determining how standards were identified, indicators were
selected, goals and targets were established, measures were
defined, data systems were developed, and data were collected from
recipients. We obtained and analyzed CDC and HRSA documents to
identify the development of performance measures from program
inception to the present, recipient reporting requirements, and
systems for collecting data from cooperative agreement recipients.
We also obtained reports and interviewed officials from federal
agencies that had evaluated CDC's and HRSA's public health and
hospital preparedness programs, including HHS's Office of
Inspector General, HHS's Agency for Healthcare Research and
Quality, and the Congressional Research Service. We also obtained
reports and interviewed officials from professional associations
involved in emergency preparedness and from policy research
organizations that had published assessments or evaluations of
public health and hospital preparedness programs. The professional
associations included
o American Hospital Association,
o Association of Professionals in Infection Control,
o Association of Public Health Laboratories,
o Association of State and Territorial Health Officials,
o National Association of County and City Health Officials,
o National Association of Public Hospitals, and
o The Joint Commission (formerly Joint Commission on Accreditation
of Healthcare Organizations).
The policy research organizations we contacted included
o Center for Studying Health System Changes,
o National Center for Disaster Preparedness at Columbia
University,
o Public Health Foundation,
o Rand Corporation,
o The Century Foundation, and
o Trust for America's Health.
We did not evaluate the actual performance measures adopted by CDC
or HRSA.
To determine how CDC and HRSA measure the progress of their
preparedness programs, we interviewed CDC and HRSA officials to
identify and document how they oversee and evaluate their
cooperative agreement programs. To identify procedures used for
reviewing recipient data and reporting results to applicable
program managers, we obtained and analyzed documents and
recipient-submitted progress reports from CDC and HRSA for program
year 2004 and the first half of program year 2005 and interviewed
CDC and HRSA project officers. Additionally, we analyzed documents
to identify procedures in place for providing feedback and
suggestions for program improvements to cooperative agreement
recipients. We also reviewed documents and conducted interviews
about the procedures used by project officers to provide
recipients with feedback on their performance, share expertise on
developing plans or conducting exercises, and disseminate
"promising practices" information. We did not examine the accuracy
or completeness of recipients' self-reported data in the progress
reports submitted to CDC or HRSA. We conducted our work from June
2006 to March 2007 in accordance with generally accepted
government auditing standards.
Enclosure III
Comments from Department of Health and Human Services
Enclosure IV
GAO Contact and Staff Acknowledgments
GAO Contact
Cynthia A. Bascetta at (202) 512-7101or bascettac@gao.gov
Acknowledgments
In addition to the contact name above, Karen Doran, Assistant
Director; La Sherri Bush; Jeffrey Mayhew; Roseanne Price; Lois
Shoemaker; and Cherie' Starck.
(290537)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
*** End of document. ***