Hospital Quality Data: HHS Should Specify Steps and Time Frame
for Using Information Technology to Collect and Submit Data
(25-APR-07, GAO-07-320).
Hospitals submit data in electronic form on a series of quality
measures to the Centers for Medicare & Medicaid Services (CMS)
and receive scores on their performance. Increasingly, the
clinical information from which hospitals derive the quality data
for CMS is stored in information technology (IT) systems. GAO was
asked to examine (1) hospital processes to collect and submit
quality data, (2) the extent to which IT facilitates hospitals'
collection and submission of quality data, and (3) whether CMS
has taken steps to promote the use of IT systems to facilitate
the collection and submission of hospital quality data. GAO
addressed these issues by conducting case studies of eight
hospitals with varying levels of IT development and interviewing
relevant officials at CMS and the Department of Health and Human
Services (HHS).
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-320
ACCNO: A68725
TITLE: Hospital Quality Data: HHS Should Specify Steps and Time
Frame for Using Information Technology to Collect and Submit Data
DATE: 04/25/2007
SUBJECT: Data collection
Data integrity
Data transmission
Electronic data processing
Health resources utilization
Hospitals
Information management
Information technology
Medical records
Medical information systems
Health information architecture
Annual Payment Update Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-320
* [1]Results in Brief
* [2]Background
* [3]Hospitals Use Six Basic Steps to Collect and Submit Quality
* [4]Hospitals Collect and Submit Quality Data by Completing Six
* [5]Two Most Complex Steps Were Locating Relevant Clinical Infor
* [6]Clinical Staff Abstract Quality Data at Most Hospitals
* [7]Adding Quality Measures Required a Proportionate Increase in
* [8]Hospitals Value and Use Quality Data
* [9]Existing IT Systems Can Help Hospitals Gather Some Quality D
* [10]Existing IT Systems Help Abstractors Obtain Information from
* [11]Full Automation of Quality Data Collection Is Not Imminent
* [12]CMS Sponsored Studies and Joined Broader HHS Initiatives to
* [13]CMS Sponsored Studies Examining Use of IT Systems for Collec
* [14]CMS Has Joined HHS's Efforts to Promote Greater Use of Healt
* [15]Conclusions
* [16]Recommendations for Executive Action
* [17]Agency Comments and Our Evaluation
* [18]Appendix I: Medicare Quality Measures Required for Full Annu
* [19]Appendix II: Data Elements Used to Calculate Hospital Perfor
* [20]Appendix III: Tables on Eight Case Study Hospitals
* [21]Appendix IV: Scope and Methodology
* [22]Appendix V: Comments from the Centers for Medicare & Medicai
* [23]Appendix VI: GAO Contact and Staff Acknowledgments
* [24]GAO Contact
* [25]Acknowledgments
* [26]Order by Mail or Phone
Report to the Committee on Finance, U.S. Senate
United States Government Accountability Office
GAO
April 2007
HOSPITAL QUALITY DATA
HHS Should Specify Steps and Time Frame for Using Information Technology
to Collect and Submit Data
GAO-07-320
Contents
Letter 1
Results in Brief 5
Background 6
Hospitals Use Six Basic Steps to Collect and Submit Quality Data, Two of
Which Involve Complex Abstraction by Hospital Staff 9
Existing IT Systems Can Help Hospitals Gather Some Quality Data but Are
Far from Enabling Automated Abstraction 22
CMS Sponsored Studies and Joined Broader HHS Initiatives to Promote Use of
IT for Quality Data Collection and Submission, but HHS Lacks Detailed
Plans, Milestones, and Time Frame 27
Conclusions 31
Recommendations for Executive Action 32
Agency Comments and Our Evaluation 32
Appendix I Medicare Quality Measures Required for Full Annual Payment
Update 35
Appendix II Data Elements Used to Calculate Hospital Performance on a
Heart Attack Quality Measure 36
Appendix III Tables on Eight Case Study Hospitals 38
Appendix IV Scope and Methodology 45
Appendix V Comments from the Centers for Medicare & Medicaid Services 50
Appendix VI GAO Contact and Staff Acknowledgments 53
Tables
Table 1: Case Study Hospital Characteristics 38
Table 2: How Case Study Hospital Officials Described the Steps Taken to
Complete Quality Data Collection and Submission 40
Table 3: Resources Used for Abstraction and Data Submission at Eight Case
Study Hospitals 42
Table 4: Electronic and Paper Records at Eight Case Study Hospitals 44
Figures
Figure 1: Six Basic Steps for Hospitals Collecting and Submitting Quality
Data 10
Figure 2: Example of the Process for Locating and Assessing Clinical
Information to Determine the Appropriate Value for One Data Element 18
Figure 3: Data Elements Used to Calculate Hospital Performance on the
Heart Attack Quality Measure That Asks Whether a Beta Blocker Was Given
When the Patient Arrived at the Hospital 36
Abbreviations
ACEI angiotensin-converting enzyme inhibitor
AHIC American Health Information Community
AHIMA American Health Information Management Association
AHRQ Agency for Healthcare Research and Quality Alliance
National Alliance for Health Information Technology
AMI acute myocardial infarction
APU Annual Payment Update
ARB angiotensin receptor blocker
CART CMS Abstraction & Reporting Tool
CCHIT Certification Commission for Health Information Technology
CHI Consolidated Healthcare Informatics CMS Centers for Medicare
& Medicaid Services
CPOE computerized physician order entry
DICOM Digital Imaging Communications in Medicine
DRA Deficit Reduction Act of 2005
FTE full-time equivalent
H&P history and physical
HCAHPS Hospital Consumer Assessment of Healthcare Providers and
Systems
HHS Department of Health and Human Services
IMSS Healthcare Information and Management Systems Society
HITSP Healthcare Information Technology Standards Panel
ICD-9 International Classification of Diseases, Ninth Revision
IFMC Iowa Foundation for Medical Care
IT information technology
JCAHO Joint Commission on Accreditation of Healthcare Organizations
LOINC Laboratory Logical Observation Identifier Name Codes
LPN licensed practical nurse
LVSD left ventricular systolic dysfunction
MAR medication administration record
MMA Medicare Prescription Drug, Improvement, and Modernization Act
MSA metropolitan statistical area
NCDCP National Council on Prescription Drug Programs
ONC Office of the National Coordinator for Health Information
Technology
POS provider of services
QIO quality improvement organization
RN registered nurse
SNOMED-CT Systematized Nomenclature of Medicine Clinical Terms
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
April 25, 2007
The Honorable Max Baucus
Chairman
The Honorable Charles E. Grassley
Ranking Minority Member
Committee on Finance
United States Senate
The Medicare Prescription Drug, Improvement, and Modernization Act of 2003
(MMA) created a financial incentive for hospitals to submit to the Centers
for Medicare & Medicaid Services (CMS) data that are used to calculate
hospital performance on measures of the quality of care provided.^1 CMS
established the Annual Payment Update (APU) program^2 to implement that
incentive. The APU program requires that participating hospitals submit
these quality data^3 on a quarterly basis in order to avoid a reduction in
their full Medicare payment update each fiscal year.^4 Although the APU
program was originally set to expire in 2007, the Deficit Reduction Act of
2005^5 (DRA) made the APU program permanent. The act also raised the
reduction^6 and required the Secretary of Health and Human Services (HHS)
to increase the number of measures for which hospitals participating in
the APU program would have to provide data in order to receive their full
Medicare payment update.^7 CMS plans to continue expanding the number of
required measures in future years.^8 Furthermore, DRA directed the
Secretary to develop a plan to implement a value-based purchasing program
for Medicare that beginning in fiscal year 2009 would adjust payments to
hospitals based on factors related to the quality of care they provide.
Such pay-for-performance programs are intended to strengthen the financial
incentives for hospitals to invest in quality improvement efforts.
^1See Pub. L. No. 108-173, S 501(b), 117 Stat. 2066, 2289-90.
^2Throughout this report, we refer to CMS's Reporting Hospital Quality
Data for the Annual Payment Update program as the "APU program."
^3Throughout this report, we refer to the data that hospitals submit to
CMS that the agency uses to calculate their performance on its quality
measures as "quality data."
^4Most acute care hospitals (i.e., those paid under the Medicare inpatient
prospective payment system) receive an annual payment update that
increases the standardized payment amount that Medicare pays them per
patient, based on projected increases in hospital operating expenses. For
fiscal year 2007, 3,319 hospitals received their full payment update,
about 95 percent of those eligible to participate in the APU program, and
the remaining 5 percent of eligible hospitals received a reduced annual
payment update. CMS posts on a public Web site the performance scores that
hospitals receive on quality measures derived from the data they submit.
^5See Pub. L. No. 109-171, S 5001(a), 120 Stat. 4, 28-29.
^6The magnitude of the reduction in the annual payment update for
hospitals not submitting the quality data rose from 0.4 percentage points
to 2 percentage points, starting in fiscal year 2007.
Each quality measure consists of a set of standardized data elements,
which define the specific data that hospitals need to submit to CMS.
Hospitals determine a value for each data element of a measure for
patients--Medicare and non-Medicare--who have a medical condition covered
by the APU program, that is, heart attack, heart failure, pneumonia, or
surgery. The values for the data elements consist of numerical data and
other administrative and clinical information that are obtained from the
medical records of the patients.^9 For example, there are 8 required
quality measures for the heart attack condition, one of which is whether a
beta blocker was given to the patient upon arrival at the hospital.^10
This single measure, in turn, consists of 11 data elements, including
administrative data elements, such as the patient's date of arrival at the
hospital, and clinical data elements, such as whether the patient received
a beta blocker within 24 hours after hospital arrival (see app. II). The
values entered for data elements are used to calculate hospital
performance on the 21 quality measures that are in effect as of fiscal
year 2007. For a hospital submitting data on all 21 measures, CMS receives
values for a total of 73 unique data elements. For heart attack measures
alone, the 8 measures utilize 35 of the 73 data elements. (Some data
elements are used in more than 1 measure. See app. I for the number of
data elements required for each measure.) Hospitals submit their quality
data electronically, over the Internet, to a clinical data warehouse
operated by a CMS contractor.
^7Initially, CMS designated 10 required quality measures under the APU
program that applied to patients treated for heart attacks, heart failure,
or pneumonia. In accordance with DRA, the Secretary increased the number
of required quality measures to 21. Nine of the new measures related to
the original three conditions, and 2 related to surgery, a new condition
for the program. (See app. I for the list of measures.)
^8CMS recently announced the addition of three surgery measures for fiscal
year 2008. In addition, to receive their full annual update payment,
hospitals will have to submit to CMS the responses provided by a random
sample of their discharged patients on a specified survey instrument--the
Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS)
survey--which is designed to obtain patient assessments of the care they
received. See 71 Fed. Reg. 67960, 68201-10 (Nov. 24, 2006).
^9Although the term value is often associated with numerical data, we use
it in this report to identify the information that hospitals submit to CMS
for a given data element. Some data elements call for numerical values,
and others call for nonnumerical values, such as Y or N for "yes" or "no."
^10Beta blockers are medications that decrease the rate and force of heart
contractions, which over time improves the heart's pumping ability.
Increasingly, the information in patients' medical records that provides
the basis for hospital quality data submissions may be stored and accessed
in electronic form in information technology (IT) systems. Currently, many
hospitals record and store such clinical information on patients in a
combination of paper and electronic systems. Over time, hospitals have
added new health IT systems to expand the amount of information that is
stored electronically. In 2005, the Secretary of HHS established the
American Health Information Community (AHIC) to advance the adoption of
electronic health records, after the President called in 2004 for the
widespread adoption of interoperable electronic health records within 10
years and appointed a National Coordinator for Health Information
Technology to promote that goal. On August 12, 2005, CMS issued a
regulation for the APU program that stated a goal of facilitating the use
of health IT by hospitals to make it easier for them to collect the
quality data from the medical record and submit them to CMS.^11 In the
preamble to the regulation, CMS said that it intended to begin working
toward modifying its requirements and mechanisms for accepting quality
data to allow hospitals to transfer their data directly from hospital IT
systems without having to first transfer the data into specially formatted
files as is currently required.
Because the vast majority of acute care hospitals treating Medicare
patients choose to submit quality data each quarter to CMS, rather than
accept a reduced annual payment update, you asked us to examine (1) how
hospitals collect and submit quality data for the Medicare hospital
quality measures, (2) the extent to which IT facilitates hospitals'
collection and submission of quality data for the Medicare hospital
quality measures, and (3) whether CMS has taken steps to promote the
development and use of IT systems that could facilitate the collection and
submission of hospital quality data.
^1170 Fed. Reg. 47278, 47420 (Aug. 12, 2005).
To assess how hospitals collect and submit quality data, we conducted case
studies of eight individual acute care hospitals to obtain information
about the processes they used to collect and submit the data.^12 The
hospitals varied on a number of standard hospital characteristics,
including size, urban/rural location, and teaching status (see app. III,
table 1). We visited each case study hospital, and we interviewed the
individuals responsible for collecting and submitting the quality data to
CMS, managers of the hospital's quality department, and hospital
administrators. To assess the extent to which IT facilitates hospitals'
collection and submission of quality data, we selected the case study
hospitals to include both hospitals with relatively well-developed IT
systems that supported electronic patient records and hospitals with
less-developed levels of IT, based on screening interviews done at the
time we selected the case study hospitals.^13 During our site visits, we
also interviewed IT staff involved in the process of collecting and
submitting the quality data. To assess whether CMS has taken steps to
promote the development and use of IT systems that could facilitate the
collection and submission of hospital quality data, we reviewed relevant
federal regulations, reports, and related documents and interviewed CMS
officials and CMS contractors, as well as officials in HHS's Office of the
National Coordinator for Health Information Technology (ONC). Because our
evidence is limited to the eight case studies, it does not offer a basis
for relating any differences we observed among these particular hospitals
to their differences on specific dimensions, such as size or teaching
status. Nor can we generalize from the group of eight as a whole to acute
care hospitals across the country. Where appropriate, we obtained relevant
information about these hospitals from CMS documents and databases;
however, most of our information for these case studies was reported by
hospital officials.^14 Furthermore, although we examined the processes
hospitals used to collect and submit quality data and the role that IT
plays in that process, we did not examine general IT adoption in the
hospital industry. We conducted our work from February 2006 to April 2007
in accordance with generally accepted government auditing standards. For a
complete description of our methodology, see appendix IV.
^12All eight hospitals participated in the APU program and had their
performance scores on the quality measures posted on CMS's Hospital
Compare Web site, www.hospitalcompare.hhs.gov.
^13Available research suggested that only a handful of hospitals had
developed a high level of IT implementation for purposes of documenting
patient care in electronic records, with the large majority having reached
just the initial stages of this process. See D. Garets and M. Davis,
"Electronic Medical Records vs. Electronic Health Records: Yes, There Is a
Difference" (Chicago, Ill.: HIMSS Analytics, LLC, updated Jan. 26, 2006).
^14Information obtained from CMS sources included the projected amount of
Medicare payments that the hospitals would lose if they had not submitted
quality data under the APU program and the number of patients for whom
they submitted data to CMS. See app. IV.
Results in Brief
The case study hospitals we visited used six steps to collect and submit
quality data, two of which (steps 2 and 3) involved complex
abstraction--the process of reviewing and assessing all relevant pieces of
information in a patient's medical record to determine the appropriate
value for each data element. Whether that patient information was recorded
electronically, on paper, or as a mix of both, the six steps were (1)
identify the patients, (2) locate information in their medical records,
(3) determine appropriate values for the data elements, (4) transmit the
quality data to CMS, (5) ensure that the quality data have been accepted
by CMS, and (6) supply copies of selected medical records to CMS to
validate the data. Several factors account for the complexity of the
abstraction process (steps 2 and 3), including the content and
organization of the medical record, the scope of information and clinical
judgment required for the data elements, and frequent changes by CMS in
its data specifications. Due in part to these complexities, most of our
case study hospitals relied on clinical staff to abstract the quality
data. Increases in the number of quality measures required by CMS led to
increased demands on clinical staff resources. Offsetting the demands
placed on clinical staff were the benefits that case study hospitals
reported finding in the quality data. For example, all the hospitals
reported having a process in place to track changes in their performance
over time and provide feedback to clinicians and reports to hospital
administrators and trustees.
Our case studies showed that existing IT systems can help hospitals gather
some quality data but are far from enabling hospitals to automate the
abstraction process. IT systems helped hospital staff abstract information
from patients' medical records, in particular by improving accessibility
to and legibility of the medical record and by enabling hospitals to
incorporate CMS's required data elements into the medical record. The
limitations reported by officials in the case study hospitals included
having a mix of paper and electronic records, which required staff to
check multiple places to get the needed information; the prevalence of
data recorded as unstructured narrative or text, which made locating the
information time-consuming because it was not in a prescribed place in the
record; and the inability of some IT systems to access related data stored
in another IT system in the same hospital, which required hospital staff
to access each IT system separately to obtain related pieces of
information. While hospital officials expected the scope and functionality
of their IT systems to increase over time, they projected that this
process would occur incrementally over a period of years.
CMS has sponsored studies and joined HHS initiatives to examine and
promote the current and potential use of hospital IT systems to facilitate
the collection and submission of quality data, but HHS lacks detailed
plans, including milestones and a time frame against which to track its
progress. CMS sponsored two studies that examined the use of hospital IT
systems for quality data collection and submission. Promoting the use of
health IT for quality data collection is also 1 of 14 objectives that HHS
has identified in its broader effort to encourage the development and
nationwide implementation of interoperable IT in health care. CMS has
joined this broader effort by HHS, as well as the Quality Workgroup that
AHIC created in August 2006 to specify how IT could capture, aggregate,
and report inpatient and outpatient quality data. Through its
representation in AHIC and the Quality Workgroup, CMS has participated in
decisions about the specific focus areas to be examined through contracts
with nongovernmental entities. These contracts currently address the use
of health IT for a range of purposes, which may also include quality data
collection and submission in the near future. However, HHS has identified
no detailed plans, milestones, or time frames for either its broad effort
to encourage IT in health care nationwide or its specific objective to
promote the use of health IT for quality data collection.
To support the expansion of quality measures for the APU program, we
recommend that the Secretary of HHS identify the specific steps that the
department plans to take to promote the use of health IT for the
collection and submission of data for CMS's hospital quality measures and
inform interested parties on those steps and the expected time frame,
including milestones for completing them. In commenting on a draft of this
report on behalf of HHS, CMS expressed its appreciation of our thorough
analysis of the processes that hospitals use to report quality data and
the role that IT systems can play in that reporting, and it concurred with
our two recommendations.
Background
The quality data submitted by hospitals are collected from the medical
records of patients admitted to the hospital. Hospital patient medical
records contain many different types of information, which are organized
into different sections. Frequently found examples of these sections
include
o the face sheet, which summarizes basic demographic and billing
data, including diagnostic codes;
o history and physicals (H&P), which record both patient medical
history and physician assessments;
o physician orders, which show what medications, tests, and
procedures were ordered by a physician;
o medication administration records (MAR), which show that a
specific medication was given to a patient, when it was given, and
the dosage;
o laboratory reports, radiology reports, and test results, such as
an echocardiogram reading;
o progress notes, in which physicians, nurses, and other
clinicians record information chronologically on patient status
and response to treatments during the patient's hospital stay;
o operative reports for surgery patients;
o physician and nursing notes for patients treated in the
emergency department; and
o discharge summaries, in which a physician summarizes the
patient's hospital stay and records prescriptions and instructions
to be given to the patient at discharge.
Hospitals have discretion to determine the structure of their
patient medical records, as well as to set general policies
stating what, where, and how specific information should be
recorded by clinicians. To guide the hospital staff in the
abstraction process--that is, in finding and properly assessing
the information in the patient's medical record needed to fill in
the values for the data elements--CMS and the Joint Commission^15
have jointly issued a Specifications Manual.^16 It contains
detailed specifications that define the data elements for which
the hospital staff need to collect information and determine
values and the correct interpretation of those data elements. The
Joint Commission also requires hospitals to submit the same data
that they submit to CMS for the APU program (and some additional
data) to receive Joint Commission accreditation.
In many hospitals, information in a patient's medical record is
recorded and stored in a combination of paper and electronic
systems. Patient medical records that clinicians record on paper
may be stored in a folder in the hospital's medical record
department and contain all the different forms, reports, and notes
prepared by different individuals or by different departments
during the patient's stay. Depending on the length of the
patient's hospital stay and the complexity of the care, an
individual patient medical record can amount to hundreds of
pages.^17 For information stored electronically, clinicians may
enter information directly into the electronic record themselves,
as they do for paper records, or they may dictate their notes to
be transcribed and added to the electronic record later.
Information may also be recorded on paper and then scanned into
the patient's electronic record. For example, if a patient is
transferred from another hospital, the paper documents from the
transferring hospital may be scanned into the patient's electronic
record.
The patient medical information that hospitals store
electronically, rather than on paper, typically resides in
multiple health IT systems. One set of IT systems usually handles
administrative tasks such as patient registration and billing.
Hospitals acquire other IT systems to record laboratory test
results, to store digital radiological images, to process
physician orders for medications, and to record notes written by
physicians and nurses. Hospitals frequently build their health IT
capabilities incrementally by adding new health IT systems over
time.^18 If the systems that hospitals purchase come from
different companies, they are likely to be based on varying
standards for how the information is stored and exchanged
electronically. As a result, even in a single hospital, it can be
difficult to access from one IT system clinical data stored in a
different health IT system.
One of the main objectives of ONC is to overcome the problem of
multiple health IT systems, within and across health care
providers, that store and exchange information according to
varying standards. The mission of ONC is to promote the
development and nationwide implementation of interoperable health
IT in both the public and the private sectors in order to reduce
medical errors, improve quality of care, and enhance the
efficiency of health care.^19 Health IT is interoperable when
systems are able to exchange data accurately, effectively,
securely, and consistently with different IT systems, software
applications, and networks in such a way that the clinical or
operational purposes and meaning of the data are preserved and
unaltered.
Hospitals Use Six Basic Steps to Collect and Submit Quality Data,
Two of Which Involve Complex Abstraction by Hospital Staff
The case study hospitals we visited used six steps to collect and
submit quality data, two of which involved complex
abstraction--the process of reviewing and assessing all relevant
pieces of information in a patient's medical record to determine
the appropriate value for each data element. Factors accounting
for the complexity of the abstraction process included the content
and organization of the medical record, the scope of information
required for the data elements, and frequent changes by CMS in its
data specifications. Due in part to these complexities, most of
our case study hospitals relied on clinical staff to abstract the
quality data. Increases in the number of required quality measures
led to increased demands on clinical staff resources. However, all
case study hospitals reported finding benefits in the quality data
that helped to offset the demands placed on clinical staff.
Hospitals Collect and Submit Quality Data by Completing Six Basic
Steps
We found that whether patient information was recorded
electronically, on paper, or as a mix of both, all the case study
hospitals collected and submitted their quality data by carrying
out six sequential steps (see fig. 1). These steps started with
identifying the patients for whom the hospitals needed to provide
quality data to CMS and continued through the process of examining
each patient's medical record, one after the other, to find the
information needed to determine the appropriate values for each of
the required data elements for that patient. Then, for each
patient, those values were entered by computer into an electronic
form or template listing each of the data elements for that
condition. These forms were provided by the data vendor with which
the hospital had contracted to transmit its quality data to CMS.
The vendors also assisted the hospitals in checking that the data
were successfully received by CMS. Finally, the hospitals sent
copies of the medical records of a selected sample of patients to
a CMS contractor that used those records to validate the accuracy
of the quality data submitted by the hospital.
Figure 1: Six Basic Steps for Hospitals Collecting and Submitting
Quality Data
Note: Patient information may be obtained from either electronic
or paper records.
Specifically, the six steps, which are summarized for each case
study hospital in appendix III, table 2, were as follows:
Step 1: Identify patients--The first step was to identify the
patients for whom the hospitals needed to submit quality data to
CMS. Staff at three case study hospitals identified these patients
using information on the patient's principal diagnosis, or
principal procedure in the case of surgery patients, obtained from
the hospital's billing data.^20 Five case study hospitals had
their data vendor use the hospital's billing data to identify the
eligible patients for them. Every month, all eight hospitals that
we visited identified patients discharged in the prior month for
whom quality data should be collected. The hospitals identified
all patients retrospectively for quality data collection because
hospitals have to wait until a patient is discharged to determine
the principal diagnosis.^21
CMS permits hospitals to reduce their data collection effort by
providing quality data for a representative sample of patients
when the total number of patients treated for a particular
condition exceeds a certain threshold.^22 Five case study
hospitals drew samples for at least one condition. The data vendor
performed this task for four of those case study hospitals, and
assisted the hospital in performing this task for the fifth
hospital.
Only one of the case study hospitals reported using nonbilling
data sources to check the accuracy of the lists of patients
selected for quality data collection that the hospitals drew from
their billing data (see app. III, table 3). Several stated that
they occasionally noted discrepancies, such as patients selected
for heart attack measures who, upon review of their medical
record, should not have had that as their principal diagnosis.
However, the hospital officials we interviewed told us that
discrepancies of this sort were likely to be minor. Officials at
three hospitals noted that hospitals generally have periodic
routine audits conducted of the coding practices of their medical
records departments, which would include the accuracy of the
principal diagnoses and procedures.
Step 2: Locate information in the medical record--Steps 2 and 3
were in practice closely linked in our case study hospitals.
Abstractors^23 at the eight case study hospitals examined each
selected patient's medical record, looking for all of the discrete
pieces of information that, taken together, would determine what
they would decide--in step 3--was the correct value for each of
the data elements. For some data elements, there was a one-to-one
correspondence between the piece of information in the medical
record and the value to be entered. Typical examples included a
patient's date of birth and the name of a medication administered
to the patient. For other data elements, the abstractors had to
check for the presence or absence of multiple pieces of
information in different parts of the medical record to determine
the correct value for that data element. For example, to determine
if the patient did, or did not, have a contraindication for
aspirin, abstractors looked in different parts of the medical
record for potential contraindications, such as the presence of
internal bleeding, allergies, or prescriptions for certain other
medications such as Coumadin.^24
In order for abstractors to find information in the patient's
medical record, it had to be recorded properly by the clinicians
providing the patient's care. Officials at all eight case study
hospitals described efforts designed to educate physicians and
nurses about the specific data elements for which they needed to
provide information in each patient's medical record. The hospital
officials were particularly concerned that the clinicians not
undermine the hospital's performance on the quality measures by
inadequately documenting what they had done and the reasons why.
For example, one heart failure measure tracks whether a patient
received each of six specific instructions at the time of
discharge, but unless information was explicitly recorded in a
heart failure patient's medical record for each of the six data
elements, that patient was counted by CMS as one who had not
received all pertinent discharge instructions and therefore did
not meet that quality measure.^25 This particular measure was
cited by officials at several hospitals as one that required a
higher level of documentation than had previously been the norm at
their hospital.
Step 3: Determine appropriate data element values--Once
abstractors had located all the relevant pieces of information
pertaining to a given data element, they had to put those pieces
together to arrive at the appropriate value for the data element.
The relevance of that information was defined by the detailed
instructions provided by the hospitals' vendors, as well as the
Specifications Manual jointly issued by CMS and the Joint
Commission that serves as the basis for the vendor instructions.
The Specifications Manual sets out the decision rules for choosing
among the allowable values for each data element. It also
identifies which parts of the patient's medical record may or may
not provide the required information, and often lists specific
terms or descriptions that, if recorded in the patient's medical
record, would indicate the appropriate value for a given data
element. In addition, the Specifications Manual provides
abstractors with guidance on how to interpret conflicting
information in the medical record, such as a note from one
clinician that the patient is not a smoker and a note elsewhere in
the record from another clinician that the patient does smoke. To
help keep track of multiple pieces of information, many
abstractors reported that they first filled in the data element
values on a paper copy of the abstraction form provided by the
data vendor. In this way, they could write notes in the margin to
document how they came to their conclusions.
Step 4: Transmit data to CMS--In order for the quality data to be
accepted by the clinical data warehouse, they must pass a battery
of edit checks that look for missing, invalid, or improperly
formatted data element entries.^26 All the case study hospitals
contracted with data vendors to submit their quality data to CMS.
They did so, in part, because all of the hospitals submitted the
same data to the Joint Commission, and it requires hospitals to
submit their quality data through data vendors that meet the Joint
Commission's requirements. The additional cost to the hospitals to
have the data vendors also submit their quality data to CMS was
generally minimal (see app. III, table 3).
All of the case study hospitals submitted their data to the data
vendor by filling in values for the required data elements on an
electronic version of the vendor's abstraction form.^27 Many
abstractors did this for a batch of patient records at a time,
working from paper copies of the form that they had filled in
previously. Some abstractors entered the data online at the same
time that they reviewed the patient's medical records. In other
cases, someone other than the abstractor who filled in the paper
form used the completed form to enter the data on a computer.
Step 5: Ensure data have been accepted by CMS--The case study
hospitals varied in the extent to which they actively monitored
the acceptance of their quality data into CMS's clinical data
warehouse. After the data vendors submitted the quality data
electronically, they and the hospitals could download reports from
the clinical data warehouse indicating whether the submitted data
had passed the screening edits for proper formatting and valid
entries. The hospitals could use these reports to detect data
entry errors and make corrections prior to CMS's data submission
deadline. Three case study hospitals shared this task with their
data vendors, three hospitals left it for their data vendors to
handle, and two hospitals received and responded to reports on
data edit checks produced by their data vendors, rather than
reviewing the CMS reports. Approximately 2 months after hospitals
submitted their quality data, CMS released reports to the
hospitals showing their performance scores on the quality measures
before posting the results on its public Web site.
Step 6: Supply copies of selected medical records--CMS has put in
place a data validation process to ensure the accuracy of hospital
quality data submissions. It requires hospitals to supply a CMS
contractor with paper copies of the complete medical record for
five patients selected by CMS each quarter.^28 Officials at five
hospitals noted that they check to make sure that all parts of the
medical records that they used to abstract the data originally are
included in the package shipped to the CMS contractor. Most of the
case study hospitals relied on CMS's data validation to ensure the
accuracy of their abstractions. However, two hospitals reported
that they also routinely draw their own sample of cases, which are
abstracted a second time by a different abstractor in the
hospital, followed by a comparison of the two sets of results (see
app. III, table 3).
Two Most Complex Steps Were Locating Relevant Clinical Information
and Determining Appropriate Values for Data Elements
The description by hospital officials of the processes they used
to collect and submit quality data indicated that locating the
relevant clinical information and determining appropriate values
for the data elements (steps 2 and 3) were the most complex steps
of the six identified, due to several factors. These included the
content and organization of the medical record, the scope of the
information encompassed by the data elements, and frequent changes
in data specifications.
The first complicating factor related to the medical record was
that the information abstractors needed to determine the correct
data element values for a given patient was generally located in
many different sections of the patient's medical record. These
included documents completed for admission to the hospital,
emergency department documents, laboratory and test results,
operating room notes, medication administration records, nursing
notes, and physician-generated documents such as history and
physicals, progress notes and consults, orders for medications and
tests, and discharge summaries. In addition, the abstractors may
have had to look at documents that came from other providers if
the patient was transferred to the hospital. Much of the clinical
information needed was found in the sections of the medical record
prepared by clinicians. Often the information in question, such as
contraindications for aspirin or beta blockers, could be found in
any of a number of places in the medical record where clinicians
made entries. As a result, abstractors frequently had to read
through multiple parts of the record to find the information
needed to determine the correct value for just one data element.
At two case study hospitals, abstractors said that they routinely
read each patient's entire medical record.
Experienced abstractors often knew where they were most likely to
find particular pieces of information. They nevertheless also had
to check for potentially contradictory information in different
parts of the medical record. For example, as noted, patients may
have provided varying responses about their smoking history to
different clinicians. If any of these responses indicated that the
patient had smoked cigarettes in the last 12 months, the patient
was considered to be a smoker according to CMS's data
specifications. Another example concerns the possibility that a
heart attack or heart failure patient may have had multiple
echocardiogram results recorded in different parts of the medical
record. Abstractors needed to find all such results in order to
apply the rules stated in the Specifications Manual for
identifying which result to use in deciding whether the patient
had left ventricular systolic dysfunction (LVSD). This data
element is used for the quality measure assessing whether an
angiotensin-converting enzyme inhibitor (ACEI) or angiotensin
receptor blocker (ARB) was prescribed for LVSD at discharge.^29
The second factor was related to the scope of the information
required for certain data elements. Some of the data elements that
the abstractors had to fill in represented a composite of related
data and clinical judgment applied by the abstractor, not just a
single discrete piece of information. Such composite data elements
typically were governed by complicated rules for determining the
clinical appropriateness of a specific treatment for a given
patient. For example, the data element for contraindications for
both ACEIs and ARBs at discharge requires abstractors to check for
the presence and assess the severity of any of a range of clinical
conditions that would make the use of either ACEIs or ARBs
inappropriate for that patient.^30 (See fig. 2.) These conditions
may appear at any time during the patient's hospital stay and so
could appear at any of several places in the medical record.
Abstractors must also look for evidence in the record from a
physician^31 linking a decision not to prescribe these drugs to
one or more of those conditions.
^15The Joint Commission (previously the Joint Commission on Accreditation
of Healthcare Organizations or JCAHO) is a private, not-for-profit
organization that accredits approximately 82 percent of hospitals that
participate in Medicare.
^16Current and past versions of the Specifications Manual for National
Hospital Quality Measures are available at www.qualitynet.org.
^17Generally, the records for multiple hospital admissions for the same
patient are stored together, creating an even more voluminous collection
of documents in a single patient record.
^18For example, many hospitals have IT systems to record laboratory test
results electronically, but fewer have added IT systems to record
radiology results electronically. D. Blumenthal et al., Health Information
Technology in the United States: The Information Base for Progress
(Princeton, N.J.: Robert Wood Johnson Foundation, 2006), 3:26.
^19Exec. Order No. 13335, 69 Fed. Reg. 24059 (Apr. 27, 2004).
^20The CMS data specifications list the International Classification of
Diseases, Ninth Revision (ICD-9) diagnostic codes that make a patient
eligible for quality data collection. The other factor determining basic
patient eligibility is age at the time of admission, derived from the
patient's admission date and date of birth, also available from billing
data. For pneumonia patients, secondary diagnoses may also affect
eligibility.
^21Medicare bases its payments for inpatient care on principal diagnosis,
which it defines as the condition established after study to be chiefly
responsible for the admission. Principal diagnosis may also affect
determination of the principal procedure, if several procedures were
performed.
^22CMS's specific sampling requirements vary by medical condition. For
example, hospitals that have more than 78 heart attack patients in a given
quarter can submit quality data for a random sample of those patients, as
long as their sample includes a minimum of 78 patients and applies a
sampling rate of 20 percent up to a maximum required sample of 311.
^23Throughout this report, we use the term abstractor to indicate hospital
staff who are trained to follow a detailed protocol in order to extract
specified information in a consistent fashion from the medical records of
multiple patients.
^24Coumadin is a medication that acts as an anticoagulant. It is used to
prevent and treat harmful blood clots that increase the risk of heart
attack and stroke.
^25These discharge instructions are supposed to cover recommended level of
activity, diet, follow-up care after discharge, medications, weight
monitoring, and what to do if symptoms worsen. The abstractor fills in a
value for six separate data elements, one for each of the six specific
instructions. The value is either a yes, written discharge instructions
addressing the specified activity were provided, or no, instructions
addressing the specified activity were not provided or unable to determine
from medical record documentation. Anything less than a yes on all six
data elements leads to a negative score on this quality measure for that
patient.
^26This process is described in GAO, Hospital Quality Data: CMS Needs More
Rigorous Methods to Ensure Reliability of Publicly Released Data,
[27]GAO-06-54 (Washington, D.C.: Jan. 31, 2006), 14.
^27Abstractors at many hospitals entered the data through an online
connection to the vendor. Other hospitals submitted their quality data in
the form of electronic files.
^28CMS draws a sample of five patient records from among those submitted
by each hospital that provided data on six or more patients to the
clinical data warehouse in a given quarter. CMS then tells each hospital
which patients need to have their records copied, or printed out in the
case of electronic records, and shipped to its contractor. The contractor
then abstracts values for clinical data elements from those records,
following the CMS/Joint Commission Specifications Manual, and the results
are compared--data element by data element--with the values originally
abstracted and submitted by the hospital. If hospitals do not achieve a
match of at least 80 percent of their data element values with those of
the CMS contractor, following the outcome of any appeals, the hospitals
will not receive a full payment update from Medicare for the subsequent
fiscal year. See [28]GAO-06-54 , 14-16.
^29ACEIs and ARBs are two classes of drugs that have been shown in
clinical trials to reduce mortality and morbidity in patients with LVSD.
^30ACEIs or ARBs may be contraindicated if the patient is known to be
allergic to such drugs or suffers from certain medical conditions, such as
moderate to severe aortic stenosis or renal disease.
^31A nurse practitioner or physician assistant may also provide this
documentation in the patient's medical record.
Figure 2: Example of the Process for Locating and Assessing Clinical
Information to Determine the Appropriate Value for One Data Element
Note: In this illustrative case, adapted from CMS training materials, an
abstractor would find that the patient was given an ACEI, Zestril, in the
emergency department (see MAR,1/30), but because of its apparent effect on
the patient's pulse and blood pressure (see Progress Notes, 01/31), it was
not continued during the hospital stay (see Progress Notes, 02/03) and no
ACEI was prescribed at discharge (see Discharge Summary). However, there
is no mention in the patient's record of ARBs or aortic stenosis. The
arrows point to some of the key pieces of information an abstractor would
take note of in determining that the appropriate value for this data
element was "N" for "no."
The third factor is the necessity abstractors at the case study hospitals
faced to adjust to frequent changes in the data specifications set by CMS.
Since CMS first released its detailed data specifications jointly with the
Joint Commission in September 2004, it has issued seven new versions of
the Specifications Manual.^32 Therefore, from fall 2004 through summer
2006, roughly every 3 months hospital abstractors have had to stop and
take note of what had changed in the data specifications and revamp their
quality data collection procedures accordingly. Some of these changes
reflected modifications in the quality measures themselves, such as the
addition of ARBs for treatment of LVSD. Other changes revised or expanded
the guidance provided to abstractors, often in response to questions
submitted by hospitals to CMS. CMS recently changed its schedule for
issuing revisions to its data specifications from every 3 months to every
6 months, but that change had not yet affected the interval between new
revisions issued to hospitals at the time of our case study site visits.
Clinical Staff Abstract Quality Data at Most Hospitals
Case study hospitals typically used registered nurses (RN), often
exclusively, to abstract quality data for the CMS quality measures (see
app. III, table 3). One hospital relied on a highly experienced licensed
practical nurse, and two case study hospitals used a mix of RNs and
nonclinical staff. Officials at one hospital noted that RNs were familiar
with both the nomenclature and the structure of the hospital's medical
records and they could more readily interact with the physicians and
nurses providing the care about documentation issues. Even when using RNs,
all but three of the case study hospitals had each abstractor focus on one
or two medical conditions with which they had expertise.
Four hospitals had tried using nonclinical staff, most often trained as
medical record coders, to abstract the quality data. Officials at one of
these hospitals reported that this approach posed challenges. They said
that it was difficult for nonclinical staff to learn all that they needed
to know to abstract quality data effectively, especially with the constant
changes being made to the data specifications. At the second hospital,
officials reported that using nonclinical staff for abstraction did not
work at all and they switched to using clinically trained staff. At the
third hospital, the chief clinician leading the quality team stated that
the hospital's nonclinical abstractors worked well enough when clinically
trained colleagues were available to answer their questions. Officials at
the fourth hospital cited no concerns about using staff who were not RNs
to abstract quality data, but they subsequently hired an RN to abstract
patient records for two of the four conditions.
^32This tally of revisions only takes account of new versions of the
Specifications Manual. Recently, the release of new versions has been
followed by multiple addendums to the revision, weeks or months later, to
provide further modifications and clarifications.
Case study hospitals drew on a mix of existing and new staff resources to
handle the collection and submission of quality data to CMS. In two
hospitals, new staff had been hired specifically to collect quality data
for the Joint Commission and CMS. In other hospitals, quality data
collection was assigned to staff already employed in the hospital's
quality management department or performing other functions.
Adding Quality Measures Required a Proportionate Increase in Staff Resources
All the case study hospitals found that, over time, they had to increase
the amount of staff resources devoted to abstracting quality data for the
CMS quality measures, most notably as the number of measures on which they
were submitting data expanded. Officials at the case study hospitals
generally reported that the amount of staff time required for abstraction
increased proportionately with the number of conditions for which they
reported quality data. The hospitals had all begun to report most recently
on the surgical quality measures. They found that the staff hours needed
for this new set of quality measures were directly related to the number
of patient records to be abstracted and the number of data elements
collected. In other words, they found no "economies of scale" as they
expanded the scope of quality data abstraction. At the time of our site
visits, four hospitals continued to draw on existing staff resources,
while others had hired additional staff. Hospital officials estimated that
the amount of staff resources devoted to abstracting data for the CMS
quality measures ranged from 0.7 to 2.5 full-time equivalents (FTE) (app.
III, table 3).^33
Hospitals Value and Use Quality Data
Hospital officials reported that the demands that quality data collection
and submission placed on their clinical staff resources were offset by the
benefits that they derived from the resulting information on their
clinical performance. Each one had a process for tracking changes in their
performance over time. Based on those results, they provided feedback to
individual clinicians and reports to hospital administrators and trustees.
Because they perceived feedback to clinicians to be much more effective
when provided as soon as possible, several of the case study hospitals
found ways to calculate their performance on the quality measures
themselves, often on a monthly basis, rather than wait for CMS to report
their results for the quarter.
Officials at all eight case study hospitals pointed to specific changes
they had made in their internal procedures designed to improve their
performance on one or more quality measures. Most of the case study
hospitals developed "standing order sets" for particular diagnoses. Such
order sets provide a mechanism for standardizing both the care provided
and the documentation of that care, in such areas as prescribing beta
blockers and aspirin on arrival and at discharge for heart attack
patients. Another common example involved prompting physicians to
administer pneumococcal vaccinations to pneumonia patients. However, at
most of the case study hospitals, use of many standing order sets was
optional for physicians, and hospital officials reported widely varying
rates of physician use, from close to 100 percent of physicians at one
hospital using its order set for heart attack patients to just a few
physicians using any order sets in another hospital.
Case study hospitals also responded to the information generated from
their quality data by adjusting their treatment protocols, especially for
patients treated in their emergency departments. For example, five
hospitals developed or elaborated on procedural checklists for emergency
department nurses treating pneumonia patients. The objective of these
changes was to more quickly identify pneumonia patients when they arrived
at the emergency department and then expeditiously perform required blood
tests so that the patients would score positively for the quality measure
on receiving antibiotics within 4 hours of arrival at the hospital. Three
hospitals strengthened their procedures to identify smokers and make sure
that they received appropriate counseling.
^33These represent the FTEs devoted specifically to quality data
collection and submission. Hospital officials noted that additional FTEs
were involved in analyzing the hospital's performance on the quality
measures and achieving improvements through changes in clinical process
and educational efforts with the hospital's clinicians.
Hospital officials noted that they provided quality of care data to
entities other than CMS and the Joint Commission, such as state
governments and private insurers, but for the most part they reported that
the CMS quality measures had two advantages. First, the CMS quality
measures enabled hospitals to benchmark their performance against the
performances of virtually every other hospital in the country. Second,
officials at two hospitals noted that the CMS measures were based on
clinical information obtained from patient medical records and therefore
had greater validity as measures of quality of care than measures based
solely on administrative data.^34 Many hospital officials said that they
wished that state governments and other entities collecting quality data
would accept the CMS quality measures instead of requiring related quality
data based on different definitions and patient populations. Hospital
officials in two states reported some movement in that direction.
Existing IT Systems Can Help Hospitals Gather Some Quality Data but Are Far from
Enabling Automated Abstraction
In the case studies, existing IT systems helped hospital abstractors to
complete their work more quickly, but the limitations of those IT systems
meant that trained staff still had to examine the entire patient medical
record and manually abstract the quality data submitted to CMS. IT systems
helped abstractors obtain information from patients' medical records, in
particular by improving their accessibility and legibility, and by
enabling hospitals to incorporate CMS's required data elements into those
medical records. The challenges reported by hospital officials included
having a mix of paper and electronic records, which required abstractors
to check multiple places to get the needed information; the prevalence of
unstructured data, which made locating the information time-consuming
because it was not in a prescribed place in the record; and the presence
of multiple IT systems that did not share data, which required abstractors
to separately access each IT system for related pieces of information that
were in different parts of the medical record. While hospital officials
expected the scope and functionality of their IT systems to increase over
time, they projected that this would occur incrementally over a period of
years.^35
34For example, hospital officials identified several private insurers that
assess quality based on patient outcomes derived from administrative data,
such as hospital billing data.
Existing IT Systems Help Abstractors Obtain Information from Medical Records but
Have Notable Limitations
Hospitals found that their existing IT systems could facilitate the
collection of quality data, but that there were limits on the advantages
that the systems could provide. IT systems, and the electronic records
they support, offered hospitals two key benefits: (1) improving
accessibility to and legibility of the medical record, and (2)
facilitating the incorporation of CMS's required data elements into the
medical record.
Many hospital abstractors noted that existing electronic records helped
quality data collection by improving accessibility and legibility of
patient records. In general, paper records were less accessible than
electronic records because it took time to find them or to have them
transported if hospitals had stored them in a remote location after the
patients were discharged. Also, paper records were more likely to be
missing or in use by someone else. However, in one case study hospital, an
abstractor noted difficulties in gaining access to a computer terminal to
view electronic medical records. Many abstractors noted improvements in
legibility as a fundamental benefit of electronic records. This advantage
applied in particular to the many sections of the medical record that
consisted of handwritten text, including history and physicals, progress
notes, medication administration records, and discharge summaries.
Some hospitals have used their existing IT systems to facilitate the
abstraction of information by designing a number of discrete data fields
that match CMS's data elements. For example, two hospitals incorporated
prompts for pneumococcal vaccination in their electronic medication
ordering system. These prompts not only reminded physicians to order the
vaccination (if the patient was not already vaccinated) but also helped to
insure documentation of the patient's vaccination status. One hospital
developed a special electronic discharge program for heart attack and
heart failure patients that had data elements for the quality measures
built into it. Another hospital built a prompt into its electronically
generated discharge instructions to instruct patients to measure their
weight daily. This enabled the hospital to document more consistently one
of the specific instructions that heart failure patients are supposed to
receive on discharge but that physicians and nurses tended to overlook in
their documentation.
^35For example, one case study hospital began several years ago to use an
IT system to record nursing notes. Hospital officials told us that they
planned to initiate a pilot test of a new component of that IT system that
would provide computerized physician order entry (CPOE) beginning in
October 2006. The officials reported that they would assess their
experience with the pilot test in one hospital unit before deciding how
quickly to expand it to the rest of the hospital. They said that they were
planning ultimately to store all patient medical records in electronic
form but there was no fixed timeline for that objective. The timing would
depend, they said, on the success of their CPOE pilot test.
The limitations that hospital officials reported in using existing IT
systems to collect quality data stemmed from having a mix of paper and
electronic systems; the prevalence of data recorded in IT systems as
unstructured paragraphs of narrative or text, as opposed to discrete data
fields reserved for specific pieces of information; and the inability of
some IT systems to access related data stored on another IT system in the
same hospital. Because all but one of the case study hospitals stored
clinical records in a mix of paper and electronic systems, abstractors
generally had to consult both paper and electronic records to obtain all
needed information. What was recorded on paper and what was recorded
electronically varied from hospital to hospital (see app. III, table 4).
However, admissions and billing data were electronic at all the case study
hospitals. Billing data include principal diagnosis and birth date, which
are among the CMS-required data elements. With regard to clinical data,
all case study hospitals had test results, such as echocardiogram
readings, in an electronic form. In contrast, nurse progress notes were
least likely to be in electronic form at the case study hospitals.
Moreover, it was not uncommon for a hospital to have the same type of
clinical documentation stored partly in electronic form and partly on
paper. For example, five of the eight case study hospitals had a mix of
paper and electronic physician notes, reflecting the differing personal
preferences of the physicians. Discharge summaries and medication
administration records, on the other hand, tended to be either paper or
electronic at a given hospital.
Many of the data in existing IT systems were recorded in unstructured
formats--that is, as paragraphs of narrative or other text, rather than in
data fields designated to contain specific pieces of information--which
created problems in locating the needed information. For example,
physician notes and discharge summaries were often dictated and
transcribed. Abstractors typically read through the entire electronic
document to make sure that they had found all potentially relevant
references, such as for possible contraindications for a beta blocker or
an ACEI. By contrast, some of the data in existing IT systems were in
structured data fields so that specific information could be found in a
prescribed place in the record. One common example was a list of
medication allergies, which abstractors used to quickly check for certain
drug contraindications. However, officials at several hospitals said that
developing and implementing structured data fields were labor intensive,
both in terms of programming and in terms of educating clinical staff in
their use. That is why many of the data stored in electronic records at
the case study hospitals remained in unstructured formats.
Another limitation with existing IT systems was the inability of some
systems to access related data stored on another IT system in the same
hospital. This situation affected six of the eight case study hospitals to
some degree. For example, one hospital had an IT system in the emergency
department and an IT system on the inpatient floors, but the two systems
were independent and the information in one was not linked to the
information in the other. Abstractors had to access each IT system
separately to obtain related pieces of information, which made abstraction
more complicated and time-consuming.
Existing IT systems helped hospital abstractors to complete their work
more quickly, but the limitations of those IT systems meant that, for the
most part, the nature of their work remained the same. Existing IT systems
enabled abstractors at several hospitals to more quickly locate the
clinical information needed to determine the appropriate values for at
least some of the data elements that the hospitals submitted to CMS. Where
hospitals designed a discrete data field in their IT systems to match a
specific CMS data element, abstractors could simply transcribe that value
into the data vendor's abstraction form. However, in all the case study
hospitals there remained a large number of data elements for which there
was no discrete data field in a patient's electronic record that could
provide the required value for that data element. As a result, trained
staff still had to examine the medical record as a whole and manually
abstract the quality data submitted to CMS, whether the information in the
medical record was recorded electronically or on paper.^36
36Some of the data vendors captured values for certain data elements from
the hospital's billing data, such as the patient's birth date and
discharge status, and entered those values in the abstraction form that it
provided to the hospital for that patient. The abstractors were supposed
to make sure those entries were consistent with the information found in
the patient's medical record.
Full Automation of Quality Data Collection Is Not Imminent
All the case study hospitals were working to expand the scope and
functionality of their IT systems, but this expansion was generally
projected to occur incrementally over a period of years. Hospital
officials noted that with wider use of IT systems, the advantages of these
systems--including accessibility, legibility, and the use of discrete data
fields--would apply to a larger proportion of the clinical records that
abstractors have to search. As the case study hospitals continue to bring
more of their clinical documentation into IT systems, and to link separate
systems within their hospital so that data in one system can be accessed
from another, it should reduce the time required to collect quality data.
However, most officials at the case study hospitals viewed full-scale
automation of quality data collection and submission through
implementation of IT systems as, at best, a long-term prospect. They
pointed to a number of challenges that hospitals would have to overcome
before they could use IT systems to achieve full-scale automation of
quality data collection and submission. Primary among these were
overcoming physician reluctance to use IT systems to record clinical
information and the intrinsic complexity of the quality data required by
CMS. One hospital with unusually extensive IT systems had initiated a
pilot project to see how close it could get to fully automating quality
data collection for patients with heart failure. Drawing to the maximum
extent on the data that were amenable to programming, which excluded
unstructured physician notes, the hospital found that it could complete
data collection for approximately 10 percent of cases without additional
manual abstraction. Reflecting on this effort, the hospital official
leading this project noted that at least some of the data elements
required for heart failure patients represented "clinical judgment calls."
An official at another hospital observed that someone had to apply CMS's
complex decision rules to determine the appropriate value for the data
elements. If a hospital wanted to eliminate the need for an abstractor,
who currently makes those decisions retrospectively after weighing
multiple pieces of information in the patient's medical record, the same
complex decisions would have to be made by the patient's physician at the
time of treatment. The official suggested that it was preferable not to
ask physicians to take on that additional task when they should be focused
on making appropriate treatment decisions.
Another barrier to automated quality data collection mentioned by several
hospital officials was the frequency of change in the data specifications.
As noted above, hospitals had to invest considerable staff resources for
programming and staff education to develop structured data fields for the
clinical information required for the data elements. Officials at one
hospital stated that it would be difficult to justify that investment
without knowing how long the data specifications underlying that
structured data field would remain valid.
CMS Sponsored Studies and Joined Broader HHS Initiatives to Promote Use of IT
for Quality Data Collection and Submission, but HHS Lacks Detailed Plans,
Milestones, and Time Frame
CMS has sponsored studies and joined HHS initiatives to examine and
promote the current and potential use of hospital IT systems to facilitate
the collection and submission of quality data, but HHS lacks detailed
plans, including milestones and a time frame against which to track its
progress. CMS sponsored two studies that examined the use of hospital IT
systems for quality data collection and submission. Promoting the use of
health IT for quality data collection is also 1 of 14 objectives that HHS
has identified in its broader effort to encourage the development and
nationwide implementation of interoperable IT in health care. CMS has
joined this broader effort by HHS, as well as the Quality Workgroup that
AHIC created in August 2006 to specify how IT could capture, aggregate,
and report inpatient and outpatient quality data. Through its
representation in AHIC and the Quality Workgroup, CMS has participated in
decisions about the specific focus areas to be examined through contracts
with nongovernmental entities. These contracts currently address the use
of health IT for a range of purposes, which may also include quality data
collection and submission in the near future. However, HHS has identified
no detailed plans, milestones, or time frames for either its broad effort
to encourage IT in health care nationwide or its specific objective to
promote the use of health IT for quality data collection.
CMS Sponsored Studies Examining Use of IT Systems for Collection and Submission
of Quality Data
Over the past several years, CMS sponsored two studies to examine the
current and potential capacity of hospital IT systems to facilitate
quality data collection and submission. These studies identified
challenges to using existing hospital IT systems for quality data
collection and submission, including gaps and inconsistencies in
applicable data standards, as well as in the content of clinical
information recorded in existing IT systems. Data standards create a
uniform vocabulary for electronically recorded information by providing
common definitions and coding conventions for a specified set of medical
terms. Currently, an array of different standards apply to different
aspects of patient care, including drug ordering, digital imaging,
clinical laboratory results, and overall clinical terminology relating to
anatomy, problems, and procedures.^37 The studies also found that existing
IT systems did not record much of the specific clinical information needed
to determine the appropriate data element values that hospitals submit to
CMS. To achieve CMS's goal of enabling hospitals to transmit quality data
directly from their own IT systems to CMS's nationwide clinical database,
the sets of data in the two systems should conform to a common set of data
standards and capture all the data necessary for quality measures.^38 A
key element in the effort to create this congruence is the further
development and implementation of data standards.
In the first study, completed in March 2005, CMS contracted with the
Colorado Foundation for Medical Care to test the potential for directly
downloading values for data elements for CMS's hospital quality measures
using patient data from electronic medical records in three hospitals and
one hospital system.^39 The study found that numerous factors impeded this
process under current conditions, including the lack of certain key types
of information in the hospitals' IT systems, such as emergency department
data, prearrival data, transfer information, and information on medication
contraindications. The study also noted that hospitals differed in how
they coded their data, and that even when they had implemented data
standards, the hospitals had used different versions of the standards or
applied them in different ways.^40 For example, the study found wide
variation in the way that the hospitals recorded drug names and laboratory
results in their IT systems, as none of the hospitals had implemented the
existing data standards in those areas.
^37The following standards apply in these areas: the National Council on
Prescription Drug Programs (NCDCP) for drug ordering, Digital Imaging
Communications in Medicine (DICOM) for radiological and other images,
Laboratory Logical Observation Identifier Name Codes (LOINC) for clinical
laboratory results, and Systematized Nomenclature of Medicine Clinical
Terms (SNOMED-CT) for clinical terminology.
^38This congruence is one component within the broader initiative
announced by the President to promote the adoption of interoperable
electronic health records.
^39They were MedStar Health hospital system in Baltimore, New York
Presbyterian Hospital in New York, Vanderbilt University Medical Center in
Nashville, and Wishard Memorial Hospital in Indianapolis. All had
volunteered to participate in a demonstration project called Connecting
for Health sponsored by the independent, nonprofit eHealth Initiative. See
Colorado Foundation for Medical Care, Analysis of Data from the
"Healthcare Collaborative Network" (HCN) Project, CMS Special Study
SS-CO-08, Final Report (Denver, Colo., Mar. 31, 2005).
^40Most notably, the hospitals used the messaging standard for
transmitting clinical and administrative data--HL7--in different ways,
including their coding for such data fields as admission source and
discharge disposition.
In the second study, which was conducted by the Iowa Foundation for
Medical Care and completed in February 2006, CMS examined the potential to
expand its current data specifications for heart attack, heart failure,
pneumonia, and surgical measures to incorporate the standards adopted by
the federal Consolidated Healthcare Informatics (CHI) initiative.^41
Unlike the first study, which focused on actual patient data in existing
IT systems, this study focused on the relationship of current data
standards to the data specifications for CMS's quality data. It found that
there were inconsistencies in the way that corresponding data elements
were defined in the CMS/Joint Commission Specifications Manual and in the
CHI standards that precluded applying those standards to all of CMS's data
elements. Moreover, it found that some of the data elements are not
addressed in the CHI standards. These results suggested to CMS officials
that the data standards needed to undergo further development before they
could support greater use of health IT to facilitate quality data
collection and submission.
CMS Has Joined HHS's Efforts to Promote Greater Use of Health IT for Quality
Data Collection and Submission, but HHS Lacks Detailed Plans, Milestones, and a
Time Frame to Track Progress
CMS has joined efforts by HHS to promote greater use of health IT in
general and, more recently, in facilitating the use of health IT for
quality data collection and submission. The overall goal of HHS's efforts
in this area, working through AHIC and ONC, is to encourage the
development and nationwide implementation of interoperable health IT in
both the public and the private sectors. To guide those efforts, ONC has
developed a strategic framework that outlines its goals, objectives, and
high-level strategies. One of the 14 objectives involves the collection of
quality information.^42
41The results of the Hospital Data Collection Consolidated Healthcare
Informatics Adaptation Project were summarized in an internal CMS memo
dated March 9, 2006. The CHI initiative is a collaborative agreement among
federal agencies to adopt a common set of health information
interoperability standards encompassing a wide range of clinical domains,
including the data standards referred to in footnote 37. It is a component
of the Federal Health Architecture, which is a partnership of
approximately 20 federal agencies that use health IT.
^42See GAO, Health Information Technology: HHS Is Continuing Efforts to
Define Its National Strategy, [29]GAO-06-1071T (Washington, D.C.: Sept. 1,
2006), 17-18. Other objectives that are in the strategic framework, and
that ONC has initiated specific activities to address, include encouraging
widespread adoption of data standards, promoting consumer use of personal
health information, and expanding health information support in disasters
and crises.
CMS, through its participation in AHIC, has taken part in the selection of
specific focus areas for ONC to pursue in its initial activities to
promote health IT. Those activities have largely taken place through a
series of contracts with a number of nongovernmental entities. ONC has
sought through these contracts to address issues affecting wider use of
health IT, including standards harmonization, the certification of IT
systems, and the development of a Nationwide Health Information Network.
For example, the initial work on standards harmonization, conducted under
contract to ONC by the Healthcare Information Technology Standards Panel
(HITSP), focused on three targeted areas: biosurveillance,^43 sharing
laboratory results across institutions, and patient registration and
medication history. Meanwhile, the Certification Commission for Health
Information Technology (CCHIT) has worked under a separate contract with
ONC to develop and apply certification criteria for electronic health
record products used in physician offices, with some initial work on
certification of electronic health record products for inpatient care as
well.^44
CMS is also represented on the Quality Workgroup that AHIC created in
August 2006 as a first step in promoting the use of health IT for quality
data collection and submission. One of seven workgroups appointed by AHIC,
the Quality Workgroup received a specific charge to specify how health IT
should capture, aggregate, and report inpatient as well as outpatient
quality data. It plans to address this charge by adding activities related
to using IT for quality data collection to the work performed by HITSP and
CCHIT addressing other objectives under their ongoing ONC contracts.
Members of the Quality Workgroup, along with AHIC itself, have recently
begun to consider the specific focus areas to include in the directions
given to HITSP and CCHIT for their activities during the coming year.^45
Early discussions among AHIC members indicated that they would try to
select focus areas that built on the work already completed by ONC's
contractors and that targeted specific improvements in quality data
collection that could also support other priorities for IT development
that AHIC had identified.^46 The focus areas that AHIC selects will, over
time, influence the decisions that HHS makes regarding the resources it
will allocate and the specific steps it will take to overcome the
limitations of existing IT systems for quality data collection and
submission.
^43Biosurveillance generally refers to the automated monitoring of
information sources of potential value in detecting an emerging epidemic,
whether naturally occurring or the result of bioterrorism.
^44CCHIT is a voluntary, private-sector organization set up in July 2004
by three leading health IT industry associations--the American Health
Information Management Association (AHIMA), the Healthcare Information and
Management Systems Society (HIMSS), and the National Alliance for Health
Information Technology (Alliance)--to certify health IT products.
^45As discussed at the AHIC Meeting, Washington, D.C., October 31, 2006.
These discussions resulted in a set of recommendations that the workgroup
presented at AHIC's Meeting on March 13, 2007.
In a previous report and subsequent testimony, we noted that ONC's overall
approach lacked detailed plans and milestones to ensure that the goals
articulated in its strategic framework were met. We pointed out that
without setting milestones and tracking progress toward completing them,
HHS cannot tell if the necessary steps are in place to provide the
building blocks for achieving its overall objectives.^47 HHS concurred
with our recommendation that it establish detailed plans and milestones
for each phase of its health IT strategic framework, but it has not yet
released any such plans, milestones, or a time frame for completion.
Moreover, HHS has not announced any detailed plans or milestones or a time
frame relating to the efforts of the Quality Workgroup to promote the use
of health IT to capture, aggregate, and report inpatient and outpatient
quality data. Without such plans, it will be difficult to assess how much
the focus areas AHIC selects in the near term on its contracted activities
will contribute to enabling the Quality Workgroup to fulfill its charge in
a timely way.
Conclusions
There is widespread agreement on the importance of hospital quality data.
The Congress made the APU program permanent to provide a financial
incentive for hospitals to submit quality data to CMS and directed the
Secretary of HHS to increase the number of measures for which hospitals
would have to provide data. In addition, the hospitals we visited reported
finding value in the quality data they collected and submitted to CMS to
improve care.
Collecting quality data is a complex and labor-intensive process. Hospital
officials told us that as the number of quality measures required by CMS
increased, the number of clinically trained staff required to collect and
submit quality data increased proportionately. They also told us that
increased use of IT facilitates the collection and submission of quality
data and thereby lessens the demand for greater staff resources. The
degree to which existing IT systems can facilitate data collection is,
however, constrained by limitations such as the prevalence of data
recorded as unstructured narrative or text. Overcoming these limitations
would enhance the potential of IT systems to ease the demand on hospital
resources.
^46AHIC has identified priority areas involving consumer empowerment,
biosurveillance, electronic health records, and chronic care.
^47GAO, Health Information Technology: HHS Is Taking Steps to Develop a
National Strategy, [30]GAO-05-628 (Washington, D.C.: May 27, 2005), 3;
[31]GAO-06-1071T , 18.
Promoting the use of health IT for quality data collection is 1 of 14
objectives that HHS has identified in its broader effort to encourage the
development and nationwide implementation of interoperable IT in health
care. The extent to which HHS can overcome the limitations of existing IT
systems and make progress on this objective will depend in part on where
this objective falls on the list of priorities for the broader effort. To
date, HHS has identified no detailed plans, milestones, or time frames for
either the broad effort or the specific objective on promoting the use of
health IT for collecting quality data. Without such plans, HHS cannot
track its progress in promoting the use of health IT for collecting
quality data, making it less likely that HHS will achieve that objective
in a timely way. Our analysis indicates that unless activities to
facilitate greater use of IT for quality data collection and submission
proceed promptly, hospitals may have difficulty collecting and submitting
quality data required for an expanded APU program.
Recommendations for Executive Action
To support the expansion of quality measures for the APU program, we
recommend that the Secretary of HHS take the following actions:
o identify the specific steps that the department plans to take to
promote the use of health IT for the collection and submission of
data for CMS's hospital quality measures; and
o inform interested parties about those steps and the expected
time frame, including milestones for completing them.
Agency Comments and Our Evaluation
In commenting on a draft of this report on behalf of HHS, CMS
expressed its appreciation of our thorough analysis of the
processes that hospitals use to report quality data and the role
that IT systems can play in that reporting, and it concurred with
our two recommendations. (CMS's comments appear in app. V.) With
respect to the recommendations, CMS stated that it will continue
to participate in relevant HHS studies and workgroups, and, as
appropriate, it will inform interested parties regarding progress
in the implementation of health IT for the collection and
submission of hospital quality data as specific steps, including
time frames and milestones, are identified. In addition, as health
IT is implemented, CMS anticipates that a formal plan will be
developed that includes training for providers in the use of
health IT for reporting quality data. CMS also provided technical
comments that we incorporated where appropriate.
CMS made two additional comments relating to the information
provided on our case study hospitals and our discussion of
patients excluded from the hospital performance assessments. CMS
suggested that we describe the level of health IT adoption in the
case study hospitals in table 1 of appendix III; this information
was already provided in table 4 of appendix III. CMS suggested
that we highlight the application of patient exclusions in
adapting health IT for quality data collection and submission. We
chose not to because our analysis showed that the degree of
challenge depended on the nature of the information required for a
given data element. Exclusions based on billing data, such as
discharge status, pose much less difficulty than other exclusions,
such as checking for contraindications to ACEIs and ARBs for LVSD,
which require a wide range of clinical information.
CMS noted that the AHIC Quality Workgroup had presented its
initial set of recommendations at AHIC's most recent meeting on
March 13, 2007, and provided a copy of those recommendations as an
appendix to its comments. The agency characterized these
recommendations as first steps, with initial timelines, to address
the complex issues that affect implementation of health IT for
quality data collection and submission. Specifically with
reference to collecting quality data from hospitals as well as
physicians, the Quality Workgroup recommended the appointment of
an expert panel that would designate a set of quality measures to
have priority for standardization of their data elements, which,
in turn, would enable automation of their collection and
submission using electronic health records and health information
exchange. The first recommendations from the expert panel are due
June 5, 2007. The work of the expert panel is intended to guide
subsequent efforts by HITSP to fill identified gaps in related
data standards and by CCHIT to develop criteria for certifying
electronic health record products. In addition, the Quality
Workgroup recommended that CMS and the Agency for Healthcare
Research and Quality (AHRQ) both work to bring together the
developers of health quality measures and health IT vendors, so
that development of future health IT systems would take greater
account of the data requirements of emerging quality measures.
AHIC approved these recommendations from the Quality Workgroup at
its March 13 meeting.
We also sent to each of the eight case study hospitals sections
from the appendixes pertaining to that hospital. We asked each
hospital to check that the section accurately described its
processes for collecting and submitting quality data as well as
related information on its characteristics and resources.
Officials from four of the eight hospitals responded and provided
technical comments that we incorporated where appropriate.
As arranged with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 30 days after its issue date. At that time, we will send
copies of this report to the Secretary of HHS, the Administrator
of CMS, and other interested parties. We will also make copies
available to others on request. In addition, the report will be
available at no charge on GAO's Web site at http://www.gao.gov
.
If you or your staffs have any questions about this report, please
contact me at (202) 512-7101 or [email protected]. Contact points
for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix VI.
Cynthia A. Bascetta
Director, Health Care
Appendix I: Medicare Quality Measures Required for Full Annual
Payment Update
Number of
required data
Condition Quality measure elements
Heart attack Aspirin at hospital arrival^a 11
Aspirin prescribed at discharge^a 7
Angiotensin-converting enzyme inhibitor or
angiotensin receptor blocker for left
ventricular systolic dysfunction^a 9
Beta blocker at hospital arrival^a 11
Beta blocker prescribed at discharge^a 7
Thrombolytic agent received within 30 minutes
of hospital arrival 13
Percutaneous coronary intervention received
within 120 minutes of hospital arrival 16
Adult smoking cessation advice/counseling 7
Heart failure Left ventricular function assessment^a 7
Angiotensin-converting enzyme inhibitor or
angiotensin receptor blocker for left
ventricular systolic dysfunction^a 10
Discharge instructions 12
Adult smoking cessation advice/counseling 8
Pneumonia Initial antibiotic received within 4 hours of
hospital arrival^a 16
Oxygenation assessment^a 11
Pneumococcal vaccination status^a 8
Blood culture performed before first
antibiotic received in hospital 19
Adult smoking cessation advice/counseling 9
Appropriate initial antibiotic selection 24
Influenza vaccination status 9
Surgery Prophylactic antibiotic received within 1 hour
prior to surgical incision 14
Prophylactic antibiotics discontinued within
24 hours after surgery end time 17
Sources: Federal Register, CMS, GAO (analysis).
Notes: The 21 measures are listed in 71 Fed. Reg. 47870,
48033-48034, 48045 (Aug. 18, 2006), and we analyzed the
Specifications Manual for National Hospital Quality Measures,
version 2.1a, to calculate the number of required data elements
for each. This set of quality measures is effective for discharges
from July 2006 on. The Centers for Medicare & Medicaid Services
(CMS) uses 73 different data elements to calculate hospital
performance on the 21 measures required for the APU program. The
total number of unique data elements is less than the sum of the
data elements used to calculate each measure because some data
elements are included in the calculation of more than one quality
measure. In addition, CMS obtains from hospitals approximately 20
other data elements on each patient, including demographic and
billing data.
aOne of the 10 original quality measures.
Appendix II: Data Elements Used to Calculate Hospital Performance
on a Heart Attack Quality Measure
Figure 3: Data Elements Used to Calculate Hospital Performance on
the Heart Attack Quality Measure That Asks Whether a Beta Blocker
Was Given When the Patient Arrived at the Hospital
Notes: The boxes represent data elements and the circles and
rounded rectangles represent values for those elements. In
addition to the seven data elements shown in the figure (including
arrival date and discharge date that appear in the same box), an
eighth data element, comfort measures only, is first applied for
this quality measure, as well as all the other heart attack, heart
failure, and pneumonia quality measures, to screen out terminal
patients receiving palliative care. Three other data
elements--principal diagnosis, admission date, and birthdate--are
used to initially identify the patients for whom the heart failure
quality measures apply in a given quarter.
aIncluded codes consist of eight different values for admission
source that represent patients who were admitted from any source
other than those listed in footnote b, including physician
referral, skilled nursing facility, and the hospital's emergency
room.
bExcluded codes consist of three different values for admission
source that represent patients who were transferred to this
hospital from another acute care hospital, from a critical access
hospital, or within the same hospital with a separate claim.
cPatients may be excluded from the population used to calculate a
hospital's performance for a variety of reasons, including
inappropriateness of beta blockers for their treatment--for
example, if they have a contraindication for their use--or prior
treatment in another acute care facility.
dIncluded codes consist of 13 different values for discharge
status that represent patients who were discharged to any setting
other than those listed in footnote e, including home care,
skilled nursing facility, and hospice.
eExcluded codes consist of five different values for discharge
status that represent patients who were discharged to another
acute care hospital or federal health care facility, left against
medical advice, or died.
Appendix III: Tables on Eight Case Study Hospitals
Table 1: Case Study Hospital Characteristics
Case study hospital
A B C D E F G H
Number of 300-349 500+ 50-99 500+ 100-149 500+ 150-199 500+
beds
Urban/rural Urban Urban Rural Urban Suburban Urban Suburban Urban
Major Yes Yes No Yes No Yes No Yes
teaching
Member of Yes Yes Yes No No No No No
multihospital
system
Joint Yes Yes Yes Yes Yes Yes Yes Yes
Commission
accredited
Vendor Yes Yes Yes Yes Yes Yes Yes Yes
submits
quality data
Patients Monthly Monthly Weekly Monthly Monthly Monthly Monthly Monthly
identified
for data
collection
how often
Abstraction Vendor's Vendor's Vendor's CART^a Vendor's Vendor's Vendor's Vendor's
tool used
Conditions Heart Heart Heart Heart Heart Heart Heart Heart
reported on attack, attack, attack, attack, attack, attack, attack, attack,
heart heart heart heart heart heart heart heart
failure, failure, failure, failure, failure, failure, failure, failure,
pneumonia, pneumonia, pneumonia, pneumonia, pneumonia, pneumonia, pneumonia, pneumonia,
surgery surgery surgery surgery surgery surgery surgery surgery
Entities that CMS, Joint CMS, Joint CMS, Joint CMS, Joint CMS, Joint CMS, Joint CMS, Joint CMS, Joint
receive Commission Commission, Commission Commission Commission, Commission Commission Commission
Annual vendor vendor
Payment database, database
Update (APU) private
program data insurers
Entities that Leapfrog^b Leapfrog, Private Leapfrog, Private Leapfrog, State Private
receive state insurer private insurer private health insurer
different health insurer insurer department,
quality data department, private
private insurers
insurers
Amount of $139,000 $608,000 $33,000 $449,000 $57,000 $430,000 $93,000 $123,000
projected
reduction in
fiscal year
2006 Medicare
payments if
quality data
not
submitted^c
Amount of $801,000 $3,250,000 $161,000 $2,298,000 $283,000 $2,451,000 $503,000 $608,000
projected
reduction in
fiscal year
2007 Medicare
payments if
quality data
not
submitted^c
Sources: American Hospital Association, GAO, Centers for Medicare &
Medicaid Services (CMS).
aCART, which stands for the CMS Abstraction and Reporting Tool, was
developed by CMS and made available to hospitals at no charge for
collecting and submitting quality data.
bThe Leapfrog Group is a consortium of large private and public health
care purchasers that publicly recognizes hospitals that have implemented
certain specific quality and safety practices, such as computerized
physician order entry.
cThe projected reduction in fiscal year 2006 and fiscal year 2007 Medicare
payments (rounded to the nearest $1,000) represents the amount that the
hospital's revenue from Medicare would have decreased for that fiscal year
had the hospital not submitted quality data under the Annual Payment
Update program. These estimates are based on information on the number and
case mix of Medicare patients served by these hospitals during the
previous period. This is the information that was available to hospital
administrators from CMS at the beginning of the fiscal year. The actual
reduction would ultimately depend on the number and case mix of the
Medicare patients that the hospital actually treated during the course of
that fiscal year. The projected reduction for fiscal year 2007 was
substantially larger because that was the first year in which the higher
rate of reduction mandated by the Deficit Reduction Act of 2005--from 0.4
percentage points to 2.0 percentage points--took effect.
Table 2: How Case Study Hospital Officials Described the Steps Taken to
Complete Quality Data Collection and Submission
Case study hospital
A B C D
1. Identify Vendor prepares Vendor Vendor Hospital IT
patients^a list of prepares list prepares list department
patients to of patients of patients identifies
abstract, based on to abstract patients
sampling heart diagnosis based on based on
failure, codes, and billing data, billing data,
pneumonia, and draws samples no sampling no sampling
surgery for heart
failure,
pneumonia, and
surgery
2. Locate Abstractor Abstractor Abstractor Abstractor
information searches starts search searches clicks
in the through with through through
medical emergency room electronic different various
record and inpatient discharge components of electronic
electronic and summary, then paper record, screens
paper records, other including representing
checking electronic printouts different
multiple forms records and from types of
and screens paper electronic records, plus
where relevant documents records some scanned
information documents,
could be found for example,
from other
providers
3. Determine Some Some Some Some
appropriate demographic demographic demographic demographic
data element data data data data
values prepopulated; prepopulated; prepopulated; prepopulated;
abstractor other data other data most
notes ambiguous elements elements abstractors
or conflicting written on entered fill in data
information on paper directly into elements on
paper abstraction vendor's paper
abstraction form online abstraction
form abstraction form
tool
4. Transmit Data elements Data elements Data elements Data elements
data to CMS copied from copied from entered copied from
paper paper directly into paper
abstraction abstraction vendor's abstraction
form to form to online form to
vendor's online vendor's abstraction vendor's
form online form tool electronic
form; data
manager
checks data
and uploads
file to
vendor
5. Ensure Performed by Hospital staff Performed by Hospital
data have vendor reviews error vendor staff reviews
been reports from error reports
accepted by clinical data from vendor
CMS warehouse and
corrects
errors
6. Supply Hospital copies Hospital Hospital Hospital
copies of and ships copies, checks copies, copies and
selected requested completeness checks ships
medical patient records of, and ships completeness requested
records requested of, and ships patient
patient requested records
records patient
records
Case study hospital
E F G H
Hospital Hospital provides Hospital creates Hospital submits
prepares list of billing data to list from billing billing data to
patients from vendor; vendor draws data; vendor vendor, which
billing data, no samples and generates provides identifies
sampling list of patients to instructions to eligible
abstract draw sample of patients and
pneumonia cases draws samples
Abstractor works Abstractor starts Abstractor starts Abstractor
through paper with electronic searching through searches through
records, such as records (for heart paper records, both electronic
face sheet, attack and heart then looks for and paper
emergency room failure)--first additional records
treatment forms, structured records information in
progress notes, (discharge) and then electronic records
and discharge free text--and then (e.g., for
summary examines paper echocardiogram
records if needed; results)
paper records
searched for
pneumonia and surgery
Data elements Some demographic data Some demographic Some demographic
entered into prepopulated; data prepopulated; data
computerized abstractors fill out other data prepopulated;
abstraction form abstraction form, elements written other data
some on paper and on paper elements written
some online abstraction form on paper
abstraction form
Completed For pneumonia and Data elements Data elements
abstraction surgery, abstractor copied from paper copied from
forms sent on enters data online, abstraction form paper
disk to vendor; for heart attack and to vendor's online abstraction form
will change soon heart failure, form to vendor's
to completion of hospital scans paper online form
forms online abstraction forms and
sends electronic file
to vendor, which
submits data to CMS
Hospital reviews Performed by vendor Hospital receives Hospital reviews
error reports error report from error reports
from vendor and vendor and from vendor and
clinical clinical data makes
warehouse warehouse and corrections;
makes corrections vendor deals
with clinical
data warehouse
Hospital copies Hospital copies and Hospital copies, Hospital copies,
and ships ships requested checks checks
requested patient records; completeness of, completeness of,
patient records before shipping and ships and ships
hospital flags requested patient requested
relevant information records patient records
Source: GAO.
Note: Information summarized from hospital case study interviews.
aThe identifying patients step included both determining all the patients
who met the CMS criteria for inclusion and the application of the CMS
sampling procedures, if applicable. CMS only permitted hospitals to sample
patients for a given condition in a given quarter if the number of
eligible patients met a certain threshold. Otherwise, the hospital was
required to abstract quality data for all patients who met the inclusion
criteria for any one of the four conditions. Hospitals could also choose
not to sample, even if it were permitted under the CMS sampling
procedures.
Table 3: Resources Used for Abstraction and Data Submission at Eight Case
Study Hospitals
Case study hospital
A B C D E F G H
Qualifications Medical Registered All RN All RN Licensed Medical RN and All RN
of abstractors record coders nurse (RN) practical records coder LPN^a
and a Master and nurse and RN with
of Public nonclinical (LPN) physician
Health support
Number of 3 3 3 9 2 3 3 4
abstractors
Estimated full 0.7 <2.0 <1.5 2.5 1.2 1.3 1.2 2.0
time
equivalents
for
abstraction of
data elements
Estimated time 60 minutes 10 to 15 20 3 to 120 5 to 60 5 to 60 10 to 30 10 to 90
to abstract (average) minutes minutes minutes minutes minutes minutes minutes
one chart (average)
Average number 222 399 86 686 118 252 190 202
of heart
attack, heart
failure, and
pneumonia
charts
abstracted per
quarter^b
Average number 94^b 218^c 6^c 553^b 105^b 186^b 82^c 61^c
of surgery
charts
abstracted per
quarter
Data vendor $8,200 $5,000 to $3,600 $3,500 $560 $1,800 $12,450 $13,000
costs for CMS $7,500
quality data
services per
year
Checks for Some A few Relies on Checks None Some None Checks
accuracy of discrepancies discrepancies vendor only that discrepancies patient
case selection observed observed processes data were observed lists for
against and submitted heart
another data audits of to CMS for attack
source medical all patients
records patients against
coding on medical
original records
list to be
abstracted
Checks for Hospital None beyond Hospital Only for Only for Only for Not None
accuracy of reabstracts 5 reviews by redoes 5 cases cases cases where routinely, beyond
data percent of CMS to 10 where where quality only for reviews by
abstraction cases each contractor cases per quality quality standard not startup in CMS
quarter measure standard standard met new contractor
set every not met not met condition
quarter
Sources: GAO, CMS.
aThe LPN was abstracting cases for one condition temporarily until an RN
could be hired to perform the work.
bBased on submissions to the clinical warehouse for four quarters of
discharges from April 2005 through March 2006.
cBased on submissions to the clinical warehouse for one quarter of
discharges from January through March 2006.
Table 4: Electronic and Paper Records at Eight Case Study Hospitals
Case study hospital
A B C D E F G H
Admissions E E E E E E E E
Billing E E E E E E E E
Emergency department E&P E&P P E P E P P
Medication administration E E P E P P E P
Physician orders including prescriptions E&P E&P P E P E P E
Nursing notes P P P E P P E P
Laboratory and test results E E E E E E E E
Physician notes P E&P P E E&P E&P E&P E&P
Discharge summaries and instructions P E P E P E&P E E
Operating room P E&P E&P E P E&P E E
Source: GAO.
Note: E = electronic, P = paper.
Appendix IV: Scope and Methodology
To examine how hospitals collect and submit quality data, and to determine
the extent to which information technology (IT) facilitates those
processes, we conducted case studies of eight individual acute care
hospitals that collect and submit quality data to the Centers for Medicare
& Medicaid Services (CMS). We chose this approach to obtain an in-depth
understanding of these processes as they are currently experienced at the
hospital level. For background information on the requirements that the
hospitals had to satisfy, we reviewed CMS documents relevant to the Annual
Payment Update (APU) program. In particular, we examined multiple
revisions of the Specifications Manual for National Hospital Quality
Measures, which is issued jointly by CMS and the Joint Commission
(formerly the Joint Commission on Accreditation of Healthcare
Organizations).
We structured our selection of hospitals for the eight case studies to
provide a contrast of hospitals with highly sophisticated IT systems and
hospitals with an average level of IT capability. We excluded critical
access hospitals from this selection process because they are not included
in the APU program.^1 The selected hospitals varied on several hospital
characteristics, including urban/rural location, size, teaching status,
and membership in a system that linked multiple hospitals through shared
ownership or other formal arrangements. (See app. III, table 1.)
To select four hospitals with highly sophisticated IT systems, we relied
on recommendations from interviews with a number of experts in the field
of health IT, as well as on a recent review of the research literature on
the costs and benefits of health IT^2 and other published articles. Three
of the four hospitals we chose were among those where much of the
published research has taken place. They were all early adopters of health
IT, and each had implemented internally developed IT systems. The fourth
hospital had more recently acquired and adapted a commercially developed
system. This hospital was distinguished by the extent to which it had
replaced its paper medical records with an integrated system of electronic
patient records. Each of these four case study hospitals was located in a
different metropolitan area.
^1Some critical access hospitals submit quality data to CMS voluntarily,
but this does not affect their Medicare payments.
^2P.G. Shekelle, S.C. Morton, and E.B. Keeler, Costs and Benefits of
Health Information Technology, Evidence Report/Technology Assessment No.
132 (prepared by the Southern California Evidence-based Practice Center
under Contract No. 290-02-0003), Agency for Healthcare Research and
Quality Publication No. 06-E006 (Rockville, Md., April 2006).
We selected the four hospitals with less sophisticated IT systems from the
geographic vicinity of the four hospitals already chosen, thus providing
two case study hospitals from each of four metropolitan areas. We decided
that one should be a rural hospital, using the Medicare definition of
rural, which is located outside of a Metropolitan Statistical Area (MSA).
To determine from which of the four metropolitan areas we should select a
neighboring rural hospital, we analyzed data on Medicare-approved
hospitals drawn from CMS's Provider of Services (POS) file. We identified
the rural hospitals located within 150 miles of each of the first four
hospitals. From among those four sets of rural hospitals, we chose the set
with the largest number of acute care hospitals as the set from which to
choose our rural case study hospital. For each of the remaining three
metropolitan areas, we used the hospitals listed in the POS file as
short-term acute care hospitals located in the same MSAs as the three sets
from which to choose our remaining three hospitals. We excluded hospitals
located in a different state from the first hospital selected for that
metropolitan area, so that all of the hospitals under consideration for
that area would come under the jurisdiction of the same Quality
Improvement Organization (QIO).^3
To select the second case study hospital from among those available in or
near each of the four metropolitan areas, we applied a procedure designed
to produce a straightforward and unbiased selection. We began by recording
the total number of cases for which each of these hospitals had reported
results on CMS's Web site for heart attack, heart failure, and pneumonia
quality measures. We obtained this information from the Web site itself,
running reports for each hospital that showed, for each quality measure,
the number of cases that the hospital's quality performance score was
based on. Since some quality measures apply only to certain patients, we
recorded the largest number of cases listed for any of the quality
measures reported for a given condition. Next we summed the cases for the
three conditions and rank ordered the hospitals in each of the three MSAs,
and the rural hospitals in the fourth metropolitan area, from most to
least total cases submitted. We then made a preliminary selection by
taking the hospital with the median value in each of those lists.^4 By
selecting the hospital with the median number of cases reported, we
attempted to minimize the chances of picking a hospital that would
represent an outlier compared to other hospitals in the selection pool.^5
3QIOs are independent organizations that work under contract to CMS to
monitor quality of care for the Medicare program within a given state and
help providers to improve their clinical practices. CMS has assigned
primary responsibility to the QIOs to inform hospitals about the APU
program's requirements and to provide technical assistance to hospitals in
meeting those requirements.
Before selecting the final four case study hospitals, we checked to make
sure that the hospitals did not happen to have an unusually high level of
IT capabilities with respect to electronic patient records. To do this, we
contacted each of the selected hospitals and obtained a description of its
current IT systems. We compared this description to the stages of
electronic medical record implementation laid out by the Healthcare
Information and Management Systems Society (HIMSS).^6 The HIMSS model
identifies eight stages based on the scope and sophistication of clinical
functions implemented through a hospital's system of electronic medical
records. According to HIMSS, the large majority of hospitals in the United
States are at the lower three stages. Based on the descriptions of these
stages, we determined that none of the prospectively selected hospitals
had IT systems that exceeded the third stage.
We collected information about the processes used to collect and submit
quality data from each of the eight case study hospitals through on-site
interviews with hospital abstractors, quality managers, IT staff, and
hospital administrators. We told these officials that neither they
personally nor their hospitals would be identified by name in our report.
The site visits took place between mid-July and early September 2006 and
ranged in duration from 3 to 8 hours. Our data collection at each hospital
was guided by a protocol that specified a series of topics to cover in our
interviews. These topics included a description of the processes used at
each hospital and the financial and staff resources devoted to quality
data collection and submission. We pretested the protocol at two hospitals
not included in our set of eight case study hospitals.
^4Any ties on median values, which were possible if the list had an even
number of hospitals, were resolved by implementing a decision rule that
alternated between taking the higher number of cases for the first
instance, the lower number for the second instance, and so on.
^5For example, a few hospitals on these lists had submitted results for
only one condition.
^6D. Garets and M. Davis, "Electronic Medical Records vs. Electronic
Health Records: Yes, There Is a Difference" (Chicago, Ill.: HIMSS
Analytics, LLC, updated Jan. 26, 2006).
As part of the protocol, we asked abstractors at each hospital to explain
in detail how they found the information needed to determine the
appropriate values for each of the data elements required for two specific
quality measures: (1) angiotensin-converting enzyme inhibitor (ACEI) or
antiotensin receptor blocker (ARB) for left ventricular systolic
dysfunction (LVSD) for heart failure patients and (2) initial antibiotic
received within 4 hours of hospital arrival for pneumonia patients. We
selected these measures because they covered a number of different types
of data elements, including those involving administration of medications,
determining contraindications, date and time variables, and making
clinical assessments such as whether a patient had LVSD.
To determine the extent to which IT facilitated these processes at the
eight case study hospitals, we included several topics on IT systems in
our site visit protocol. We asked about any IT systems used by the
abstractors in locating relevant clinical information in patient medical
records and the specific advantages and limitations they encountered in
using those systems. We also asked hospital officials to assess the
potential for IT systems to provide higher levels of assistance for
quality data collection and submission over time. If separate IT staff
were involved in the hospital's quality data collection and submission
process, we included them in the interviews.
Where possible, we supplemented the information provided through
interviews with direct observation of the processes used by hospitals to
collect and submit quality data. We asked the case study hospitals to show
us how they performed these processes, and five of the eight hospitals
arranged for us to observe the collection of quality data for all or part
of a patient record. We observed abstractors accessing clinical
information from both paper and electronic records.
We also obtained pertinent information about the case study hospitals from
CMS documents and contractors. The estimated amount of dollars that the
case study hospitals would have lost had they not submitted quality data
to CMS, presented in appendix III, table 1, was calculated from data
provided in documents made available to all hospitals at the start of each
of the fiscal years.^7 Information on the average number of patient charts
abstracted quarterly by each case study hospital, shown in appendix III,
table 3, was drawn from a table showing the number of patients for whom
quality data were submitted to CMS's clinical data warehouse. We obtained
that table from the Iowa Foundation for Medical Care (IFMC), which is the
CMS contractor that operates the clinical data warehouse. The IFMC table
provided this information for all hospitals submitting quality data for
discharges that occurred from April 2005 through March 2006. These were
the most recent data available.
The evidence that we obtained from our eight case study hospitals is
specific to those hospitals. In particular, it does not offer a basis for
relating any differences we observed among these individual hospitals to
their differences on specific dimensions, such as size or teaching status.
Nor can we generalize from the group of eight as a whole to acute care
hospitals across the country. Furthermore, although we examined the
processes hospitals used to collect and submit quality data and the role
that IT plays in that process, we did not examine general IT adoption in
the hospital industry.
To obtain information on whether CMS has taken steps to promote the
development of IT systems to facilitate quality data collection and
submission, we interviewed CMS officials as well as CMS contractors and
reviewed documents including reports on related studies funded by CMS. We
also interviewed officials at the Office of the National Coordinator for
Health Information Technology (ONC) regarding the plans and activities of
the American Health Information Community (AHIC) quality workgroup. In
addition, we downloaded relevant documents from the AHIC Web site,
including meeting agendas, prepared presentations, and meeting minutes for
both AHIC as a whole and its Quality Workgroup.
We conducted our work from February 2006 to April 2007 in accordance with
generally accepted government auditing standards.
^7These included the final rule for Inpatient Prospective Payment System
updates for fiscal years 2006 and 2007, 70 Fed. Reg. 47507 (Aug. 12, 2005)
and 71 Fed. Reg. 48166 (Aug. 18, 2006), and the "Impact file for IPPS FY
2006 Final Rule" and "Impact file for IPPS FY 2007 Final Rule" downloaded
from http://www.cms.hhs.gov/AcuteInpatientPPS/FFD/list.asp#TopOfPage on
October 12, 2006.
Appendix V: Comments from the Centers for Medicare & Medicaid Services
Appendix VI: GAO Contact and Staff Acknowledgments
GAO Contact
Cynthia A. Bascetta, (202) 512-7101 or [email protected]
Acknowledgments
In addition to the contact named above, Linda T. Kohn, Assistant Director;
Mohammad S. Khan; Eric A. Peterson; Roseanne Price; Jessica C. Smith; and
Teresa F. Tucker made key contributions to this report.
(290520)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or (202)
512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
www.gao.gov/cgi-bin/getrpt?GAO-07-320 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Cynthia A. Bascetta, (202) 512-7101 or
[email protected].
Highlights of [40]GAO-07-320 , a report to the Committee on Finance, U.S.
Senate
April 2007
HOSPITAL QUALITY DATA
HHS Should Specify Steps and Time Frame for Using Information Technology
to Collect and Submit Data
Hospitals submit data in electronic form on a series of quality measures
to the Centers for Medicare & Medicaid Services (CMS) and receive scores
on their performance. Increasingly, the clinical information from which
hospitals derive the quality data for CMS is stored in information
technology (IT) systems.
GAO was asked to examine (1) hospital processes to collect and submit
quality data, (2) the extent to which IT facilitates hospitals' collection
and submission of quality data, and (3) whether CMS has taken steps to
promote the use of IT systems to facilitate the collection and submission
of hospital quality data. GAO addressed these issues by conducting case
studies of eight hospitals with varying levels of IT development and
interviewing relevant officials at CMS and the Department of Health and
Human Services (HHS).
[41]What GAO Recommends
GAO recommends that the Secretary of HHS identify the specific steps the
department plans to take to promote the use of health IT for the
collection and submission of data for CMS's hospital quality measures and
inform interested parties about those steps, the expected time frame, and
associated milestones. In commenting on a draft of this report on behalf
of HHS, CMS concurred with these recommendations.
The eight case study hospitals used six steps to collect and submit
quality data: (1) identify the patients, (2) locate information in their
medical records, (3) determine appropriate values for the data elements,
(4) transmit the quality data to CMS, (5) ensure that the quality data
have been accepted by CMS, and (6) supply copies of selected medical
records to CMS to validate the data. Several factors account for the
complexity of abstracting all relevant information in a patient's medical
record, including the content and organization of the medical record, the
scope of information and the clinical judgment required for the data
elements, and frequent changes by CMS in its data specifications. Due in
part to these complexities, most of the case study hospitals relied on
clinical staff to abstract the quality data. Increases in the number of
quality measures required by CMS led to increased demands on clinical
staff resources. Offsetting the demands placed on clinical staff were the
benefits that case study hospitals reported finding in the quality data,
such as providing feedback to clinicians and reports to hospital
administrators.
GAO's case studies showed that existing IT systems can help hospitals
gather some quality data but are far from enabling hospitals to automate
the abstraction process. IT systems helped hospital staff to abstract
information from patients' medical records, in particular by improving
accessibility to and legibility of the medical record. The limitations
reported by officials in the case study hospitals included having a mix of
paper and electronic records, which required staff to check multiple
places to get the needed information; the prevalence of data recorded as
unstructured narrative or text, which made locating the information
time-consuming because it was not in a prescribed place in the record; and
the inability of some IT systems to access related data stored in another
IT system in the same hospital, which required staff to access each IT
system separately to obtain related pieces of information. Hospital
officials expected the scope and functionality of their IT systems to
increase over time, but this process will occur over a period of years.
CMS has sponsored studies and joined HHS initiatives to examine and
promote the current and potential use of hospital IT systems to facilitate
the collection and submission of quality data, but HHS lacks detailed
plans, including milestones and a time frame against which to track its
progress. CMS has joined efforts by HHS to promote the use of IT in health
care, including a Quality Workgroup charged with specifying how IT could
capture, aggregate, and report inpatient and outpatient quality data. HHS
plans to expand the use of health IT for quality data collection and
submission through contracts with nongovernmental entities that currently
address the use of health IT for a range of other purposes. However, HHS
has identified no detailed plans, milestones, or time frames for either
its broad effort to encourage IT in health care nationwide or its specific
objective to promote the use of health IT for quality data collection.
References
Visible links
27. http://www.gao.gov/cgi-bin/getrpt?GAO-06-54
28. http://www.gao.gov/cgi-bin/getrpt?GAO-06-54
29. http://www.gao.gov/cgi-bin/getrpt?GAO-06-1071T
30. http://www.gao.gov/cgi-bin/getrpt?GAO-05-628
31. http://www.gao.gov/cgi-bin/getrpt?GAO-06-1071T
40. http://www.gao.gov/cgi-bin/getrpt?GAO-07-320
*** End of document. ***