Hospital Quality Data: Issues and Challenges Related to How	 
Hospitals Submit Data and How CMS Ensures Data Reliability	 
(06-MAR-08, GAO-08-555T).					 
                                                                 
Hospitals submit data on a series of quality measures to the	 
Centers for Medicare & Medicaid Services (CMS) and receive scores
on their performance. CMS instituted the Reporting Hospital	 
Quality Data for Annual Payment Update Program (APU program) to  
collect the quality data from hospitals and report their rates on
the measures on its Hospital Compare Web site. For hospital	 
quality data to be useful to patients and other users, they need 
to be reliable, that is, accurate and complete. The Deficit	 
Reduction Act of 2005 directed CMS to implement a value-based	 
purchasing program for Medicare that beginning in fiscal year	 
2009 would adjust payments to hospitals based on factors related 
to the quality of care they provide. This statement provides	 
information on (1) how hospitals collect and submit quality data 
to CMS and (2) how CMS works to ensure the reliability of the	 
quality data submitted. This statement is based primarily on	 
Hospital Quality Data: HHS Should Specify Steps and Time Frame	 
for Using Information Technology to Collect and Submit Data	 
(GAO-07-320, Apr. 25, 2007) and Hospital Quality Data: CMS Needs 
More Rigorous Methods to Ensure Reliability of Publicly Released 
Data (GAO-06-54, Jan. 31, 2006). In preparing these reports, GAO 
conducted case studies of eight hospitals, and reviewed documents
of, and interviewed officials at CMS.				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-555T					        
    ACCNO:   A81220						        
  TITLE:     Hospital Quality Data: Issues and Challenges Related to  
How Hospitals Submit Data and How CMS Ensures Data Reliability	 
     DATE:   03/06/2008 
  SUBJECT:   Data collection					 
	     Data integrity					 
	     Data transmission					 
	     Electronic data processing 			 
	     Health information architecture			 
	     Health resources utilization			 
	     Hospitals						 
	     Information management				 
	     Information technology				 
	     Medical information systems			 
	     Medical records					 
	     Performance measures				 
	     Quality assurance					 
	     Quality control					 
	     Reporting requirements				 
	     Statistical data					 
	     Annual Payment Update Program			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-555T

This is the accessible text file for GAO report number GAO-08-555T 
entitled 'Hospital Quality Data: Issues and Challenges Related to How 
Hospitals Submit Data and How CMS Ensures Data Reliability' which was 
released on March 6, 2008.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to [email protected]. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Testimony: 

Before the Committee on Finance, U.S. Senate: 

United States Government Accountability Office:
GAO: 

For Release on Delivery: 
Expected at 2:00 p.m. EST: 
Thursday, March 6, 2008: 

Hospital Quality Data: 

Issues and Challenges Related to How Hospitals Submit Data and How CMS 
Ensures Data Reliability: 

Statement for the Record: 

Linda T. Kohn, Acting Director:
Health Care: 

GAO-08-555T: 

GAO Highlights: 

Highlights of GAO-08-555T, a statement for the record for the Committee 
on Finance, U.S. Senate. 

Why GAO Did This Study: 

Hospitals submit data on a series of quality measures to the Centers 
for Medicare & Medicaid Services (CMS) and receive scores on their 
performance. CMS instituted the Reporting Hospital Quality Data for 
Annual Payment Update Program (APU program) to collect the quality data 
from hospitals and report their rates on the measures on its Hospital 
Compare Web site. For hospital quality data to be useful to patients 
and other users, they need to be reliable, that is, accurate and 
complete. 

The Deficit Reduction Act of 2005 directed CMS to implement a value-
based purchasing program for Medicare that beginning in fiscal year 
2009 would adjust payments to hospitals based on factors related to the 
quality of care they provide. 

This statement provides information on (1) how hospitals collect and 
submit quality data to CMS and (2) how CMS works to ensure the 
reliability of the quality data submitted. This statement is based 
primarily on Hospital Quality Data: HHS Should Specify Steps and Time 
Frame for Using Information Technology to Collect and Submit Data (GAO-
07-320, Apr. 25, 2007) and Hospital Quality Data: CMS Needs More 
Rigorous Methods to Ensure Reliability of Publicly Released Data (GAO-
06-54, Jan. 31, 2006). In preparing these reports, GAO conducted case 
studies of eight hospitals, and reviewed documents of, and interviewed 
officials at CMS. 

What GAO Found: 

GAO reported in April 2007 that the eight case study hospitals visited 
used six steps to collect and submit quality data, two of which (steps 
2 and 3) involved complex abstractionï¿½the process of reviewing and 
assessing all relevant pieces of information in a patientï¿½s medical 
record to determine the appropriate value for each data element. The 
six steps were (1) identify patients for whom the quality data should 
be submitted, (2) locate needed information in the medical records, (3) 
determine the appropriate value for each data element, (4) transmit the 
data to CMS, (5) review reports to ensure acceptance of the data by 
CMS, and (6) supply copies of selected medical records to CMS for data 
validation. Several factors account for the complexity of the 
abstraction process (steps 2 and 3), including the content and 
organization of the medical record, the scope of information and 
clinical judgment required for certain data elements, and frequent 
changes by CMS in its data specifications. GAOï¿½s case studies also 
showed that existing information technology (IT) systems help hospitals 
gather some quality data but are far from enabling hospitals to 
automate the abstraction process. 

GAO reported in January 2006 that CMS had processes for ensuring the 
accuracy of the quality data submitted by hospitals but had no ongoing 
process for ensuring completeness of these data. To check accuracy, one 
CMS contractor electronically checks the data as they are submitted to 
the clinical warehouse. Another contractor conducts an independent 
audit by comparing the quality data submitted by a hospital from the 
medical records for a sample of five patients per quarter for each 
hospital to the quality data that the contractor reabstracts from the 
same medical records. The data are deemed to be accurate if there is 80 
percent or greater agreement between these two sets of results. 
However, GAO also reported that CMSï¿½s determination as to whether 
hospitals met the accuracy standard was statistically uncertain for 
some hospitals because of the small number of records examinedï¿½five 
cases per quarter per hospital regardless of the hospitalï¿½s size. In 
2006 GAO also reported that CMS did not have an ongoing process for 
assessing the completeness of quality data submitted by hospitals and 
recommended that CMS take steps to improve its processes for ensuring 
the accuracy and completeness of the hospital quality data. CMS agreed 
the process needed improvement. For fiscal year 2008 and subsequent 
years, CMS required that hospitals attest each quarter to the 
completeness and accuracy of their data. Further, in a 2007 report to 
Congress that lays out a plan to implement a value-based purchasing 
program, CMS recognized the need to redesign the data infrastructure 
and validation process to support a value-based purchasing program by, 
for example, increasing the number of patient medical records sampled 
from selected hospitals. 

To view the full product, including the scope and methodology, click on 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?gao-08-555T]. For more information, contact 
Linda T. Kohn at (202) 512-7114 or [email protected]. 

[End of section] 

Mr. Chairman and Members of the Committee: 

I am pleased to have the opportunity to comment as requested on topics 
related to the Centers for Medicare & Medicaid Services's (CMS) Value-
based Purchasing Program Implementation Plan. On November 21, 2007, CMS 
issued a report to Congress that lays out its plan to implement this 
program. The plan builds on the foundation of CMS's Annual Payment 
Update (APU) program that requires participating hospitals to submit 
data--referred to here as quality data--that are used to calculate 
hospital performance on measures of the quality of care provided in 
order to avoid a reduction in their full Medicare payment update each 
fiscal year.[Footnote 1] The vast majority of acute care hospitals 
treating Medicare patients choose to submit quality data each quarter 
to CMS, rather than accept a reduced annual payment update. 

In the APU program, each quality measure consists of a set of 
standardized data elements, which define the specific data that 
hospitals need to submit to CMS. Hospitals determine a value for each 
data element of a measure for patients--Medicare and non-Medicare--who 
have a medical condition covered by the APU program, that is, heart 
attack, heart failure, pneumonia, or surgery. The values for the data 
elements consist of numerical data and other administrative and 
clinical information that are obtained from the medical records of the 
patients. Hospitals submit their quality data electronically, over the 
Internet, to a clinical data warehouse operated by a CMS contractor. 

In order to inform the public about hospital quality, CMS posts on a 
public Web site--Hospital Compare--the performance scores that 
hospitals receive on the quality measures derived from the data they 
submit. For hospital quality data to be useful to patients and other 
users, the data need to be reliable--that is, both accurate and 
complete. If a hospital submits complete data, that is, data on all the 
cases that meet the specific inclusion criteria for eligible patients, 
but the data are not collected, or abstracted, from the patients' 
medical records accurately, the data will not be reliable. Similarly, 
if a hospital submits accurate data, but those data are incomplete 
because the hospital leaves out eligible cases, the data will not be 
reliable. 

Although the APU program was originally set to expire in 2007, the 
Deficit Reduction Act of 2005[Footnote 2] (DRA) made the APU program 
permanent. The act also raised the Medicare payment reduction[Footnote 
3] and required the Secretary of Health and Human Services (HHS) to 
increase the number of measures for which hospitals participating in 
the APU program would have to provide data in order to receive their 
full Medicare payment update. Furthermore, DRA directed the Secretary 
to develop a plan to implement a value-based purchasing program for 
Medicare that beginning in fiscal year 2009 would adjust payments to 
hospitals based on factors related to the quality of care they provide. 

My statement today provides information on (1) how hospitals collect 
and submit quality data to CMS and (2) how CMS works to ensure the 
reliability of the quality data submitted by hospitals. 

My statement is based primarily on findings from our two reports on 
hospital quality data.[Footnote 4] In April 2007, we reported on case 
studies that we conducted at eight individual acute care hospitals, 
which were participating in the APU program, in order to obtain 
information about the processes they used to collect and submit quality 
data to CMS. As we noted in our report, because our evidence was 
limited to the eight case studies, we cannot generalize to acute care 
hospitals across the country. In January 2006, we reported on the 
reliability of publicly reported information on hospital quality 
obtained through the APU program that included a review of CMS 
documents and interviews with CMS officials. We also reviewed CMS's 
November 21, 2007, report to Congress which discusses options to 
implement a value-based purchasing program.[Footnote 5] All the work 
for our two reports on hospital quality data was done in accordance 
with generally accepted government auditing standards. We conducted 
this performance audit from February to March 2008, in accordance with 
generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings 
based on our audit objectives. We believe that the evidence obtained 
provides a reasonable basis for our findings based on our audit 
objectives. 

In summary, in April 2007, we reported that the eight case study 
hospitals we visited used six steps to collect and submit quality data, 
two of which involved complex abstraction--the process of reviewing and 
assessing all relevant pieces of information in a patient's medical 
record to determine the appropriate value for each data element. 
Several factors account for the complexity of the abstraction process, 
including the content and organization of the medical record, the scope 
of information and clinical judgment required for certain data 
elements, and frequent changes by CMS in its data specifications. Our 
case studies also showed that existing IT systems can help hospitals 
gather some quality data but are far from enabling hospitals to 
automate the abstraction process. In January 2006 we reported that CMS 
had a process in place to assess the accuracy of the APU program data 
submitted by hospitals, but had no ongoing process to assess the 
completeness of those data. 

Hospitals Use Six Steps to Collect and Submit Quality Data and IT 
Systems Can Help: 

In our April 2007 report,[Footnote 6] we found that whether patient 
information was recorded electronically, on paper, or as a mix of both, 
all eight of the case study hospitals collected and submitted their 
quality data by carrying out six sequential steps: (1) identify 
patients for whom the quality data should be submitted, (2) locate 
needed information in the medical records, (3) determine the 
appropriate value for each data element, (4) transmit the data to CMS, 
(5) review reports to ensure acceptance of the data by CMS, and (6) 
supply copies of selected medical records to CMS for data validation. 

The description by hospital officials of the processes they used to 
collect and submit quality data indicated that steps 2 and 3 (locating 
the relevant clinical information and determining appropriate values 
for the data elements), which involve the process of abstraction, were 
the most complex steps of the six identified, due to several factors. 
The first complicating factor was that the information abstractors 
[Footnote 7] needed to determine the correct data element values for a 
given patient was generally located in many different sections of the 
patient's medical record. Much of the clinical information needed was 
found in the sections of the medical record prepared by clinicians. 
Often the information in question, such as contraindications for 
aspirin or beta blockers, could be found in any of a number of places 
in the medical record where clinicians made entries. As a result, 
abstractors frequently had to read through multiple parts of the 
record--sometimes the entire record--to find the information needed to 
determine the correct value for just one data element. 

The second factor was related to the scope of the information required 
for certain data elements. Some of the data elements that the 
abstractors had to fill in represented a composite of related data and 
clinical judgment applied by the abstractor, not just a single discrete 
piece of information. Such composite data elements typically were 
governed by complicated rules for determining the clinical 
appropriateness of a specific treatment for a given patient. 

The third factor was the necessity abstractors at the case study 
hospitals faced to adjust to frequent changes in the data 
specifications set by CMS. For example, from fall 2004 through summer 
2006, roughly every 3 months hospital abstractors had to stop and take 
note of what had changed in the data specifications and revamp their 
quality data collection procedures accordingly. Some of these changes 
reflected modifications in the quality measures themselves. CMS changed 
its schedule for issuing revisions to its data specifications from 
every 3 months to every 6 months. 

All the case study hospitals found that, over time, they had to 
increase the amount of staff resources devoted to abstracting quality 
data for the CMS quality measures, most notably as the number of 
measures on which they were submitting data expanded. Officials at the 
case study hospitals generally reported that the amount of staff time 
required for abstraction increased proportionately with the number of 
conditions for which they reported quality data. For example, as the 
hospitals began to report on the surgical quality measures, they found 
that the staff hours needed for this new set of quality measures were 
directly related to the number of patient medical records to be 
abstracted and the number of data elements collected. In other words, 
they found no "economies of scale" as they expanded the scope of 
quality data abstraction. Officials at the case study hospitals 
estimated that the amount of staff resources devoted to abstracting 
data for the CMS quality measures ranged from 0.7 to 2.5 full-time 
equivalents (FTE),[Footnote 8] typically registered nurses. On the 
other hand, officials at the case study hospitals reported that the 
demands that quality data collection and submission placed on their 
clinical staff resources were offset by the benefits that they derived 
from the resulting information on their clinical performance. Each one 
had a process for tracking changes in their performance over time and 
providing feedback to individual clinicians and reports to hospital 
administrators and trustees. 

We found that the existing IT systems in the case study hospitals could 
facilitate the collection of quality data, but that there were limits 
on the advantages that the systems could provide. IT systems, and the 
electronic records they support, offered hospitals two key benefits: 
(1) improving accessibility to and legibility of the medical record, 
and (2) facilitating the incorporation of CMS's required data elements 
into the medical record. On the other hand, the limitations that 
hospital officials reported in using existing IT systems to collect 
quality data stemmed from having a mix of paper and electronic systems; 
the prevalence of data recorded in IT systems as unstructured 
paragraphs of narrative or text, as opposed to discrete data fields 
reserved for specific pieces of information; and the inability of some 
IT systems to access related data stored on another IT system in the 
same hospital. All the case study hospitals were working to expand the 
scope and functionality of their IT systems, but most officials at the 
case study hospitals viewed full-scale automation of quality data 
collection and submission through implementation of IT systems as, at 
best, a long-term prospect. 

CMS Has Processes for Ensuring Accuracy but Has No Ongoing Process for 
Ensuring Completeness of Quality Data: 

We reported in January 2006[Footnote 9] that CMS had processes for 
assessing the accuracy of the quality data submitted by hospitals for 
the APU program, but had no ongoing process in place to assess the 
completeness of those data. To check accuracy, one CMS contractor 
electronically checks the data as they are submitted to the clinical 
warehouse. Another contractor conducts an independent audit by 
comparing the quality data submitted by a hospital from the medical 
records for a sample of five patients per quarter for each hospital to 
the quality data that the contractor reabstracts from the same medical 
records. The data are deemed to be accurate if there is 80 percent or 
greater agreement between these two sets of results, which allows the 
hospital to receive the full payment update from Medicare. However, we 
also reported that CMS's determination as to whether hospitals met the 
accuracy standard was statistically uncertain for some hospitals 
because of the small number of records examined--five per quarter per 
hospital, regardless of the hospital's size. Further, CMS did not have 
an ongoing process for assessing the completeness of quality data 
submitted by hospitals. Because of the purposes for which these data 
may be used, there could be an incentive for hospitals to selectively 
report data on cases that score well on the quality measures. 

In our 2006 report we recommended that CMS take steps to improve its 
processes for ensuring the accuracy and completeness of the hospital 
quality data and CMS agreed the process needed to be improved. For 
fiscal year 2008 and subsequent years it required that hospitals attest 
each quarter to the completeness and accuracy of their data, including 
the volume of data, submitted to the clinical warehouse.[Footnote 10] 
Further, in its 2007 report to Congress that lays out a plan to 
implement a value-based purchasing program, CMS recognized the need to 
redesign the data infrastructure and validation process to support a 
value-based purchasing program, by, for example, increasing the number 
of patient medical records sampled from selected hospitals. 

For more information regarding this statement, please contact Linda T. 
Kohn at (202) 512-7114 or [email protected]. Contact points for our Offices 
of Congressional Relations and Public Affairs may be found on the last 
page of this statement. Krister Friday, Shannon Slawter Legeer, and 
Eric Peterson made key contributions to this statement. 

[End of section] 

Footnotes: 

[1] The Medicare Prescription Drug, Improvement, and Modernization Act 
of 2003 created a financial incentive for hospitals, and CMS 
established the Reporting Hospital Quality Data for Annual Payment 
Update (RHQDAPU) Program (the "APU program") to implement that 
incentive. See Pub. L. No. 108-173, ï¿½ 501(b), 117 Stat. 2066, 2289-90. 
Most acute care hospitals (i.e., those paid under the Medicare 
inpatient prospective payment system) receive an annual payment update 
that increases the standardized payment amount that Medicare pays them 
per patient, based on projected increases in hospital operating 
expenses. 

[2] See Pub. L. No. 109-171, ï¿½ 5001(a), 120 Stat. 4, 28-29. 

[3] The magnitude of the reduction in the annual payment update for 
hospitals not submitting the quality data rose from 0.4 percentage 
points to 2 percentage points, starting in fiscal year 2007. 

[4] See GAO, Hospital Quality Data: HHS Should Specify Steps and Time 
Frame for Using Information Technology to Collect and Submit Data, 
GAO-07-320 (Washington, D.C.: Apr. 25, 2007) and Hospital Quality Data: 
CMS Needs More Rigorous Methods to Ensure Reliability of Publicly 
Released Data, GAO-06-54 (Washington, D.C.: Jan. 31, 2006). 

[5] Centers for Medicare & Medicaid Services, Report to Congress: Plan 
to Implement a Medicare Hospital Value-Based Purchasing Program (Nov. 
21, 2007). 

[6] See GAO-07-320. 

[7] Throughout this statement, we use the term "abstractor" to indicate 
hospital staff who are trained to follow a detailed protocol in order 
to extract specified information in a consistent fashion from the 
medical records of multiple patients. 

[8] These represent the FTEs devoted specifically to quality data 
collection and submission. Hospital officials noted that additional 
FTEs were involved in analyzing the hospital's performance on the 
quality measures and achieving improvements through changes in clinical 
process and educational efforts with the hospital's clinicians. 

[9] See GAO-06-54. 

[10] See 72 Fed. Reg. 47130, 47364 (Aug. 22, 2007). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***