Health Care Reform: "Report Cards" Are Useful but Significant Issues Need
to Be Addressed (Chapter Report, 09/29/94, GAO/HEHS-94-219).
As part of the debate over health care reform, Congress is considering
requiring health plans to provide prospective purchasers with
information on the quality of care they furnish. Presumably, purchasers
will use such "report cards" to compare health plans and choose one that
provides the desired level of quality and price. Although report cards
that compare the performance of competing health care plans could be a
positive step in preserving quality and lowering costs, experts disagree
about the type and amount of information to be published because such
data may not be reliable or valid. Some experts believe that usable
report cards can be produced within two to five years if the indicators
are limited to those known to be valid and reliable. Others believe that
it will be as long as 15 years before highly reliable and valid measures
are developed. Several states and groups such as United HealthCare
Corporation and Kaiser Permanent Northern California Region have already
issued report cards on the care they furnish, but no studies have been
done on the cards' validity or reliability. To overcome obstacles to
using report cards, most experts recommend that (1) the federal
government standardize indicators and the formulas for calculating
results and (2) an independent third party verify data before they are
published.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-94-219
TITLE: Health Care Reform: "Report Cards" Are Useful but
Significant Issues Need to Be Addressed
DATE: 09/29/94
SUBJECT: Publications
Quality assurance
Health insurance
Health care cost control
Health services administration
Health care programs
Evaluation methods
Data collection operations
Information dissemination operations
Data integrity
IDENTIFIER: New York
Pennsylvania
California
HCFA Common Procedure Coding System
Federal Employees Health Benefits Program
HHS Healthy People 2000 Program
Clinton Health Care Plan
National Health Care Reform Initiative
Health Security Act
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Chairman, Committee on Labor and Human Resources,
U.S. Senate
September 1994
HEALTH CARE REFORM - "REPORT
CARDS" ARE USEFUL BUT SIGNIFICANT
ISSUES NEED TO BE ADDRESSED
GAO/HEHS-94-219
Health Care Report Cards
Abbreviations
=============================================================== ABBREV
AHCA - Agency for Health Care Administration
CABG - coronary artery bypass graft
CHPA - Community Health Purchasing Alliances
CPT-4 - Current Procedural Terminal Coding System, Fourth Edition
DPT - diphtheria/pertussis/tetanus
DRG - diagnostic related groups
DSM-III - Diagnostic and Statistics Manual, Third Edition
GHAA - Group Health Association of America
HAP - Hospital Association of Pennsylvania
HEDIS - Health Plan Employer Data and Information Set
HCFA - Health Care Financing Administration
HMO - health maintenance organization
ICD-9-CM - International Classification of Diseases, Ninth
Revision, Clinical Modification
NCQA - National Committee on Quality Assurance
NICU - neonatal intensive care unit
PRO - peer review organization
OTA - Office of Technology Assessment
UHC - United HealthCare Corporation
VA - Department of Veterans Affairs
Letter
=============================================================== LETTER
B-255717
September 29, 1994
The Honorable Edward M. Kennedy
Chairman, Committee
on Labor and Human Resources
United States Senate
Dear Mr. Chairman:
In response to your request, we have conducted a study of health care
performance measures and how they should be developed so that quality
of care "report card" comparisons can be made between various health
care plans. We found that most experts believe measures comparing
health plan performance should be published. However, inaccurate and
misleading data sources as well as the lack of agreed-upon indicators
and formulas for calculating results might hinder the report cards'
usefulness.
Some experts recommend delaying publication of measures until the
problems are resolved. Other experts believe that selected
indicators thought to be reliable and valid should be published
immediately, otherwise the associated problems might never be
overcome. Regardless of their point of view, most experts
recommended that (1) an independent third party verify data before
publication and (2) the federal government play a role in
standardizing indicators and the formulas for calculating results.
To conduct our study, we performed an extensive literature search and
interviewed over 30 health care experts (see pp. 14-15).
We are sending copies of this report to appropriate congressional
committees and other interested parties. We will also make copies
available to others upon request.
This report was prepared under the direction of David P. Baine,
Director, Federal Health Care Delivery Issues. If you have any
questions, please call him at (202) 512-7101. Other major
contributors to this report are listed in appendix II.
Sincerely yours,
Janet L. Shikles
Assistant Comptroller General
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
Health reform proposals the Congress is considering include
provisions requiring health plans to collect and report to
prospective purchasers information about the quality of care they
furnish. Presumably, purchasers will use such "report cards" to
compare health plans and select one that provides the desired level
of quality and price. Assessing health care quality is not an
innovative concept. But evaluating the quality of care using certain
indicators and publishing the information in a manner that allows
consumers to compare health plans is new. If report card development
is not approached thoughtfully and carefully, individual consumers,
corporate purchasers, health plans, and individual providers might be
misled by inaccurate information.
In January 1994, the Chairman of the Senate Committee on Labor and
Human Resources asked GAO to undertake a study of health performance
measures and how they should be developed so that quality of care
comparisons can be made between various health care plans. The
Chairman asked GAO to report on (1) approaches presently being taken;
(2) opportunities to misrepresent or misreport the data; and (3)
potential uses of report cards by consumers, providers, and health
plans.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
Health care quality assessment efforts have been under way for
several decades. These assessments have been performed by state and
federal inspectors, private accrediting agencies, and health care
organizations themselves. But specific results of these activities
have been kept confidential. Other than personal attestations or
knowing that an organization was accredited or licensed, corporate
and individual health care purchasers had no method for determining
which organization provided the best health care.
The first widespread public disclosure of quality assessment
information took place in 1987 when the Health Care Financing
Administration (HCFA) released its report on the observed and
expected mortality rates in each hospital that performed coronary
artery bypass graft (CABG) surgery. Newspapers published the data
and ranked hospitals according to the "best" and the "worst." HCFA
continued to publish yearly CABG mortality figures through 1992.
In the mid-1980s, health policy experts advised corporate purchasers
that health care costs could be contained if purchasers considered
both cost and quality of care information when they made their health
care purchases. Corporate purchasers took this advice and began
seeking ways to evaluate health care quality to ensure they were
getting the most value for their dollars. These early efforts led to
corporate purchaser coalitions working with health care
organizations, primarily managed care plans, to develop utilization
and quality of care information purchasers believed they needed to
make informed purchasing decisions. However, these efforts
progressed slowly as providers and purchasers tried to agree on what
performance indicators would be useful.
Recently state and federal officials have determined that publishing
quality of care results could contain health care expenditures. This
conclusion has resulted in health care reform proposals that require
reporting of this information. Published summaries of health plan
performance indicators are being called report cards. As a result of
state legislative mandates, several states have begun publishing
hospital report cards. Being proactive, several health plans have
released report cards about care provided in their organizations.
Indicators that have been included are items such as mammography
rates, immunization rates, lung cancer survival rates, results of
patient satisfaction surveys, and claims turnaround time. (See ch.
2 for report card examples.)
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
Report cards can be a useful tool to educate stakeholders about the
health care provided by a health plan. Most experts believe that
publishing the results of cost and quality indicators as well as
other descriptive information comparing the performance of competing
health plans is a positive step toward preserving the quality while
lowering the cost of health care. However, experts disagree about
the type and amount of information to be published in a report card
because the data sources and indicators used may not be reliable or
valid. Furthermore, the report cards being developed may not reflect
the needs of some users, and few cost estimates for publishing report
cards are available.
Some experts believe that usable report cards can be produced within
2 to 5 years if the indicators are limited to those known to be valid
and reliable. Furthermore, they believe that data sources will
become more accurate if the information is used for reporting. Other
experts disagree. These experts believe that it will be 10 to 15
years before highly reliable and valid measures are developed.
Several states such as New York, Pennsylvania, and California and
organizations such as United HealthCare Corporation, Kaiser
Permanente Northern California Region, and U.S. Healthcare have
released report cards on the care they furnish. But no evaluative
studies have been conducted to determine the report cards' validity
or reliability. Current obstacles to using health care report cards
include (1) inaccurate, misleading, or incomplete information
sources; (2) indicators that may not measure quality; (3) little
agreement on formulas for calculating performance results; and (4) no
verification mechanisms in place to ensure the accuracy of reported
results. To address these obstacles, most experts recommended that
(1) the federal government play a role in standardizing indicators
and the formulas for calculating results and (2) an independent third
party verify data before they are published.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
VARIOUS STAKEHOLDERS ARE
DEVELOPING REPORT CARDS
-------------------------------------------------------- Chapter 0:4.1
Several health plans and governmental entities have released or plan
to release report cards to employers, the general public, or both.
Initial responses from corporate purchasers have been positive.
However, little information is available on how useful the report
cards are to other possible audiences. (Ch. 2 summarizes some of
the most publicized report card efforts.) GAO did not evaluate the
reliability and validity of these report cards nor did GAO find
evidence that others have conducted such evaluative studies.
EXPERTS DISAGREE ABOUT WHAT
A REPORT CARD SHOULD INCLUDE
-------------------------------------------------------- Chapter 0:4.2
Experts differ about how much information a report card should
include. This situation exists because providers, researchers, and
other experts have differing opinions about whether data sources are
accurate and which indicators measure quality. Some experts think
publishing data that might later prove to be incorrect will destroy
public confidence in report cards. These experts predict that it
will take from 10 to 15 years to produce a reliable and valid report
card. Other experts stated that the problems with report cards will
never be overcome unless health plans begin to publish report cards
now. These experts believe that good report cards can be produced
within 2 to 5 years.
REPORT CARDS MAY BE BASED ON
INACCURATE, MISLEADING, OR
INCOMPLETE INFORMATION
-------------------------------------------------------- Chapter 0:4.3
Information used to evaluate performance is generally obtained from
computerized administrative databases or medical records. But these
sources may contain incorrect, misleading, or incomplete data.
Because few studies have been done to determine the accuracy of data
sources, no one knows the magnitude of the problem. If the accuracy
problems are extensive, the deficiencies will make report card
results unreliable.
Administrative databases can be created using encounter forms
providers submit to the health plan. However, in a health
maintenance organization (HMO) little incentive exists to submit
these forms; because HMOs are prepaid health care plans, payment for
services is not dependent on submitting encounter forms. As a
result, the databases in those organizations may be incomplete.
Administrative databases also are created using information from
non-HMO claim forms physicians submit to health plans and other
insurers for payment of services rendered. Organizations that pay
providers based on claims submitted for services rendered use coding
systems to record diagnoses, conditions, procedures, and treatments.
However, the clinical definitions of these terms may not be clear and
coding errors result. Thus, information in the database can be
incorrect or misleading.
Medical records usually are a good source for clinical information.
However, it costs approximately $16 to retrieve data from one medical
record. Furthermore, some experts believe that medical records may
not always be accurate and complete for report card purposes because
(1) physicians sometimes err in their diagnoses or interpretations
and test results may be inaccurate, (2) providers do not always
remember to note all findings or care rendered, and (3) providers may
exaggerate medical record notations to justify a hospital stay or
satisfy insurance benefit requirements for an outpatient test or
procedure.
MEASURES SELECTED MAY NOT
REFLECT QUALITY
-------------------------------------------------------- Chapter 0:4.4
Report cards may not be measuring what they are intended to measure.
For example, one measure of quality used by many health plans and
included in report cards is the number of board-certified physicians
on staff.\1 Presumably, the more board-certified physicians a plan
has on its staff, the better the health care will be. But research
has not shown conclusively that board-certified physicians furnish
better care.
Although efforts to agree on which indicators measure quality have
been ongoing, no consensus has been reached. Many experts believe
that only indicators measuring the outcome of services rendered
should be included in report cards. However, these outcomes must be
adjusted to ensure that patient characteristics (for example, age,
concurrent diseases) do not affect the outcomes. Although a number
of adjustment systems are in use, experts have not yet proven them to
be highly valid and reliable.
--------------------
\1 To become board certified, a physician must pass a test in the
specialty area and meet other requirements established by experts in
that medical specialty.
STANDARDIZED FORMULAS FOR
CALCULATING RESULTS HAVE NOT
BEEN DEVELOPED
-------------------------------------------------------- Chapter 0:4.5
No uniform standards exist for collecting and computing the indicator
results used in report cards. However, if report card comparisons
are to be meaningful, uniform information must be obtained from data
sources. Currently, the developers of the Health Plan Employer Data
and Information Set (HEDIS) 2.0 are the only group that has created
uniform criteria and formulas for calculating results. Until uniform
standards for data gathering and reporting are accepted generally by
all report card developers, report card results cannot be compared.
Thus, purchasers will not be able to determine which plan provides
the care that meets their needs. Some experts believe that the
federal government could contribute in this area by establishing
consistent collection and calculation standards.
REPORT CARD RESULTS ARE NOT
VERIFIED
-------------------------------------------------------- Chapter 0:4.6
To ensure that health plan administrators are reporting accurate and
uniform data, most experts agree that report card information should
be verified by an independent party. Currently, however, independent
verification is planned or is done in only a few instances. No
consensus has been reached about who should perform that task and who
should pay for the service. Possibilities suggested include
independent audits performed by private organizations or state or
federal entities. The costs of these services might be borne by the
health plan or passed along to the consumer.
DIFFERENT STAKEHOLDERS GAIN
DIFFERENT BENEFITS FROM
REPORT CARDS
-------------------------------------------------------- Chapter 0:4.7
Various stakeholders can derive different benefits from report cards
if the report cards reflect their areas of interest. Corporate
purchasers could select health plans offering the greatest value.
Public health officials could realize greater progress toward meeting
national public health goals. Providers could determine which health
plans they want to associate with. Health plan administrators and
researchers could identify areas for improvment. And individual
consumers could select health plans that meet their needs. However,
individual consumers have had minimal input into selecting report
card indicators and little is known about their needs or interests.
As a result, their needs may not be met.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
This report was intended to describe the report card development
process and thus contains no recommendations.
INTRODUCTION
============================================================ Chapter 1
In January 1994, the Chairman of the Senate Committee on Labor and
Human Resources requested that we study health care performance
measures and how they should be developed so that quality of care can
be compared among various health care plans. The Chairman requested
that specific areas of discussion be included in this report: (1)
approaches presently being taken; (2) opportunities to misrepresent
or misreport data; and (3) potential uses of report cards by
consumers, providers, and health plans.
EVOLUTION OF THE REPORT CARD
CONCEPT
---------------------------------------------------------- Chapter 1:1
Most health care reform proposals being considered by the Congress
and various state legislatures include provisions requiring health
plans to publish information about the quality of care they provide.
These publications are referred to in the news media as "report
cards." Report cards can include both clinical and administrative
performance measures such as the mortality rates for patients who had
coronary artery bypass graft (CABG) surgery, childhood immunization
rates, cancer survival rates, results of patient satisfaction
surveys, and claims turnaround time. Presumably, purchasers will use
these report cards to choose the health plan with the best
performance for the least cost. (See ch. 2 for examples of current
report cards.)
Widespread professional interest in monitoring the quality of health
care services began after World War II and was focused primarily on
hospitals. Passage of federal Medicare legislation in 1965 and the
Joint Commission on Accreditation of Healthcare Organizations'\2
mandate in the early 1970s that hospitals operate an internal quality
assurance program to be accredited focused further attention on the
area. But the results of these quality assessment efforts were
usually considered confidential and only for the use of government
regulators, accreditors, and internal quality management personnel.
Information was rarely distributed to the public.
In 1987, hospital mortality data were released nationwide. In 1986,
as part of its Medicare oversight responsibilities, the Health Care
Financing Administration (HCFA) analyzed data from all hospitals
about the number of deaths that occurred within 180 days of hospital
admission after Medicare beneficiaries had CABG surgery. This
information was to be used only by peer review organizations (PRO)
and hospitals for quality assessment purposes.\3
HCFA assumed that a quality of care problem might be present if a
hospital's observed mortality rate was higher than expected when
considering the patients' conditions. However, the news media
obtained the results of the individual hospitals' performance by
filing a Freedom of Information Act request. The media ranked
hospitals from "best" to "worst" on the basis of observed and
expected mortality rates even though HCFA advised against doing so.
HCFA officials continued to release CABG mortality rates until 1993
when they discontinued the practice citing problems with the
reliability of their methods for adjusting the data to account for
the influence of patient characteristics on the outcomes.
Since the mid-1980s, corporate purchasers have been the driving force
behind efforts to obtain information from hospitals and health plans
about the quality of care they furnish. Before that time, large
corporate health care purchasers simply paid the bills submitted to
them from health care providers without evaluating the quality of the
services received. However, as health care costs began to escalate,
purchasers--primarily corporate leaders--started to think about
value, that is, the quality of health care services in relation to
price. Furthermore, purchasers and patients became concerned that
health care organizations might withhold needed services to hold down
costs. As a result, corporate efforts focused on obtaining quality
of care and utilization information from hospitals and health
maintenance organizations (HMO). Many hospitals were ambivalent
about releasing the information because they believed it might be
misinterpreted. HMOs were unable to provide the information because
they did not have the data readily accessible. However, corporate
purchasers continued their efforts and convinced state and federal
legislators that publishing comparative information would be
beneficial. As a result, state legislation exists mandating
publication of such information, and other provisions requiring
publication of data are being considered nationwide at both the state
and federal levels.
--------------------
\2 The Joint Commission is a national, private, not-for-profit
accrediting organization with expertise in establishing quality
assurance standards.
\3 PROs were created by the Congress to ensure that services rendered
through Medicare are necessary, appropriate, and of high quality
(Peer Review Improvement Act, title I, subtitle C of the Tax Equity
and Fiscal Responsibility Act of 1982 [P.L. 97-248]). They are
private, generally statewide organizations that must be composed of
at least 10 percent of the licensed physicians practicing in the
area.
QUALITY OF CARE IS DIFFICULT TO
DEFINE
---------------------------------------------------------- Chapter 1:2
Quality health care has been difficult for experts to define. Most
experts now agree that quality includes measuring attributes related
to appropriateness--providers giving the right care at the right
time; accessibility--patients being able to obtain care when needed;
and acceptability--patients being satisfied with the care. These
attributes are measured using indicators that represent the (1)
structure of care--resources and organizational arrangements in place
to deliver care; (2) process of care--physician and other provider
activities carried out to deliver the care; and (3) outcomes of
care--the results of physician and provider activities. Although
most experts believe that outcomes are the best measure of quality,
outcomes are the most difficult to interpret because patient
characteristics such as age or prior health status might affect them
significantly. Unless outcomes are adjusted to determine whether
patient characteristics or quality of care affected the results,
conclusions about quality based on an evaluation of outcomes might be
erroneous.\4
Indicators used to measure quality must be reliable, valid, or
comparable if they are to be useful. If an indicator is reliable,
the same result should be obtained regardless of who measures the
result or when the measurement is taken. If an indicator is valid,
it should measure what is intended. If indicators are comparable
with one another, both indicators should measure identical
attributes.
--------------------
\4 These adjustments are referred to as risk adjustments, a term that
can refer to two concepts. It can be used to describe methods for
(1) determining whether patient characteristics or poor quality are
responsible for undesirable patient outcomes and (2) predicting
resource consumption and cost based on patient characteristics.
Generally, when using the term "risk adjustment," we will be
referring to the former. However, some experts have attempted to use
the latter methods to predict patient outcomes.
SCOPE AND METHODOLOGY
---------------------------------------------------------- Chapter 1:3
In conducting our study of report card development, we undertook an
extensive review of the literature on performance measures and
ensuring quality health care. In addition to many journal
publications, we reviewed documents prepared by the Agency for Health
Care Policy and Research, California Office of Statewide Health
Planning and Development, HCFA, Institute of Medicine, Jackson Hole
Group, John A. Hartford Foundation, Massachusetts Healthcare
Purchaser Group, Michigan Peer Review Organization, National
Committee for Quality Assurance, and the Office of Technology
Assessment.
We also interviewed health care experts representing the following
organizations: the American Association of Retired Persons, American
Medical Association, Assurqual, Inc., Beth Israel Hospital, Brandeis
University, Center for Health Care Policy and Evaluation (a
subsidiary of United HealthCare Corporation of Minneapolis), Codman
Research Group, Inc., Consumers' Checkbook, Florida Agency for Health
Care Administration, Foundation for Health Care Quality, George
Washington University Medical Center, Group Health Cooperative of
Puget Sound, Harvard School of Public Health, HCFA, Health Institute
of the New England Medical Center Hospitals, Health Pages, Jefferson
Medical College, Joint Commission on Accreditation of Healthcare
Organizations, Kaiser Permanente Northern California Region, Managed
Health Care Association, Maryland Health Care Access and Cost
Commission, Minneapolis Citizens League, National Association of
Health Data Organizations, National Committee for Quality Assurance,
Pennsylvania Health Care Cost Containment Council, U.S. Algorithms
(a subsidiary of U.S. Healthcare of Pennsylvania), and U.S.
Department of Health and Human Services.
Additionally, we reviewed a sample of proposed and distributed report
cards from many states and organizations, including the states of
California, Florida, Minnesota, New York, and Pennsylvania, and from
Kaiser Permanente Northern California Region, the Massachusetts
Healthcare Purchaser Group, Michigan PRO, U.S. Healthcare of
Pennsylvania, and United HealthCare Corporation of Minneapolis.
However, we did not evaluate the validity, reliability, or
readability of those report cards.
Developments related to report cards are being reported daily. Most
of our work was performed between November 1993 and July 1994 in
accordance with generally accepted government auditing standards.
The report was updated to reflect more recent happenings as
appropriate.
VARIOUS HEALTH CARE STAKEHOLDERS
ARE DEVELOPING REPORT CARDS
============================================================ Chapter 2
Health plans and governmental entities have undertaken initiatives to
inform the public about the quality of care hospitals and health
plans furnish. Pennsylvania, New York, and California have published
report cards about certain services provided by hospitals in their
states. The National Committee for Quality Assurance (NCQA), an
organization that accredits HMOs, is coordinating a 1-year pilot
study in which 21 health plans are collecting data according to
specifications developed by a committee of health plan
representatives and corporate purchasers. At the conclusion of this
effort in December 1994, a report card will be distributed for
consumer use. In addition, some health plans have created and
distributed their own report cards. Florida is developing a report
card for health plans providing services to certain health plan
enrollees as part of its health reform initiative. Furthermore, HCFA
is in the process of developing performance measures it expects PROs
to use for quality assessment purposes. Although this effort is not
intended to result in a report card released to the public, HCFA's
efforts will result in a report card to be used by the PROs.
REPORT CARDS ARE AVAILABLE IN
THREE STATES ON SELECTED
HOSPITAL SERVICES
---------------------------------------------------------- Chapter 2:1
Since 1991, three states--New York, Pennsylvania, and California--
have released report cards to the general public about certain
hospital services provided in their states. In 1993 and 1994,
Pennsylvania released report cards on the care furnished in 175
hospitals to patients grouped into 53 diagnostic categories.\5
Additionally, in 1990 and 1992, New York\6 and Pennsylvania,
respectively, released report cards on hospitals and physicians in
their states performing CABG surgery. In 1993, California published
information about the outcomes of patients admitted with acute
myocardial infarctions and those who underwent back surgery. The
states collected this information because of concerns about the
quality of health care being provided. Pennsylvania also was
concerned about the cost of care. The following discussion focuses
on how Pennsylvania developed its report card system.
In the early 1980s, Pennsylvania employers were considering cutting
back employee health care benefits because of their rising costs.
Labor unions, a strong and influential force in Pennsylvania, were
fighting this action and sometimes went on strike to prevent these
cutbacks. To resolve the problem, corporate purchasers and labor
union officials began discussing with state officials the concept of
cost controls. Health providers resisted the idea. As a compromise,
each of the stakeholders agreed to support legislation intended to
encourage health care competition based on the principles of cost and
quality. This legislation, the Health Care Cost Containment Act,
passed in 1986.
As a result of the 1986 legislation, the Pennsylvania Health Care
Cost Containment Council was created to promote cost containment and
to identify appropriate utilization practices. Council membership
consists primarily of representatives from the business community and
organized labor. However, a consumer, a hospital administrator,
insurers, providers, and state appointees are also members. Through
quarterly reports published by the Council, current and accurate data
and information on both the cost and quality of certain health care
services are to be made available to all purchasers and consumers of
health care. However, because its resources are limited, the Council
is concentrating its efforts on hospital inpatient care.
In 1993, the Council published the Hospital Effectiveness Report on
care provided in 175 Pennsylvania hospitals for each of 53 diagnostic
categories during 1991.\7 A second report about care provided in 1992
was published in 1994. For each of the 175 hospitals, data were
provided about the number of patients admitted, the average severity
of illness of those patients when admitted, the percentage of
patients aged 65 and older, the actual and expected number of deaths
and complications, the average length of stay, and the hospital's
average charge per patient. (See fig. 2.1.) Consumers were
encouraged to take the report with them when visiting their health
care providers and to ask questions about the information in it.
Figure 2.1: Excerpt From a
Report Card Published by the
Pennsylvania Health Care Cost
Containment Council
(See figure in printed
edition.)
Notes: For the purposes of illustration, this excerpt provides
information for only five Pennsylvania hospitals. The Council has
published this same information for all hospitals in the state that
admitted patients classified in this DRG as well as 52 other DRGs.
If the "Comments" column were checked, it would mean that the
hospital elected to comment on the information. These comments are
available to the public upon request.
"Admission Severity," rated on a scale of 0 to 4, is a measure of the
degree of risk of heart, lung, or kidney failure during a patient's
first 2 hospital days. The "average" admission severity is the
average of all patients admitted to a particular hospital. A score
of 4 does not mean that patients are twice as sick as those at a
hospital with a score of 2. It does mean that on average, the first
hospital's patients are sicker when admitted than the second
hospital's.
Source: Pennsylvania Health Care Cost Containment Council, Hospital
Effectiveness Report, HE 9-91 Volume 4, September 1993.
Information included in the Hospital Effectiveness Report is derived
from quarterly statements hospitals submit to the Council according
to Council-defined data requirements. Upon receipt, Council staff
review the information to ensure that obvious errors have not been
made. Any discrepancies are called to the hospital's attention and
all information is returned to the hospital for correction and
validation. The Council's first report, published in 1993, was
released 7 months after the hospitals submitted corrected data.
However, the second report, released in 1994, was published 14 months
after the Council obtained corrected hospital information. A Council
spokesperson said that the delay in issuing the 1994 report was the
result of budget cuts\8 and priority given to the CABG report
discussed in the next paragraph.
The Council released a Consumer Guide to Coronary Artery Bypass Graft
Surgery in November 1992 and February 1994. The 1994 report provides
information about hospital charges and mortality rates in 1991 as
well as surgeon-specific mortality statistics. For example, it
revealed that of the 35 hospitals performing CABG surgery in
Pennsylvania, 3 had fewer deaths than expected and 32 had deaths
similar to the expected number. In addition, for each hospital a
chart lists the total number of patients having CABG surgery, the
number who died, the expected range of patients who might die, and
the hospital's average charge. (See fig. 2.2.) This information,
excluding charges, also is provided for the physician practice groups
and individual surgeons affiliated with each hospital. As of May
1994, approximately 12,000 copies of the 1992 report and 3,000 copies
of the 1994 report had been distributed.
Figure 2.2: Example of Report
Card Data Published by the
Pennsylvania Health Care Cost
Containment Council
(See figure in printed
edition.)
Note: In its November 1992 report of the 35 hospitals that performed
CABG surgery in Pennsylvania during 1990, 4 hospitals had fewer
deaths than expected, 7 had more deaths than expected, and the
remaining 24 hospitals had a number of deaths in the expected range.
However, the Council did not provide information to allow comparison
between years.
Source: Pennsylvania Health
Care Cost Containment Council,
A Consumer Guide to Coronary
Artery Bypass Graft Surgery,
Volume II, 1994.
(See figure in printed
edition.)
Two organizations have estimated the hospital costs associated with
reporting data to the Council. The Hospital Association of
Pennsylvania (HAP) estimated that in 1991 hospitals spent $26.5
million, or $14.20 per patient discharge, to collect and report data.
These costs varied according to hospital size, teaching status, and
type of admission and were associated with labor, computer software
and hardware, licensing fees, training, data retrieval, and data
validation. Although hospitals expressed concern about the expense,
a HAP survey found that about 37 percent of hospitals would continue
to collect the data for internal quality assessment regardless of the
legislative mandate. The Pennsylvania State University Center for
Health Policy Research estimated that the average hospital incurred a
cost of approximately $17.43 per discharge because of the legislated
reporting mandates. (See table 2.1 for estimates as they relate to
total dollars, cost per hospital patient day, and cost per hospital
discharge.) Regardless of whether they were in rural or urban areas,
large hospitals spent approximately half the amount per discharge
smaller institutions did because larger hospitals could spread the
costs over more discharges.
Table 2.1
Estimates of 1989 Total Hospital
Expenses to Comply With Data Submission
Requirements
Small Small Large Large
Expenses Total rural urban rural urban
---------------------------------------- ------ ------ ------ ------ ------
Total dollars $110,0 $70,00 $67,00 $102,0 $134,0
00 0 0 00 00
Per patient day $2.75 $3.87 $4.87 $2.01 $1.98
Per patient discharge $17.43 $22.95 $26.65 $12.76 $14.15
Percent of total hospital expenditures 0.36% 0.58% 0.55% 0.31% 0.27%
--------------------------------------------------------------------------------
Source: Pennsylvania State University Center for Health Policy
Research.
--------------------
\5 Examples of the diagnostic categories reported on include DRG
87--pulmonary edema and respiratory failure, DRG 148--major small and
large bowel procedures with complication/comorbidity, DRG
324--urinary stones without complication/comorbidity, and DRG
370--cesarean section with complication/comorbidity. DRG, a
classification system adopted by HCFA, stands for diagnostic related
groups. It sorts all medical conditions and surgical procedures into
492 categories based on the patients' diagnoses.
\6 In 1990, New York released information to a medical journal and
the news media. The state did not release a consumer guide until
1992.
\7 The Council chose to report only on care furnished in Pennsylvania
hospitals having at least 100 beds.
\8 The Council's budget was $2.6 million for fiscal year 1991, $1.6
million for fiscal year 1992, and $1.4 million for fiscal year 1993.
However, it received a $3 million budget for fiscal year 1994--an
increase of $1.6 million.
AN EFFORT IS UNDER WAY TO
DEVELOP UNIFORM REPORT CARD
INDICATORS
---------------------------------------------------------- Chapter 2:2
Health plans, providers, and corporate purchasers working under the
auspices of NCQA published a set of performance measures in November
1993. Efforts to refine these measures are continuing. NCQA also is
coordinating a 1-year pilot study of 21 health plans' ability to
collect data and the related costs. At the completion of the study
in December 1994, a report card for consumer use comparing the 21
plans' performance will be published.
NCQA has been coordinating efforts to create a standardized set of
performance indicators. In 1989, a committee of health plan
administrators, health plan physicians, and corporate purchasers set
out to develop a consensus list of performance measures that could be
used by corporate purchasers to assess health plan value. This led
to the development in November 1993 of what commonly is referred to
today as HEDIS 2.0--the Health Plan Employer Data and Information
Set.\9 When selecting the indicators, the developers considered (1)
the indicators' relevance and value to the employer community, (2)
the ability of health plans to develop and provide the requested data
in a specified manner, and (3) the indicators' potential to improve
the processes of health care delivery. HEDIS 2.0 indicators are
measures of health plans' structure and process. Developers did not
include indicators that directly measure the long-term results or
outcomes of care. They believe that (1) outcomes measurement is not
yet an established field of study and (2) many outcomes may not be
meaningful until a lengthy period has elapsed after an intervention.
However, HEDIS developers expect to include outcomes measures in
future revisions.
Over 60 indicators are included in HEDIS 2.0 that describe
performance in 5 areas: quality, access and patient satisfaction,\10
membership and utilization, finance, and health plan management
activities. These indicators include items such as
childhood immunization rates, low birth-weight rates, and asthma
inpatient admission rates;
the number and percentage of primary care physicians accepting new
patients and percentage of members who are "satisfied" with the
plan according to health plan survey information;
membership enrollment and disenrollment information, inpatient
utilization rates, and frequency and average cost of certain
high-occurrence, high-cost procedures;
financial liquidity, compliance with statutory requirements, and
premium trend information; and
descriptive information about provider recredentialing.
Information needed for HEDIS 2.0 performance measures can be obtained
from either administrative databases or medical records. Guidelines
and specifications have been developed for collecting the data and
calculating performance so that the results of one health plan can be
compared with the results of another. However, the committee
recognizes that the information in administrative databases and
medical records--the primary sources of data for the report card--may
be incomplete or inaccurate. They also know that newly developed
indicator definitions may be unclear. For these reasons, the HEDIS
User Group, comprising approximately 200 organizations, is reviewing
the capability of health plans to collect data that conform to HEDIS
specifications. As a result of this review, several revisions have
been made to HEDIS measures and others have been clarified.
NCQA also is coordinating the efforts of 21 health plans that are
conducting a 1-year pilot study of their ability to collect complete,
accurate data to calculate report card results and the cost
associated with this effort. These health plans selected 21 HEDIS
measures they believe will be of interest to the public. Some of
these measures were clarified and modified when data could not be
collected as specified in HEDIS.\11 In December 1994, at the
conclusion of the study, a report comparing the performance of the 21
health plans and a technical report describing the project will be
published.
--------------------
\9 Initial development efforts were organized by The HMO Group, a
coalition of group and staff model HMOs. The resulting draft
document, HEDIS 1.0, was released in September 1991. In October
1992, NCQA assumed responsibility for that document's revision.
\10 Other than health plan reports about the percentage of surveyed
members who indicated that they were "satisfied with the health
plan," standardized patient satisfaction measures were not included
in HEDIS 2.0. The measures were omitted because of a lack of
consensus regarding an appropriate instrument that could be used
across all types of health plans. However, health plans are
encouraged to display and explain the results of their own patient
satisfaction surveys.
\11 Modifications were communicated to the HEDIS User Group for their
consideration as well.
SOME MAJOR HEALTH PLANS ARE
PUBLISHING REPORT CARDS
---------------------------------------------------------- Chapter 2:3
In 1993, three health plans published report cards on various aspects
of their plans. All three organizations were involved in the
development of HEDIS 2.0, but two of the organizations did not
collect all of their data in accordance with HEDIS 2.0
specifications. Thus, their report card results cannot be compared
with one another. In addition, in 1994, 16 Massachusetts insurers
jointly issued a report card on 6 measures they mutually agreed on.
Developers of this report card, however, urged purchasers not to
compare the results of one plan with another's because little is
known about what constitutes good performance.
In March 1993, United HealthCare Corporation (UHC), an organization
that manages 20 health plans in cities throughout the nation,
released a sample report card that it ultimately issued in final form
on each of its plans in September 1993. (See fig. 2.3.) Indicators
used in UHC's report card are similar to those in HEDIS 2.0 and were
selected by UHC managers based on their interpretation of (1) public,
group purchaser, and government interests; (2) areas where the data
would be scientifically valid and reliable; and (3) areas where there
was clinical agreement regarding treatment. The 28 measures in the
report represent 4 areas of performance: health care quality,
consumer satisfaction, administrative efficiencies, and controlling
costs. Data for these measures were obtained from administrative
data sets, medical records, and patient surveys. Also in the report
card are data to allow comparisons with other reference values such
as national immunization rates published by the Children's Defense
Fund, state pap smear rates for cervical cancer detection published
by the state of Minnesota, and national cesarean section rates
published by the National Center for Health Statistics.
Developers of UHC's report card did not provide an estimate of their
development costs. However, a UHC representative said information
for their report was derived from a previously established internal
quality monitoring system.\12 Using this system, the annual
production of the report card that covers 12 clinical conditions for
each of UHC's health plans requires approximately 8 to 10 weeks. The
UHC representative also stated that the cost of collecting
information for report card development could run into millions of
dollars depending on the computer resources and personnel expertise
within the plan.
Figure 2.3: Example of
Information in a Medica Report
Card
(See figure in printed
edition.)
Notes: Medica is a health plan managed by UHC.
All of the Medica data shown here, with the exception of organ
transplant survival rates, reflect 1992 performance and are derived
from UHC's Center for Health Care Policy and Evaluation quality
screening and management software, and/or Medica's decision support
software, both of which use computerized databases as their sources.
Source: The Center for Health
Care Policy and Evaluation (a
division of United HealthCare
Corp.), Medica Report Card on
Health Care, September 1993.
(See figure in printed
edition.)
Kaiser Permanente Northern California Region and U.S. Healthcare
published report cards in October 1993 and November 1993,
respectively. Kaiser's report card includes over 100 performance
measures, some of which follow HEDIS methodology. (See fig. 2.4.)
Kaiser did not include utilization and financial measures because
Kaiser management did not view them as important quality issues.
Instead, Kaiser chose to measure areas it believes its providers
could influence given the current state of medical knowledge. In
addition to drawing upon HEDIS 2.0, Kaiser reviewed the work of the
RAND Corporation, the Healthy People 2000 report,\13 and other
leaders in quality assessment. Data about provider activities or the
process of care, as well as measures representing the results or
outcomes of care, were included. For example, in addition to
reporting its mammography screening rate, Kaiser reported its rate
for diagnosing breast cancer at various stages of development.
According to Kaiser officials, the stage of cancer at diagnosis
indicates the effectiveness of the screening efforts. Kaiser chose
to collect all of its information from existing quality assessment
databases rather than perform medical record reviews they believe are
expensive. Kaiser also compared its performance to published
national and state reference values. However, Kaiser officials
avoided comparing performance on preventive measures with the
self-reported results of other plans because they believed that the
results of these measures "are usually inflated" when compared with
rates derived from patient records.
Figure 2.4: Example of
Information in a Kaiser
Permanente Report Card
(See figure in printed
edition.)
Source: Kaiser Permanente
Northern California Region,
1993 Quality Report Card.
(See figure in printed
edition.)
The development of the Kaiser report card took approximately 9 months
from the time data were collected until publication. The cost of
development was approximately $200,000. This amount represents
consulting fees and printing costs but not administrative costs.\14
U.S. Healthcare chose to publish information on 11 HEDIS 2.0
measures using HEDIS 2.0 specifications for preventive services
(childhood immunizations, cholesterol screening, mammography
screening, and cervical cancer screening), prenatal care (low
birth-weight and prenatal care in the first trimester), acute and
chronic disease (asthma inpatient admission rate and diabetic retinal
exam), mental health (ambulatory follow-up after hospitalization),
and member access and satisfaction.\15 (See fig. 2.5.) Results for
1992 were compiled over a 2-month period using information from
existing administrative data sets and medical records. No reference
values from other sources were provided for comparative purposes.
U.S. Healthcare has not estimated its costs to produce this report.
However, officials did indicate that many of the data were available
through the corporation's ongoing quality assessment program.
Figure 2.5: U.S. Healthcare's
1992 Quality Report Card
(See figure in printed
edition.)
Source: U.S. Quality
Algorithms (a subsidiary of
U.S. Healthcare Corporation),
1993.
(See figure in printed
edition.)
In March 1994, the 41-member Massachusetts Healthcare Purchaser Group
requested that local insurers restrict annual premium growth and
report on clinical quality indicators. As a result, 14 HMOs, 1
indemnity plan, and the Massachusetts Division of Medical Assistance
agreed to issue a report card on 6 performance measures--adult and
pediatric asthma inpatient admissions, mental health inpatient
readmissions, prenatal care furnished during the first 3 months of
pregnancy, mammography screening, the cesarean section rate, and high
blood pressure screening. These indicators were selected because
they were related to the areas of highest cost for purchasers and the
Group believed there was substantial evidence to support the
indicators' validity. Although each plan's performance on each
measure was published, these measures were not intended to be
compared with national or regional reference values because the
developers believed that the indicators had not been widely used and
there was little knowledge about what constitutes good performance.
In fact, purchasers were advised against comparing the rate achieved
in one plan against the rate achieved in another. Instead,
developers calculated the average performance of the plans and urged
purchasers to compare each plan with that average. (See fig. 2.6.)
Figure 2.6: Example of
Information in the
Massachusetts Healthcare
Purchaser Group Report Card
(See figure in printed
edition.)
Note: Pie circles are statistically adjusted while bar graphs are
not.
Source: Massachusetts
Healthcare Purchaser Group, The
Cost/Quality Challenge, March
1994.
(See figure in printed
edition.)
--------------------
\12 Most managed care programs have a quality assurance program.
However, the extent to which information is maintained in a
computerized database or otherwise stored in a way that is easily
retrieved or summarized varies considerably.
\13 Healthy People 2000: National Health Promotion and Disease
Prevention Objectives was compiled by the Public Health Service after
3 years of study and submitted to the Secretary of Health and Human
Services in 1990. The document contains a national strategy for
improving the nation's health over a 10-year period. The report
presents over 300 objectives in 22 priority areas grouped into 3
broad categories: health promotion, health protection, and
preventive services.
\14 Kaiser prepared its report card in collaboration with Arthur
Andersen & Co., S.C.
\15 In August 1994, U.S. Healthcare released a report card
containing results for 1993.
FLORIDA IS DEVELOPING REPORT
CARDS FOR SELECTED HEALTH PLANS
---------------------------------------------------------- Chapter 2:4
In 1993, Florida legislatively mandated that health plans publish
report card information. To implement this requirement, an advisory
committee of employers, medical providers, hospitals, HMOs, indemnity
insurers, and consumers recommended that information on 41 indicators
be included initially in a report card. However, the state Agency
for Health Care Administration (AHCA) could not determine a precise
definition for each indicator, and some health plans stated that they
would be unable to provide information about others. State officials
have delayed writing regulations implementing this proposal.
However, they plan to issue a report card in May or June 1995 using
other data currently reported to the state.
As part of its health care reform legislation, the Florida
legislature required that an advisory data committee be established
to make recommendations to state officials about the types of data to
be collected and distributed to health care purchasers. Shortly
after its formation, the committee recognized that the various
stakeholders on the committee had different data needs and
priorities. Thus, the committee divided into five subgroups to
consider various indicators for inclusion in Florida's report card as
well as to identify and discuss other factors related to performance
measurement development and dissemination. The subgroups were (1)
Community Health Purchasing Alliance (CHPA)\16
members--small business representatives, state employees, and
Medicaid recipients purchasing insurance through the CHPAs; (2) CHPA
board members and staff--paid CHPA member representatives responsible
for selecting health plans that will be offered to alliance members;
(3) state government and health care researchers--persons responsible
for evaluating health care reform; (4) accountable health
plans--HMOs, insurance companies, and some of the larger Florida
hospitals that form the health plans to be offered; and (5) health
care providers--individual physicians, small hospitals, and other
types of health care professionals.
Each subgroup reviewed a total of 62 different indicators:\17 14
indicators were in the Institute of Medicine's data model,\18 26
indicators were in the Jackson Hole Group's data model,\19 and 41
indicators were in the HEDIS 2.0 data model. Then the subgroups
ranked each indicator based on what they believed to be the most
important. Florida officials tallied the subgroups' choices, and 41
of the 62 indicators were recommended for publication in the first
year.\20 An additional 11 were recommended to be phased in subject to
further review. (See app. I.) However, when the state proposed
regulations for data collection, it was discovered that clear
definitions were not available for each indicator and some health
plans could not produce the required data. AHCA then proposed that
all of the information on HEDIS 2.0 indicators plus one other
recommended by the Institute of Medicine be collected. But state
officials have delayed writing regulations implementing this proposal
until other matters related to state health care reform have been
resolved. Meanwhile, they plan to issue a report card in May or June
1995 using information obtained by (1) linking health plan enrollment
data to existing state databases and (2) conducting a patient
satisfaction survey.
--------------------
\16 CHPAs were established by the Florida legislature to help certain
health care purchasers to buy health insurance.
\17 Twenty-two of the 62 indicators were recommended by more than 1
organization.
\18 Institute of Medicine, Access to Health Care in America, National
Academy Press, 1993.
\19 The Jackson Hole Group is a health policy organization whose
purpose is to promote reform in the American health care system.
Their recommended indicators are described in a discussion paper
released in April 1993, "Health Outcomes Accountability: Methods for
Demonstrating and Improving Health Care Quality."
\20 Thirty-one of the 41 HEDIS 2.0 indicators were recommended, 10 of
the 14 Institute of Medicine indicators were recommended, and 17 of
the 26 Jackson Hole Group indicators were recommended for publication
the first year. Thirteen of the 41 indicators were recommended by
more than 1 organization.
HCFA IS DEVELOPING HMO REPORT
CARDS FOR PROS TO USE
---------------------------------------------------------- Chapter 2:5
HCFA is identifying performance measures it will incorporate in
report cards for PROs to use in assessing the health care HMOs
provide to Medicare beneficiaries. Although these measures are not
being designed to assist beneficiaries to make choices among HMOs,
developers expect the results to become public information.
In September 1993, HCFA contracted with the Delmarva PRO to develop
performance measures and determine the minimum data needed to
generate the desired results.\21 To accomplish this task, Delmarva
convened an expert panel to rate 268 separate measures. On the basis
of this panel's rating, Delmarva recommended in its February 1994
interim report collecting data on 10 permanent or core measures that
are primarily measures of access to preventive and chronic disease
care. Delmarva also recommended that data about other diagnostically
related measures be collected on a rotating annual bases. For
example, indicator results on diabetes may be requested one year,
indicator results on heart disease may be requested the second year,
and indicator results on cancer the third year. Delmarva submitted a
draft final report for comment to 17 organizations representing
stakeholders such as the American Medical Association, Group Health
Association of America, and Blue Cross Blue Shield Associations. The
organizations' comments were considered and a final report was
submitted to HCFA in mid-August 1994.
--------------------
\21 Delmarva's project was the result of extensive collaboration with
many experts and organizations.
MANY BARRIERS IMPEDE REPORT CARD
DEVELOPMENT
============================================================ Chapter 3
Many obstacles are hindering report card development. Experts
disagree about which quality of care indicators should be included in
report cards because data sources may be inaccurate and result in
misinformation. Even if these problems can be resolved, some experts
believe that good measures of quality do not exist. Furthermore, no
consensus has been reached about methods for calculating report card
results so that performance can be compared. Most experts also
believe that health care purchasers will be skeptical of report card
results unless results are verified by an independent third party.
DISAGREEMENT EXISTS ABOUT WHICH
INDICATORS SHOULD BE MADE
PUBLIC
---------------------------------------------------------- Chapter 3:1
Experts have varying beliefs about what information should be
included in a report card because of acknowledged difficulties with
the reliability and validity of data sources and measurement systems.
At one extreme are those who believe that little information should
be released because the data sources are inaccurate and the
indicators may not measure quality. These experts caution that
public confidence in report cards may be lost if the early report
cards are found to contain erroneous information. Although most
experts agree that the current public demand for information about
health care quality must be met, some think that little information
should be furnished until it is proven to be highly reliable and
valid. If this course of action is taken, report cards will not be
available for 10 to 15 years. Several providers have suggested
collecting only a very minimal amount of information by focusing on
limited data elements. One has cautioned against taking report card
results at face value, pointing out that if data are poor, the
results will be poor. This provider also expressed the belief that
good measures of quality do not exist. The provider also stated that
no one knows how to interpret variations in a plan's performance on
any one or all the indicators. Another expert stated that report
cards might be used as a screening tool only. However, as that
expert pointed out, no one knows whether report card results can
predict the level of quality furnished in a plan.
Other experts are anxious to publish performance data because they
believe that distribution of any data is better than no data being
published. Furthermore, these experts believe that preparers will
improve the quality of data once it becomes known that the
information will be published. These experts believe that report
cards could be available within 2 to 5 years if only selected
reliable and valid indicators are published. In this group's
opinion, other indicators can be added after problems with
information sources and other factors are resolved.
Researchers currently are using administrative databases, medical
records, and patient surveys in their studies even though they are
aware that these sources may have data limitations. They explain
these limitations when information is published. Report card
information could include similar language warning consumers of data
and indicator problems. Furthermore, one expert thinks that progress
with indicator development and data source improvement will not occur
unless an incentive is provided. For example, financial incentives
might be given to providers for submitting specified information to
the health plan. Additionally, automated and integrated medical
records might be developed so that providers find the information
they or others document useful during the normal course of their
work.
INFORMATION SOURCES MAY BE
INCOMPLETE, INACCURATE, OR
MISLEADING
---------------------------------------------------------- Chapter 3:2
Data sources for report cards range from large computerized
administrative databases maintained in health plans to individual
patient medical records kept in providers' offices. But these
sources might be inaccurate, incomplete, or misleading for report
card purposes because (1) most administrative databases were designed
for financial purposes rather than clinical purposes and (2)
providers may knowingly or unknowingly place incorrect information in
medical records or not document certain interventions.
CURRENT ADMINISTRATIVE
DATABASES ARE NOT WELL
SUITED FOR REPORT CARDS
-------------------------------------------------------- Chapter 3:2.1
Most administrative databases are designed to facilitate accurate and
timely payment to providers. Although administrative databases are
good sources of patient demographic information and utilization data,
they have limitations for use as source material for report cards.
Specifically, these databases may be (1) incomplete because
information about a patient's condition or the results of services
rendered needed for some report card indicators are not collected;
(2) misleading because they use numerical codes that represent
undefined diagnoses, procedures, and treatments; and (3) inaccurate
because they are based on carelessly entered data.
All insurers and health plans have their own administrative
databases. Information in these databases is obtained from (1)
patients at the time they enroll in the health plan and (2) providers
who submit claim forms citing the services rendered or providers who
submit encounter forms required by some HMOs notifying them of
specific services provided.\22 A database might include demographic
information about members enrolled in the plan and the services
provided to them such as hospital admissions, outpatient visits, and
utilization of services such as laboratory testing and pharmacy
services.
Administrative databases use abbreviated numerical codes in lieu of
narrative diagnoses, procedures, and treatments. The most common
coding systems are (1) the International Classification of Diseases,
Ninth Revision, Clinical Modification (ICD-9-CM), a list of over
10,000 medical and surgical diagnoses or conditions; (2) the
Diagnostic and Statistics Manual, third edition (DSM-III), a list of
over 290 mental health disorders; (3) the Current Procedural
Terminal, fourth edition (CPT-4), a list of about 7,200 procedures
and treatments; and (4) the HCFA Common Procedure Coding System.
Hospital medical records personnel or billing clerks in outpatient
centers and provider offices enter these codes on the appropriate
claim/encounter form or, after examining a patient, the provider
simply checks off on a preprinted form those items that apply. These
forms are either sent to the insurer or health plan where the
information is entered into the administrative database by claims or
clerical personnel or they are sent electronically to the insurer or
health plan where the information is automatically entered into the
administrative database.
HMO administrative databases usually are incomplete sources of
information for report cards because providers are not required to
submit claim forms.\23 In some instances, HMOs ask physicians to
supply encounter data documenting the care rendered to a patient.
But only a few HMOs furnish incentives to providers for performing
this task. As a result, HMO databases frequently contain only
demographic information supplied by the patient and do not include
information about the services the provider renders. This
information is insufficient for determining performance results used
in report cards.
Experts also believe that administrative databases derived from claim
forms and used for payment purposes are incomplete for report card
purposes because (1) they do not capture clinical characteristics
about patients such as blood pressure readings or test results and
(2) they do not include the outcome of the care provided. For
example, a health plan might be able to count patients being treated
for hypertension using the diagnosis code for hypertension recorded
in its administrative database. But the health plan could not report
on the outcome of the patients' treatment based on these data alone
because changes in patients' blood pressure as a result of care are
not noted in the database.
Administrative databases also may be misleading or inaccurate.
Although coding makes it easier to store and retrieve information, it
hampers report card development because a code may not represent a
clearly defined diagnosis, condition, or treatment. As a result,
when codes are used to formulate report cards, the results may be
unreliable. For example, the ICD-9-CM code for diabetes, 250.0, does
not differentiate among patients whose conditions are controlled
through insulin injection, oral medication, or diet alone. As a
result, indicators that measure the outcome from certain medical
interventions might be inaccurate if this code alone is used to
identify diabetic patients enrolled in a plan. To be useful for
report cards, codes should differentiate between such conditions.
Not only are the codes poorly defined for report card purposes, but
the wrong codes may be entered mistakenly into the database.
Researchers studying obstetrics care in Washington State found data
in state administrative databases related to maternal and infant
outcomes to be poorly coded. Specifically, eclampsia, blood
transfusion, labor induction, neonatal seizures, and admissions to a
neonatal intensive care unit (NICU) had high error rates. Such
errors could have significant impact on report card results. For
example, only 65 percent of 211 reported admissions to NICUs were
actual admissions. If NICU admission rates were a report card
indicator, report card users would be led to believe that NICU
admissions were much greater than they actually were.
--------------------
\22 Many HMOs pay providers either a fixed salary or a capitated
rate--a fixed, prepaid, monthly payment based on the number of
patients enrolled in a program or assigned to a provider. As a
result, providers do not have to submit claim forms to be reimbursed
for services rendered.
\23 HMOs constitute about 32 percent of managed care organizations.
MEDICAL RECORDS HOLD
IMPORTANT CLINICAL DETAIL
BUT OBTAINING IT IS
DIFFICULT AND COSTLY
-------------------------------------------------------- Chapter 3:2.2
Data found in medical records are rich in clinical information.
However, obtaining that information is expensive because trained
personnel are needed to retrieve it from numerous medical record
pages. Furthermore, medical records can include misleading or
inaccurate information. Thus, experts believe that medical records
should not be used as the sole source for report card results.
Medical records contain information about a patient's medical
history; current symptoms; test, procedure, and treatment results;
medication use; and other relevant clinical data. Providers make
their notations when they have seen or communicated with the patient.
Clerks usually file laboratory and other test results in the medical
record. Although hospitals have prescribed policies and procedures
regarding the types of information that must be included in a medical
record, most providers' offices have less formal systems and their
medical records are less inclusive. (These providers do not have to
comply with accreditation standards.)
The West Virginia Medical Institute estimates that it costs
approximately $16 to find and retrieve information from one medical
record.\24 Most medical records are kept in voluminous paper files
located in doctors' offices, hospitals, or other provider sites.
These records may be located in several areas within the site or
misfiled in the file room. Furthermore, retrieving the information
frequently requires a highly trained person to search through the
medical record's many pages to locate and interpret the data needed.
Although medical records may be a better clinical source for report
cards, some experts believe that the high cost of collecting
information from medical records is forcing report card developers to
rely on administrative databases--a less reliable but more
cost-effective source of information.
For report card purposes, relying totally on medical records data is
discouraged because the records might include incorrect, incomplete,
or imprecise information. For example, one expert reported that
physicians may err in conducting the physical examination or
interpreting tests or other personnel may make faulty clinical
observations. Additionally, providers frequently do not document
items they believe to be unimportant. Although test results and
physician impressions of the patient's clinical condition usually are
noted in the medical record, the records often do not contain
information of lesser importance to the provider's continuing care
such as nutritional or other counseling services. Also, providers
sometimes document a diagnosis, treatment, or condition to ensure
insurance coverage or protect patient interests. As a result, notes
within the medical record may imply that the patient's condition is
somewhat different than it actually is. For example, when a
physician orders a mammogram, a woman's previous medical history
might be described in a way that would meet a specific insurance
company's standards for performing the mammogram. In other
instances, a physician might record a less specific diagnosis because
the precise diagnosis could harm the patient if it were brought to
public view. Acquired immunodeficiency syndrome, asthma, alcoholism,
and mental illness are examples of some diagnoses that carry social
stigmas.
--------------------
\24 The Institute has a contract with the Department of Veterans
Affairs (VA) to conduct medical reviews in VA hospitals nationwide.
The estimate cited includes retrieval of the medical record, personal
computer set-up and breakdown time, abstracting the information, as
well as administrative time such as the abstractor discussing
problems with a supervisor. The estimate does not include other
overhead costs.
REPORT CARD INDICATORS MAY NOT
BE VALID MEASURES OF QUALITY
---------------------------------------------------------- Chapter 3:3
Indicators currently used in report cards may not be good measures
for making decisions about a health plan's ability to provide quality
care. Indicators measure either organizational arrangements
(structure) or activities carried out by providers (process).
However, research has demonstrated that these indicators may not
measure quality. Although these indicators measure arrangements or
activities that experts think are related to quality, some experts
believe that measuring outcomes or the results of care is the best
and perhaps only way to evaluate quality. However, outcomes need to
be adjusted to ensure that the effects of structure and process and
not the effects of patient characteristics are being measured.
Experts have not agreed upon an acceptable risk adjustment
methodology.
SOME INDICATORS HAVE NO
BASIS IN RESEARCH
-------------------------------------------------------- Chapter 3:3.1
Structure and process indicators are used frequently when assessing
quality because it is relatively easy to measure them. But research
has not clearly demonstrated correlations between some commonly used
structure and process measures and desirable outcomes. Hence,
measures that have not been proven to be comprehensive indicators of
quality are being used in report card comparisons.
The following example illustrates how the health care community uses
quality indicators that have not been proven to be valid measures of
quality. It is commonly accepted by health care experts that a
board-certified physician provides better care than a physician who
is not board certified.\25 In fact, HEDIS 2.0 developers recommend
including as performance indicators the "percent of primary care
physicians board-certified" and the "percent of physician specialists
board-certified in the specialty in which they practice
predominantly." Other released report cards also have used this
measure. However, no one has proven that the patients of
board-certified physicians have better results than other patients.
In fact, in five of seven studies reviewed by the Office of
Technology Assessment (OTA), physician board certification showed no
effect on performance.\26 OTA provided several explanations for these
results, including the possibility that the studies may have been
flawed--the sample size too small, adjustments for patient
characteristics absent, or the tool used to measure quality not
reliable or valid. Additionally, the certification process may be
invalid--written and oral tests used to evaluate physician
performance may not measure success in clinical practice.
Some performance measures are selected for reporting because experts
believe the indicator represents a concept that is difficult to
measure directly. For example, barriers to care is a concept that
may be difficult to measure. As a result, several experts recommend
monitoring infant mortality rates. These rates explicitly measure
the quality of medical care given to pregnant women and to children 1
year old or younger. However, they also might implicitly measure a
health plan's success with reducing barriers to care experienced by
certain groups.
--------------------
\25 Board certification is a method of formally identifying a
physician who has completed a specified amount of training and a
certain set of requirements, and passed an examination required by a
medical specialty board.
\26 U.S. Congress, Office of Technology Assessment, The Quality of
Medical Care: Information for Consumers, OTA-H-386 (Washington,
D.C.: U.S. Government Printing Office, June 1988).
RISK ADJUSTMENT SYSTEMS MAY
NOT BE RELIABLE
-------------------------------------------------------- Chapter 3:3.2
Risk adjustment systems have not been perfected, and tests of systems
that are in place indicate that they may not be reliable. As a
result, experts advise against mandating a national system. Outcome
measures should be adjusted so that differences in outcomes can be
attributed to either the quality of care furnished or to patient
characteristics. If adjustments are not made, providers or health
plans may contend that poor outcomes are due to their caring for
sicker patients. At least 15 risk adjustment systems\27 have been
identified in the literature.\28 The results of the few reviews that
have been done have found that risk adjustment systems may not be
able to predict whether patient characteristics or poor quality of
care is the reason for a poor outcome. Additionally, experts believe
that risk adjustments might obscure valuable information such as the
number of persons within a certain age range who experienced specific
outcomes.
All risk adjustment systems assume that certain patient
characteristics contribute to adverse outcomes such as increased
resource consumption (a substitute for severe illness), death, or
complication. At least 10 patient characteristics have been
identified that might influence the outcome of health care: age;
acute clinical stability; principal diagnosis; severity of principal
diagnosis; extent and severity of comorbidities;\29
physical functional status; psychological, cognitive, and
psychosocial functioning; nonclinical attributes such as
socioeconomic status, race, substance abuse, and culture; health
status and quality of life; and patient attributes and preferences.
Using complex statistical modeling techniques, researchers look for a
positive or negative relationship between these patient
characteristics and the observed outcome or results of care. To the
extent that patient characteristics are associated statistically with
the adverse outcome, then differences in outcomes may be attributed
to factors other than differences in quality of care. Conversely, if
patient characteristics are not associated statistically with the
adverse outcome, then differences in outcomes are more likely to be
the result of differences in quality of care. But research has shown
that some techniques used to adjust for differences in patient
characteristics may not be predictive of the outcome observed. The
following two examples illustrate that conclusion.
One study evaluated age as a predictor of outcome. Age is a patient
characteristic frequently used for risk adjustment. However, a
system that relies heavily on age may not be appropriate in all
instances. In a 1990 study of persons 65 years of age and older 30
days after they were admitted to a hospital, age accounted for only a
1-percent variation in outcome for congestive heart failure and hip
fracture patients, only a 2-percent variation in outcome for acute
myocardial infarction patients, and only a 3-percent variation in
outcome for pneumonia patients.\30 Age may be an important factor
when analyzing the results of care furnished to patients in broad age
ranges but not as important a factor when analyzing outcomes in
narrower age ranges.
The second study evaluated the ability of six different risk
adjustment systems to predict mortality across five high-mortality
conditions: stroke, lung cancer, pneumonia, acute myocardial
infarction, and congestive heart failure.\31 One of the systems used
standard information such as age, sex, principal diagnosis, and
comorbidities found in hospital discharge summaries to adjust for
risk. Two of the systems used standard clinical information found in
the medical record that might adversely affect the outcome of any
illness. Three of the systems used specific clinical factors that
physicians believe affect the actual conditions being studied. The
first system based on discharge summary information was the least
predictive of death and the latter three systems based on
condition-specific information were most predictive.\32
The reliability of any risk adjustment system depends on the
accuracy, completeness, and credibility of the information it uses.
Risk adjustment systems use coded information from coded discharge
summaries or administrative databases, clinical information from
medical records, or functional status and satisfaction data obtained
through patient surveys. The advantages and disadvantages of
collecting and using data from these sources are discussed on pages
35-38 and pages 44-46. These factors equally affect the reliability
of risk adjustments.
An additional concern expressed by several experts was that risk
adjustments inadvertently hide valuable information. In its draft
proposal submitted to HCFA, the Delmarva PRO did not recommend
adjusting data for certain patient characteristics because it might
obscure otherwise important information, preventing HMOs from
identifying population groups needing improved care. Therefore, they
recommended categorizing results into various subgroups based on
nonclinical and clinical patient factors. For example, as previously
discussed, age differences is a patient characteristic that might be
important to analyze. But if age is an adjustment factor, the
outcomes of care furnished to people within different age ranges will
not be determined. Consumers, government officials, and others might
want to know how successful different health plans are at treating
people in different age groups.
The experts we interviewed thought that creating one national risk
adjustment system would be undesirable.\33 Three reasons were given.
First, adjustment systems are in their infancy and establishing one
preferred system might reduce the competition to create more reliable
and valid systems. Second, many states, business coalitions, and
other organizations developing report cards have invested in specific
systems they believe meet their needs. Duplication and added expense
would result if another system were mandated. Third, each
performance indicator may need its own separate adjustment because
patient characteristics have a unique effect on every condition and
disease. Separate indicators have been used in some instances. For
example, California officials are developing risk adjustment methods
specific to particular outcomes, conditions, and procedures. This
approach was adopted after a 1991 state legislative committee
concluded that (1) constructing and validating outcome measures may
be as important a factor in improving the quality of care in
California hospitals as any report ranking hospitals by risk-adjusted
mortality and morbidity; (2) no "best" risk adjustment system was
available commercially; and (3) none of the commercial systems was
designed to make interhospital comparisons of outcomes at a statewide
level.\34
--------------------
\27 Initially, these systems were developed to predict resource
consumption so hospital payment rates could be established. Some
experts believe that these same methods can be used to predict
quality of care. More recently, however, systems have been developed
for the sole purpose of predicting quality.
\28 Our forthcoming report, GAO/HEHS-95-1, discusses specific
performance data systems.
\29 Comorbidities are diseases or conditions present at the same time
as the principal condition of a patient.
\30 E. B. Keeler, K. L. Kahn, D. Draper, et al., "Changes in
Sickness at Admission Following the Introduction of the Prospective
Payment System," Journal of the American Medical Association,
1990:1962-1968.
\31 L. I. Iezzoni, A. S. Ash, G. A. Coffman, and M. A.
Moskowitz, "Predicting In-Hospital Mortality: A Comparison of
Severity Measurement Approaches," Medical Care, April 1992:347-359.
\32 The system based on discharge summary information that was least
predictive was designed originally to predict resource consumption.
Therefore, experts were not surprised that it was the poorest
predictor of death.
\33 We asked experts if they believed that one national risk
adjustment system should be mandated. Experts did not raise that
subject independent of our question.
\34 L. Johns, "State Report: Measuring Quality in California,"
Health Affairs, Spring 1992:266-270.
REPORT CARD INDICATORS MAY NOT
BE COMPARABLE
---------------------------------------------------------- Chapter 3:4
Current report cards may not be comparable and their usefulness may
be diminished because health plans have not agreed on the
measurements to be used. While relying to some extent on HEDIS 2.0
measures, most individual health plans that have published report
cards are using their own criteria for collecting data and computing
results. Additionally, these health plans have developed their own
patient surveys to determine how consumers view the services they
receive. Furthermore, little mention is made in any of the report
cards about the plan's benefit structure and the effect these
benefits may have on patients' utilization of services.
NATIONAL STANDARDS FOR
CALCULATING RESULTS DO NOT
EXIST
-------------------------------------------------------- Chapter 3:4.1
Nationwide standards for calculating indicator results have not been
established. However, individual states such as Florida and Maryland
are working toward mandating statewide standards. Although HEDIS 2.0
developers are working toward standardizing calculations, their
efforts are voluntary only and health plans are not required to
follow their guidance. As a result, health plans can continue to
collect and compute report card results using their own criteria and
purchasers will be unable to compare plans to determine which one
meets their needs.
The following example illustrates how two organizations have used
different standards to collect and compute data for their report
cards. Both methods may be accurate. However, the methods may not
meet the needs of the purchaser because the results cannot be
compared.
Administering childhood immunizations is an indicator found in many
report cards. The results Kaiser Permanente included in its report
card were based on the number of children between birth and 2 years
of age who received at least four diphtheria/pertussis/tetanus (DPT)
immunizations. But Medica, a health plan managed by UHC, showed in
its report card the number of children between the ages of 2 and 6
months who received at least three DPT immunizations.
In addition to counting the number of events differently, these two
organizations counted the population on which the results are based
differently. Kaiser noted that its results were based on (1) a
sample of 601 members born in the same year, (2) who utilized health
plan services at least once, and (3) who were continuously enrolled
in their plan for the designated 2-year period. UHC noted that its
results were based only on the total population of children who were
members of the health plan during the reporting period. Furthermore,
both plans were unclear about whether they also counted only those
children who were immunized by providers in their plan or whether
they also counted children enrolled in their plan who were immunized
by other providers.
Several experts believe that the federal government could assume the
role of establishing uniform collection and calculation standards.
They contend that this strategy will ensure that health plans do not
collect and compute indicator results to their own advantage rather
than the public's.
METHODS FOR MEASURING
PATIENTS' VIEWS ABOUT CARE
HAVE NOT BEEN AGREED ON
-------------------------------------------------------- Chapter 3:4.2
Most providers and health plan administrators are recognizing that
measuring patients' perceptions of the quality of care they receive
is as valid and important as measuring the technical quality of
services rendered. Asking patients about their satisfaction with the
care furnished, their own health status, their own health behavior,
and the care they were provided is the best, and sometimes only way,
to obtain some of the information sought by report card developers.
Researchers have developed several methods for obtaining patients'
views about the care they have received. But no standardized method
has been agreed to. Thus, health plans are administering their own
surveys, and the results obtained by individual plans cannot be
compared.
Health plans that are obtaining patient views usually collect
information through written questionnaires or telephone surveys
administered by the health plan or an independent third party.
Generally, patients are asked if they are satisfied with the services
they received. In some instances, members have been asked specific
questions about their health status and their ability to perform
daily activities such as walking or bathing. If a written
questionnaire is used, frequently it is distributed to randomly
selected members. Telephone calls may be made to members who do not
return the questionnaires. In other instances, the survey is
conducted only by telephone.
Neither administrative databases nor medical records contain
information about how satisfied patients are with the services they
received. And as pointed out previously, these sources may not
include information providers neglected to document or believe is
unimportant. However, undocumented information, such as whether
counseling was provided or whether the patient can climb stairs more
easily after hip replacement surgery, might provide valuable
information for a report card about the outcome of services provided.
Researchers have shown that patients are reliable sources for this
type of information and, in fact, sometimes better sources than
administrative databases or medical records.
Various patient survey tools are being developed. Funded by the
Xerox Corporation, GTE Corporation, and Digital Equipment
Corporation, researchers at the Health Institute of the New England
Medical Center have developed a survey to gather employees'
perceptions of the quality of care given by health plans their
employers offer. In this instance, participating employers are
interested in comparing and ranking plans. Health Institute
researchers developed a 154-item survey that was sent to over 35,000
employees enrolled in 31 different health plans offered nationwide by
the participating employers. More than 25,000 employees responded.
The survey questions were based on other accepted surveys such as the
Group Health Association of America's (GHAA) Consumer Satisfaction
survey and the Medical Outcomes Study team's SF-36 Short Form Health
Status Survey.\35 Many of the questions in the Health Institute's
survey required the respondent to answer on a five-point rating scale
ranging from poor to excellent, definitely true to definitely false,
or all of the time to none of the time. Other questions called for a
yes/no response, while others required a specific answer to a
specific question.
Researchers at Beth Israel Hospital are taking a different approach
to patient surveys. In 1988, the Picker/Commonwealth Fund gave Beth
Israel researchers a grant to develop a hospital patient satisfaction
survey that hospitals can use to improve services. Although this
survey applies only to hospital patients, researchers at Beth Israel
said the concept can be applied in health plans. Using patient
input, the Beth Israel researchers identified aspects of patient care
that most affect patients' hospital experiences.\36 The resulting
patient survey questions focused on specific actions that might be
taken by hospital staff that affect these dimensions. For example,
one question asks whether the patient usually got help in time with
eating, bathing, or getting to the bathroom. The percentage of
patients who responded that they did not receive this assistance is
reported to the hospital. Survey responses are obtained by
telephoning patients and their families. Results are sent only to
hospitals who participate in the study, although researchers say they
could be published.
Although health reform proposals call for patient surveys, not all
health plans are enthusiastic about allowing independent surveyors to
canvas their members and publish the results of their efforts. The
following example illustrates this point. Federal Employee Health
Benefit Program officials entered an agreement to cooperate with the
Center for the Study of Services\37 to provide comparative
information about health plans providing care to federal employees.
But only 272 (approximately 75 percent of plans with appropriate
sized enrollment counts) of participating health plans have agreed to
participate in the study. Some health plans and industry
representatives objected to the questions and the size of the sample
selected to answer the questions. Others would rather conduct the
survey themselves rather than contract with an independent
organization. One health plan executive expressed concern that his
organization was being required to participate in a survey it had no
say in designing. Another wanted the right to veto disclosing the
results after reviewing the responses.
--------------------
\35 GHAA is a membership organization that represents HMOs
nationwide. Its survey was designed to assess health care delivery
and health plan administration. The SF-36 was designed by
researchers at the RAND Corporation and the Health Institute to
assess patients' functional status, well-being, and general health
perceptions.
\36 The aspects of care identified include (1) respect for patients'
values, preferences, and expressed needs; (2) coordination of care
and integration of services within an institutional setting; (3)
communication between patient and providers; (4) physician care,
comfort, and the alleviation of pain; (5) emotional support and
alleviation of fears and anxiety; (6) involvement of family and
friends; and (7) transition and continuity from one site of care to
another.
\37 The Center for the Study of Services publishes Consumers'
Checkbook, a consumer guide that rates products and services,
including health care.
BENEFIT STRUCTURES MIGHT
AFFECT REPORT CARD RESULTS
-------------------------------------------------------- Chapter 3:4.3
Differences in health plan benefit structures may complicate report
card comparisons because both the cost of services covered by a plan
and the range of those services may affect patients' utilization.
Presumably, the plan with the more generous benefit structure might
appear to supply the better care because patient utilization of
services is higher than for another plan with lesser benefits.
Consider the example of two asthmatic patients who experience mild
flare-ups. The patients and their conditions are identical. But
patient A is enrolled in a plan that requires a $20 copayment for
each visit to a doctor while patient B pays nothing to see a doctor.
Because patient A does not want to pay $20, he unsuccessfully
attempts self-medication, his condition deteriorates, and he is
admitted to a hospital. Because patient B pays nothing, he
immediately sees his physician and his condition improves. Patient
A's health plan will show a higher hospital admission rate than
patient B's health plan. Without benefits information, these results
could be interpreted as meaning patient A's health plan provides
poorer quality care than patient B's plan.
REPORT CARD RESULTS ARE NOT
BEING VERIFIED
---------------------------------------------------------- Chapter 3:5
Most experts agree that there is a need to have an independent third
party verify the information contained in report cards. However, no
consensus exists regarding who should do the verification or what
information should be included. Although the possibility exists that
health plans may purposely alter their performance results, experts
believe that there is a greater probability that innocent errors will
be made because data collection is a complex task.
Currently, few organizations are requesting that objective third
parties to verify their data. NCQA has contracted with an
independent agent to verify data gathered by the 21 health plans
participating in its report card project. The state of Maryland is
also planning to verify information gathered as a part of its newly
mandated report card project. But few other organizations are taking
similar steps.
Several experts suggested that organizations similar to those
conducting financial audits could perform data audits.
Administrative database information could be cross-referenced to
medical record information to verify accuracy. Others suggested that
audits be performed on a random basis or that audits be required only
in situations where a designated authority has reason to believe that
misreporting may have occurred. No agreement exists about whether
the federal government should play a role in auditing.
REPORT CARDS MIGHT BENEFIT
DIFFERENT HEALTH CARE STAKEHOLDERS
============================================================ Chapter 4
Many different groups can potentially benefit from the information
contained in report cards. Employers could select better plans for
their employees; public health officials could determine whether
health care goals for the nation are being met; health plan
administrators and researchers could improve services; and consumers
could select health plans that meet their specific needs. While the
needs of employers, public health officials, providers, health plan
administrators, and researchers are generally known, the needs of
individual consumers have not been well communicated to report card
developers. As a result, report cards may not contain all the
information consumers need to make purchasing decisions.
Additionally, publishing performance results may have unintended
adverse consequences such as health plans using the information to
avoid enrolling very sick and costly patients or putting their
resources into areas being measured to the detriment of areas not
measured.
EMPLOYERS EXPECT REPORT CARDS
TO PROVIDE BENEFITS
---------------------------------------------------------- Chapter 4:1
Employers expect report cards to help them (1) select a health plan
that provides value (that is, necessary health care at affordable
prices); (2) improve employee relations because employers can give
employees information about the quality of health care furnished by
company-selected health plans; (3) select health plans that keep
their employees healthy and working; and (4) perform oversight
functions of health care quality in company-selected plans. Early
indications reveal that report cards are achieving some of these
goals.
Officials of Hershey Foods, Inc., believe their company has benefited
from using cost and quality information published by the Pennsylvania
Health Care Cost Containment Council. Hershey officials have used
this information to negotiate contracts with primary care physicians,
specialists, and hospitals, according to a progress report published
by the Council.\38 This report also noted that Hershey estimates that
these negotiations will have at least a 10-percent positive impact on
its profits. Additionally, Hershey has redesigned its employee
benefit plans using the Council's data to shop for health care.
Furthermore, the report states, other companies are using the
information for employee education and to select health care
providers.
Another Pennsylvania company, Accutrex, is part of an alliance of
businesses in the state that saved a total of $1 million in 6 months
using the Council's reports. According to the company's president,
Accutrex experienced a 13-percent reduction in its health care
expenditures. In addition, the manufacturer managed to expand
coverage for some health care services for its employees.
--------------------
\38 Pennsylvania Health Care Cost Containment Council, A Progress
Report, 1991-93.
PUBLIC HEALTH OFFICIALS FORESEE
MEETING NATIONAL HEALTH CARE
GOALS
---------------------------------------------------------- Chapter 4:2
Public health officials believe that national health policy goals are
more likely to be met if report card indicators include public health
goals such as those identified in a 1990 Department of Health and
Human Services publication, Healthy People 2000. Public health and
health plan officials we interviewed said that health plan
administrators would make every effort to meet those goals if they
were included in a public performance measurement system.
The report card U.S. Healthcare published in 1993 illustrates how
including public health goals in the reporting system might encourage
health plan administrators to focus on meeting those goals. The
company's November 1993 press release announcing the report card's
publication clearly called attention to U.S. Healthcare's
performance in comparison with goals in Healthy People 2000. In an
instance where U.S. Healthcare exceeded the national goal, company
officials stated, "The U.S. Department of Health and Human Services,
in the publication Healthy People 2000 . . . sets a goal for the
year 2000 of at least 60% of women over age 50 to receive mammography
within the preceding one to two years. U.S. Healthcare has already
exceeded this ambitious goal" (emphasis added). U.S. Healthcare
also reported that its immunization rate and cholesterol screening
rate exceeded Healthy People 2000 goals.\39
--------------------
\39 U.S. Healthcare cited these reference values in a press release.
As discussed on pp. 28-29, the company did not include reference
values in the actual report card.
INDIVIDUAL PROVIDERS COULD JOIN
PLANS THAT MEET THEIR PRACTICE
STYLES
---------------------------------------------------------- Chapter 4:3
Providers might benefit personally from reviewing published
performance data before deciding to contract with a specific health
plan to provide services to its enrollees. Providers, particularly
primary care physicians, are being actively recruited by health plans
in various communities. These health plans have different policies
about utilization review, reimbursement, and specialty referrals that
could affect the provider's practice style. If the outcome of these
policies is reflected in report card results, the provider could make
an informed decision about affiliating with a particular health plan.
The state of Maryland has recognized that providers have unique
interests and plans to conduct a survey that addresses those
interests. After conducting patient satisfaction surveys and
collecting HEDIS-like indicator information, the state is planning to
conduct a provider survey to solicit their opinions about health plan
performance. Providers have recommended that the latter survey
address attitudes and opinions of physicians working in a managed
care environment, the accessibility of diagnostic testing, the
after-hours availability of diagnostic testing, timeliness of test
reports, and the availability of qualified subspecialty consultants.
At this time, Maryland does not intend to identify providers'
opinions of individual health plans because some physicians fear
retribution from health plans if they respond negatively about any
one health plan.
HEALTH PLAN ADMINISTRATORS,
PROVIDERS, AND RESEARCHERS
COULD TARGET AREAS FOR FURTHER
STUDY
---------------------------------------------------------- Chapter 4:4
Health plan administrators and providers should be well aware of
their own organization's quality assurance efforts and their plan's
performance on specific indicators. According to experts, report
cards could allow administrators and providers to measure their
plan's relative performance against other health plans'. This
information will help those responsible for providing care to
determine if their plan is providing optimum care and, if not, make
necessary changes in areas needing improvement. Researchers, like
administrators and providers, could use report cards to study why one
health plan performs better than another. Additionally, both groups
might find report cards useful in determining compliance with
established clinical practice guidelines.
The Massachusetts Healthcare Purchaser Group listed several ways
insurers or health plan administrators can use information contained
in its report to improve their operations. One use is monitoring
comparative performance to foster communication between health plans
and promote diffusion of best practices. For example, Neighborhood
Health Plan's pediatric asthma admission rate was 7.2 children per
1,000 members while Health New England, Inc.'s rate was 0.32 children
per 1,000 members. The average rate for the participants in the
Purchaser Group was 2.0 children per 1,000 members.\40 Neighborhood
Health Plan can use this information as the basis for evaluating its
performance in this area (for example, to determine why its admission
rate was so much higher) and, if necessary, take corrective action to
improve its performance.\41
Report cards also might be used by researchers to determine if
providers are complying with clinical practice guidelines.
Performance measures selected for reporting might reflect the
treatment processes included in the practice guidelines. For
example, if one of the practice guidelines for breast cancer
prevention were that women over 50 should receive mammograms, the
number of women over 50 who have received mammograms would be a
measure of success in meeting that guideline.\42
--------------------
\40 These measures were not risk adjusted. Additionally, they do not
incorporate the factors of prevalence of disease in a plan's
population or patient compliance with outpatient treatments. As
previously discussed, the Massachusetts Healthcare Purchaser Group
advises against comparing one plan with another and suggests that
results be compared with the average of all plans in the group.
\41 One expert expressed concern that publishing report card results
could hinder information sharing among health plans. If one health
plan's performance on an indicator is better than another's,
financial incentives might stop the better performing plan from
sharing information that might help to improve its competitor's
performance.
\42 However, a low rate only indicates a possible problem. Report
cards may not give researchers clinical information necessary to
determine how, why, and when care can be improved. More information
would be needed to determine if a problem actually exists, what the
problem is, and how to improve performance.
CONSUMERS MIGHT BENEFIT IF THE
INDICATORS REFLECT THEIR
CONCERNS
---------------------------------------------------------- Chapter 4:5
Health care experts and policymakers expect consumers to use report
cards to select a health plan that meets their needs. However,
experts doubt that report cards will be useful to consumers unless
they include information consumers want. Although consumers are
being surveyed about their satisfaction with care they received from
a health plan, they have rarely been asked what performance measures
they would be interested in evaluating. Furthermore, some providers
and experts believe that consumers will not understand report card
information. No evaluations have been done on how consumers have
used published report cards.
Some newspapers and magazines are publishing comparative health plan
data for consumers. One is Consumers' Checkbook, which is published
by a nonprofit organization that provides comparative information
about a variety of subjects on the basis of consumer surveys.
Checkbook consumer surveys are sent to its lists of subscribers who
number approximately 50,000 in the Washington, D.C., area and to
Consumer Reports subscriber lists. Although Consumers' Checkbook has
been publishing health plan information for consumers for 15 years,
the publisher is not sure what specific information is of interest to
consumers. Checkbook has received little specific consumer feedback,
such as letters to the editor or other such communication, on which
to judge how well the information it is publishing is being received.
Thus, Checkbook reports may not be addressing the right issues.
According to its publisher, Checkbook prints publicly available
information or information its staff develop themselves. Checkbook
receives very little cooperation from health plans. As a result, it
reports on what consumers say about their physicians in response to
Checkbook survey questions even though the consumers might be more
interested in other factors pertaining to health care quality. Even
though Checkbook may not be publishing exactly what consumers want,
the publisher believes that consumers will be interested in
meaningful information. If consumers see things they do not find
interesting, they ignore them.
After collecting information about health care services, Checkbook
ranks those they believe to be the best. They also report on all
information gathered even when it is voluminous or contradictory.
The publisher believes that the sometimes contradictory statements
from different sources highlight the level of uncertainty inherent in
current evaluations.
In 1993, Checkbook published information provided by three
Washington, D.C., metropolitan area health plans using HEDIS
measures. This approach was taken only to give consumers an example
of the types of information they might see in the future and was not
intended to be a report card about the plans. All of the data
reported were footnoted to explain collection methodology or other
data gathering facts.
Checkbook's publisher is concerned that some employer-motivated
performance systems may divert attention from items of interest to
consumers. For example, employers probably do not have an interest
in individual physician ratings because they realize that one plan
cannot furnish the very best provider to all patients. Therefore,
the measures developed will gauge plan performance and not provider
performance. Yet patients are interested in measures that allow them
to choose the very best provider.
Another magazine that is publishing comparative information for
consumers is Health Pages. This magazine's mission, in part, is to
help educate and empower consumers so that they will create
additional demands for health care information. In its winter 1993
St. Louis issue, Health Pages published background information about
various St. Louis physicians, including oncologists, surgeons, and
urologists. In addition to whether these physicians were board
certified, the magazine provided information about charges for
various treatments and procedures. Health Pages also published
general information on breast cancer, mammography facilities, and
prostate cancer.
That edition of Health Pages also included a report card on 11 St.
Louis HMOs. (See fig. 4.1.) These HMOs were asked to submit
information on eight indicators using HEDIS 2.0 definitions and
formulas. Consumers were asked to evaluate the information Health
Pages presented. The St. Louis Area Business Health Coalition,
which includes large companies such as McDonnell Douglas, distributed
over 60,000 copies of Health Pages to employees and retirees. In
addition, McDonnell Douglas sent its personnel a questionnaire asking
them to evaluate the magazine. Fifty-four percent of the
approximately 1,200 respondents indicated that the feature on managed
care was of interest to them. Comments received included the
following:
"I was disappointed that St. Louis University hospitals did not
participate. Our local hospital is affiliated with them. I
intend to contact the administrator."
"I liked the comparison of plan statistics."
"Congrats [sic] on fees and physicians--keep them in future
editions. We've needed something like Health Pages for years."
Figure 4.1: Example of Health
Pages' HMO Report Card
(See figure in printed
edition.)
Source: Health Pages , Winter
1993, St. Louis Edition.
(See figure in printed
edition.)
Although organizations releasing report cards provide some technical
information to consumers on how to evaluate the quality of health
care, some providers and experts continue to believe that consumers
should not be given information because they will not understand it.
But consumers have a broader outlook than providers might expect. Of
the 62 indicators presented for evaluation to Florida's CHPA Advisory
Committee, only 1 was not recommended at all by consumers who
participated in the development process. This compares with 9 that
were not recommended by providers, 11 that were not recommended by
state government officials and researchers, 20 that were not
recommended by CHPA board members and staff, and 21 that were not
recommended by accountable health plans. (See app. I.) In addition
to illustrating that consumers have a broader interest and
understanding than some providers believe, this illustrates how
consumers' interests might be overlooked if their opinions are not
sought.
REPORT CARDS MIGHT HAVE SOME
UNINTENDED ADVERSE CONSEQUENCES
---------------------------------------------------------- Chapter 4:6
Although most experts said that there are significant benefits to
publishing performance measurement results, some cautioned that
distributing this information may result in unintended adverse
consequences. Some plans might use the information to avoid
enrolling very sick and costly patients, intentionally deciding to
represent themselves unfavorably in selected areas to discourage
enrollment of patients suffering specific illnesses. For example,
breast cancer patients may need expensive and ongoing treatment. A
plan that has a low 5-year breast cancer survival rate (that is, few
of its breast cancer patients lived for 5 years or more) might not
improve performance in this area to avoid attracting more
"undesirable" patients.
Some experts also are concerned that administrators will place all
their organizations' resources in areas that are being measured.
Areas that are not highlighted in report cards will be ignored. One
way to prevent that is to alternate report card indicators so the
same areas are not always highlighted. However, this suggested
solution might be costly.
CONCLUSIONS
============================================================ Chapter 5
Report cards can be a useful tool to educate stakeholders about the
care provided by a health plan. We support the report card concept
and encourage continued development in the field. However, current
indicators chosen for publication reflect only selected aspects of
the care furnished by a plan--mammograms, immunization, mortality
from a certain procedure, and so on. These indicators may not be a
good measure of the quality of health care being provided.
Furthermore, the data sources on which these indicators are based
frequently are inaccurate. Thus, no conclusion about the overall
quality of care provided by a health plan can be drawn from these
measures.
Purchasers of health care rightfully expect to receive reliable and
valid information. Individual consumers, particularly, may be making
significant decisions based on the information found in report cards.
Given that report card indicators may not be based on good data,
today's health care purchasers and other users of the information
should be warned about the data's limitations and the conclusions
that can be drawn. This warning could take the form of a prominently
displayed cautionary notation on report cards.
We also strongly support the current dialogue among public officials,
health care researchers, providers, health plan administrators, and
purchasers--including individual consumers--on what information
should be collected and how it should be collected. In our opinion,
this dialogue should be expanded to incorporate discussions about
third party verification of reported data.
Report cards can provide stakeholders with important information if
the indicators are about pertinent subjects. Although different
stakeholders have different interests and uses for the information,
the majority of report cards being developed are directed toward one
type of audience--corporate purchasers. We do not believe that this
type of report card will be adequate for all users and, particularly,
for individual consumers. Therefore, we believe that health plans
should be encouraged to explore the feasibility of developing report
cards for individual consumers.
SOURCES OF INDICATORS FLORIDA'S
CHPA ADVISORY COMMITTEE CONSIDERED
AND ITS RECOMMENDATIONS
=========================================================== Appendix I
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
Source: State of Florida
Agency for Health Care
Administration, "Health Plan
Performance Evaluation: A Data
Based Approach for Florida,"
Report of the CHPA Data
Advisory Committee, December
1993.
(See figure in printed
edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
David P. Baine, Director, Federal Health Care Delivery Issues
James A. Carlan, Assistant Director, (202) 512-7120
Sandra K. Isaacson, Evaluator-in-Charge
Janice S. Raynor, Evaluator