Nursing Homes: Federal Actions Needed to Improve Targeting and	 
Evaluation of Assistance by Quality Improvement Organizations	 
(29-MAY-07, GAO-07-373).					 
                                                                 
In 2002, CMS contracted with Quality Improvement Organizations	 
(QIO) to help nursing homes address quality problems such as	 
pressure ulcers, a deficiency frequently identified during	 
routine inspections conducted by state survey agencies. CMS	 
awarded $117 million over a 3-year period to the QIOs to assist  
all homes and to work intensively with a subset of homes in each 
state. Homes' participation was voluntary. To evaluate QIO	 
performance, CMS relied largely on changes in homes' quality	 
measures (QM), data based on resident assessments routinely	 
conducted by homes. GAO assessed QIO activities during the 3-year
contract starting in 2002, focusing on (1) characteristics of	 
homes assisted intensively, (2) types of assistance provided, and
(3) effect of assistance on the quality of nursing home care. GAO
conducted a Web-based survey of all 51 QIOs, visited QIOs and	 
homes in five states, and interviewed experts on using QMs to	 
evaluate QIOs.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-373 					        
    ACCNO:   A69965						        
  TITLE:     Nursing Homes: Federal Actions Needed to Improve	      
Targeting and Evaluation of Assistance by Quality Improvement	 
Organizations							 
     DATE:   05/29/2007 
  SUBJECT:   Evaluation methods 				 
	     Long-term care					 
	     Medicaid						 
	     Medicare						 
	     Nursing homes					 
	     Performance measures				 
	     Quality improvement				 
	     Standards						 
	     Surveys						 
	     Quality of care					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-373

   

     * [1]Results in Brief
     * [2]Background

          * [3]Indicators of Nursing Home Quality

               * [4]Survey Deficiencies
               * [5]Quality Measures

          * [6]Evolution of the QIO Program and the Quality Improvement Pro
          * [7]CMS Contract Funding and Requirements
          * [8]Other Nursing Home Quality Improvement and Assurance Initiat

     * [9]QIOs Generally Had a Choice among Homes That Volunteered but

          * [10]QIOs Generally Had a Choice of Which Nursing Homes to Assist
          * [11]Commitment to Working with QIOs Was QIOs' Primary Considerat
          * [12]QIOs Did Not Target Intensive Assistance to Low-Performing H
          * [13]Stakeholders Often Stated QIOs Should Target Intensive Assis

     * [14]QIO Contract Flexibility Resulted in Variation in Assistance

          * [15]Most Quality Improvement Efforts Focused on Chronic Pain and
          * [16]Statewide Interventions Less Variable Than Those for Intensi

               * [17]Statewide Assistance
               * [18]Intensive Assistance
               * [19]QIO and Nursing Home Perspectives on the Interventions

          * [20]QIO Staffing and Turnover Influenced Intensive Participants'

     * [21]QIOs' Impact on Quality Is Not Clear, but Staff at Homes We

          * [22]All QIOs Met Modest Targets for QM Improvement, but the Impa

               * [23]All QIOs Met CMS's Modest Targets for Improvement in
                 Nursing
               * [24]CMS's Use of QMs to Evaluate QIO Performance Is
                 Problematic
               * [25]Influence of Other Factors on Nursing Home Quality Makes
                 It
               * [26]CMS Data Are Too Limited to Evaluate Effectiveness of
                 Specif

          * [27]Homes That Received Intensive Assistance Generally Attribute

     * [28]Conclusions
     * [29]Recommendations for Executive Action
     * [30]Agency Comments and Our Evaluation
     * [31]Appendix I: Scope and Methodology

          * [32]Site Visits

               * [33]Selection of QIOs
               * [34]Selection of Nursing Homes
               * [35]Selection of Stakeholders

          * [36]Analysis of State Survey Data

               * [37]Identifying Homes with Three Standard Surveys
               * [38]Classifying Homes as Low-, Moderately, or
                 High-Performing
               * [39]Determining Statistically Significant Differences
                 between Ho

          * [40]Web-Based Survey of QIOs

     * [41]Appendix II: Publicly Reported Quality Measures
     * [42]Appendix III: Comments from the Centers for Medicare & Medic
     * [43]Appendix IV: GAO Contact and Staff Acknowledgments

          * [44]GAO Contact
          * [45]Acknowledgments

     * [46]Related GAO Products

          * [47]Order by Mail or Phone

Report to the Ranking Member, Committee on Finance, U.S. Senate

United States Government Accountability Office

GAO

May 2007

NURSING HOMES

Federal Actions Needed to Improve Targeting and Evaluation of Assistance
by Quality Improvement Organizations

GAO-07-373

Contents

Letter 1

Results in Brief 4
Background 7
QIOs Generally Had a Choice among Homes That Volunteered but Did Not
Target Assistance to Low-Performing Homes 19
QIO Contract Flexibility Resulted in Variation in Assistance Provided to
Intensive Participants 30
QIOs' Impact on Quality Is Not Clear, but Staff at Homes We Contacted
Attributed Some Improvements to QIOs 39
Conclusions 48
Recommendations for Executive Action 51
Agency Comments and Our Evaluation 51
Appendix I Scope and Methodology 55
Appendix II Publicly Reported Quality Measures 62
Appendix III Comments from the Centers for Medicare & Medicaid Services 63
Appendix IV GAO Contact and Staff Acknowledgments 67
Related GAO Products 68

Tables

Table 1: Quality Measures on Which QIOs Could Focus Their Quality
Improvement Efforts in the 7^th SOW 16
Table 2: Examples of Other Categories of Homes Stakeholders Suggested QIOs
Should Include as Intensive Participants 28
Table 3: Examples of Resident Care Improvements Made by Homes as a Result
of Intensive Assistance Provided by QIOs, 7^th SOW 48
Table 4: QMs as of November 2002 and as of February 2007 62

Figures

Figure 1: Timeline for 7^th SOW Contract and Concurrent Special Studies by
QIOs to Improve the Quality of Nursing Home Care 13
Figure 2: Levels of QIO Assistance and Nursing Home Participation in the
7^th SOW 15
Figure 3: QIO Contract Evaluation Scoring Methodology for the 7^th SOW 17
Figure 4: QIOs' Considerations in Choosing among Homes That Volunteered
for Intensive Assistance in the 7^th SOW 23
Figure 5: Comparison of Nonintensive and Intensive Participants'
Performance on State Surveys 26
Figure 6: QMs Selected by QIOs for Statewide Interventions and QMs
Selected by Nursing Homes for Intensive Assistance, 7^th SOW 31
Figure 7: Statewide Interventions Most Relied on by QIOs, 7^th SOW 33
Figure 8: Intensive Interventions Most Relied on by QIOs and Frequency of
Interventions (Range and Median Number) during the 7^th SOW 35

Abbreviations

CMS Centers for Medicare & Medicaid Services FTE full-time-equivalent HHS
Department of Health and Human Services IOM Institute of Medicine MDS
minimum data set NQF National Quality Forum OSCAR On-Line Survey,
Certification, and Reporting system PARTner Program Activity Reporting
Tool PRO Peer Review Organization QIO Quality Improvement Organization QM
quality measure SOW statement of work

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

May 29, 2007

The Honorable Charles E. Grassley
Ranking Member
Committee on Finance
United States Senate

Dear Senator Grassley:

The federal government plays a major role in the financing and oversight
of nursing home care for individuals who are aged or disabled. Medicare
and Medicaid payments for nursing home services totaled $67 billion in
2004, including a $46 billion federal share.^1 The Centers for Medicare &
Medicaid Services (CMS) defines quality standards that the nation's
approximately 16,400 nursing homes must meet to participate in the
Medicare and Medicaid programs and contracts with state survey agencies to
assess homes' compliance through routine inspections, known as standard
surveys, and through complaint investigations. Under 3-year contracts
beginning in August 2002 and referred to as the 7^th statement of work
(SOW), CMS directed Medicare Quality Improvement Organizations (QIO) to
work with nursing homes to improve the quality of care provided to
residents in the 50 states, the District of Columbia, and the
territories.^2

^1Medicare is the federal health care program for elderly and certain
disabled individuals. Medicare may cover up to 100 days of skilled nursing
home care following a hospital stay. Medicaid is the joint federal-state
health care financing program for certain categories of low-income
individuals. Medicaid also pays for long-term care services, including
nursing home care.

^2QIOs take a variety of forms. They can be for- or not-for-profit
organizations and can be either sponsored by a significant number of
actively practicing area physicians or have available to them a sufficient
number of these physicians to assure adequate peer review. In general,
QIOs cannot be health care facilities. Prior to 1999, QIOs focused on
quality improvement in the hospital setting. Beginning in 1999, CMS
required QIOs to also work in an alternative setting; about two-thirds
selected nursing homes. The QIOs currently also work with physician
offices, home health agencies, rural or underserved populations, and
Medicare Advantage organizations to improve Medicare beneficiaries'
quality of care. For the 7th SOW, the 53 QIO contracts, one for each
state, the District of Columbia, and 2 territories (Puerto Rico and the
Virgin Islands) were held by 37 organizations. We excluded the 2
territories from our study because of substantial differences in health
care financing between the territories and the states.

As a condition of their contracts, QIOs were required to provide (1)
information to all Medicare- or Medicaid-certified nursing homes in each
state about systems-based approaches to improving resident care and
clinical outcomes and (2) intensive assistance to a subset of each state's
homes, typically 10 to 15 percent, that were selected by the QIOs from
among those homes that volunteered for assistance.

In a series of congressionally requested studies undertaken since 1998, we
have reported on the unacceptably high proportion of nursing homes
providing poor care to residents.^3 Based in part on our recommendations,
CMS has undertaken a number of enforcement initiatives to encourage
nursing home compliance with federal quality standards, including improved
oversight by both state survey agencies and CMS, and tougher enforcement
measures to ensure that homes correct deficiencies and maintain compliance
with federal standards. For example, CMS expanded its Special Focus
Facility program in which state agencies survey selected homes more
frequently and terminate those that fail to improve significantly within
18 months.

CMS's decision to offer direct assistance to nursing homes that volunteer
to work with QIOs represents a new strategy in the effort to help address
long-standing quality problems in nursing homes. To evaluate QIO
performance in improving nursing home care, CMS relied primarily on
changes in nursing homes' quality measures (QM) during the contract
period. QMs are numeric measures derived from resident assessments--known
as the minimum data set (MDS)--that nursing homes routinely conduct and
submit to CMS.^4 The QMs were developed to permit comparisons across
nursing homes of the quality of care provided to residents and have been
publicly reported on CMS's Nursing Home Compare Web site since 2002.^5

In 2005, CMS renewed the QIO contracts, including the nursing home
component, for another 3-year period, with a budget of $96 million to
assist nursing homes.^6 Given the decision to continue the program, you
asked us to assess QIOs' work with nursing homes for the 7^th SOW,
covering the period August 2002 through January 2006. For this report, we
assessed (1) characteristics of nursing homes the QIOs assisted
intensively, (2) the assistance the QIOs provided to nursing homes, and
(3) the effect of QIOs' assistance on the quality of nursing home care.

^3See Related GAO Products at the end of this report.

^4The minimum data set (MDS) consists of data that are periodically
collected to assess the care needs of residents in order to develop an
appropriate plan of care. State surveyors use MDS data to help assess the
quality of resident care, and Medicare and some state Medicaid programs
also use MDS data to adjust nursing home payments.

^5The Web site can be accessed at [48]www.Medicare.gov/NHCompare/home.asp
.

To assess the characteristics of nursing homes that the QIOs selected to
assist intensively from among the homes that volunteered, we analyzed CMS
data on deficiencies cited in standard surveys of nursing homes and
compared the results for homes assisted intensively by the QIOs with homes
that were not assisted intensively.^7 To gather information about the
QIOs' criteria for selecting homes for intensive assistance, we fielded a
Web-based survey to the 37 organizations that held the 51 QIO contracts in
the states and the District of Columbia, achieving a 100 percent response
rate.^8 To determine the type of quality improvement assistance QIOs
provided to nursing homes, our Web-based survey collected data on the
types, frequency, and perceived effectiveness of specific interventions
used to assist homes both statewide and in the group assisted intensively;
interventions included activities such as on-site visits, mailings, and
conferences. To gather more detailed information about QIOs' work with
nursing homes, we conducted site visits to five states--Colorado, Florida,
Iowa, Maine, and New York--where we interviewed QIO personnel, staff from
nursing homes that had received intensive assistance, and key
stakeholders.^9 The five states accounted for 15 percent of nursing home
beds nationwide in 2002 and represented a range in terms of such
characteristics as number of nursing home beds, region of the country, and
QIOs' performance on the nursing home component in the 7^th SOW. In the
five states, we interviewed staff from 28 nursing homes*4 to 8 per state;
in addition, we interviewed staff from 4 homes in four other states for a
total of 32 homes. We sought to select a group of homes that represented a
range in terms of state survey deficiencies, improvement in QM scores
during the 7^th SOW, distance from the QIO, and urban versus rural
location. However, the experiences of the 32 homes in our sample cannot be
generalized to all homes that received intensive assistance from the QIOs
nationwide. To assess the effect of QIOs' assistance on nursing home
quality, we reviewed performance requirements in the QIO contracts for
both the 7^th and the 8^th SOWs; reports on QIOs' work with nursing homes,
including the 2006 report on the QIO program by the Institute of Medicine
(IOM);^10 and other documents. We also conducted interviews with nursing
homes, CMS officials, officials from state quality assurance programs and
state MDS accuracy review programs, and experts on the nursing home QMs
and the MDS data on which they are based. We conducted our review from
October 2005 through May 2007 in accordance with generally accepted
government auditing standards. (For a more detailed description of our
scope and methodology, see app. I.)

^6The QIO contract is divided into tasks and subtasks; the nursing home
component is subtask 1a. The amount budgeted for this component in the 8th
SOW (the QIO contract covering the period from 2005 through 2008) was
approximately $10 million less than was budgeted in the 7th SOW.

^7We ranked nursing homes as high-, moderately, or low-performing on the
basis of the number, scope, and severity of the deficiencies for which
they were cited (relative to other homes in their state) in three standard
state surveys from 1999 through 2002. We based our classification of homes
on their performance level relative to other homes in the state to take
into account the inconsistency in how states conduct surveys, a problem we
have reported on since 1998. A limitation of our analysis is that we did
not have information about all of the homes that volunteered for intensive
assistance, only those that were selected by the QIOs, and therefore did
not know the extent to which low-performing homes volunteered for
intensive assistance.

^8Because a QIO is responsible for quality improvement activities in each
state and the District of Columbia, we refer to the 51 QIOs throughout
this report.

Results in Brief

Although QIOs generally had a choice of homes to select for intensive
assistance because more homes volunteered than CMS expected QIOs to
assist, QIOs typically did not target the low-performing homes that
volunteered. Most QIOs reported in our Web-based survey that their primary
consideration in selecting homes was their commitment to working with the
QIO. CMS did not specify selection criteria for intensive participants but
contracted with a QIO to develop guidelines, which encouraged QIOs to
select homes that appeared committed to quality improvement and to exclude
homes with a high number of survey deficiencies, high management turnover,
or QM scores that were too good to improve significantly. Consistent with
the guidelines, only 2 percent of the QIOs that responded to our survey
cited a high level of survey deficiencies among their top three
considerations in choosing among homes that volunteered for assistance,
and eight QIOs explicitly excluded such homes. QIOs reasoned that these
homes might be more focused on improving their survey results than on
committing time and resources to quality improvement projects that might
target other care areas. Our analysis of state survey data showed that,
nationwide, intensive participants were less likely to be low-performing
than other homes in their state in terms of the number, scope, and
severity of deficiencies for which they were cited in standard surveys
from 1999 through 2002. This result may reflect the nature of the homes
that volunteered for assistance, the QIOs' selection criteria, or a
combination of the two. Most of the stakeholders we interviewed who
expressed an opinion said that QIOs' resources should be targeted to
low-performing homes. CMS has directed a small share of QIO resources to
low-performing homes in the current 8^th SOW. Specifically, each QIO is
required to provide intensive assistance to up to three "persistently
poor-performing homes" identified in consultation with the state survey
agency.

^9To assist in the development of our site visit interview protocols, we
also interviewed personnel from three other QIOs. On each of our five site
visits, we interviewed officials from three stakeholder groups: (1) the
state survey agency; (2) the local affiliate for the American Health Care
Association, which generally represents for-profit homes; and (3) the
local affiliate for the American Association of Homes and Services for the
Aging, which represents not-for-profit homes.

^10IOM of The National Academies, Committee on Redesigning Health
Insurance Performance Measures, Payment, and Performance Improvement
Programs, Board on Health Care Services, Medicare's Quality Improvement
Organization Program: Maximizing Potential (Washington, D.C.: The National
Academies Press, 2006). The Medicare Prescription Drug, Improvement, and
Modernization Act of 2003, Pub. L. No. 108-173, S109(d), 117 Stat. 2066,
2173-74, directed the Secretary of Health and Human Services to ask the
IOM to conduct an evaluation of the QIO program administered by CMS. In
2006, the IOM issued a report that examined performance within the entire
QIO program, including the nursing home component, during the 7th SOW.

The 7^th SOW contracts allowed QIOs flexibility in the QMs they focused on
and the interventions they used. While the majority of QIOs selected the
same QMs and most used the same interventions to assist homes statewide,
the interventions used to assist intensive participants and staffing to
accomplish program goals varied. Of eight possible QMs, most QIOs and
intensive participants worked on chronic pain and pressure ulcers.^11
While intensive participants were supposed to have a choice of QMs to
focus on, some intensive participants told us that the QIO made the
selection and that chronic pain and pressure ulcers were not necessarily
their greatest quality-of-care challenges. The interventions QIOs relied
on most for homes statewide were conferences and the distribution of
educational materials; for intensive participants, they relied most on
on-site visits, conferences, and small group meetings.^12 Although the
interventions QIOs used with intensive participants varied, most QIOs (63
percent) considered on-site visits the most effective, and some would make
on-site visits their primary intervention if they had the opportunity to
change the interventions they used during the 7^th SOW. Insufficient
experience or expertise and high turnover among QIO personnel negatively
affected homes' satisfaction with the program and the extent of their
quality improvements. Turnover was particularly high at 24 of the 51 QIOs,
where one-quarter or more of the QIO personnel who assisted nursing homes
worked less than half of the 36-month contract. One intensive participant
home had four QIO principal contacts over the course of the 3-year
contract.

^11A pressure ulcer is an area of damaged skin and tissue that results
from constant pressure due to an individual's impaired mobility. The
pressure results in reduced blood flow and eventually causes cell death,
skin breakdown, and the development of an open wound. Pressure ulcers can
occur in individuals who are bed- or wheelchair-bound, sometimes after
only a few hours.

The impact of QIOs on the quality of nursing home care cannot be
determined from available data, but at most nursing homes we contacted,
staff attributed some improvements in the quality of resident care to
their work with QIOs. Nursing homes' QM scores generally improved enough
for all of the QIOs to meet--and some to surpass widely*the modest targets
set by CMS for improvement among homes both statewide and in the group
assisted intensively. However, the overall impact of the QIOs on the
quality of nursing home care cannot be determined from these data because
of the shortcomings of the QMs as measures of nursing home quality and
because confounding factors*including homes' participation in other
quality improvement efforts and any preexisting differences between homes
that volunteered and were selected for intensive assistance and other
homes*make it difficult to attribute quality improvements solely to the
QIOs. Multiple long-term care professionals we interviewed stated that QMs
should not be used in isolation to measure quality improvement, but
combined with other indicators, such as state survey data. In addition,
the effectiveness of the individual interventions QIOs used to assist
homes cannot be evaluated with the limited data CMS collected from the
QIOs. CMS planned to enhance evaluation of the program during the 8^th
SOW, but a determination by the Department of Health and Human Services
(HHS) Office of General Counsel that the QIO program regulations prohibit
QIOs from providing to CMS the identities of the homes they are assisting
has hampered the agency's efforts to collect the necessary data. Although
we cannot determine the overall impact of the QIOs on the quality of
nursing home care, over two-thirds of the 32 nursing homes we interviewed
attributed some improvements in care to their work with the QIOs.

^12In our survey of the QIOs, we asked them to identify the interventions
they relied on most and the interventions that were most effective in
improving the quality of nursing home care; we allowed the QIOs to define
these terms.

We are recommending that the CMS Administrator increase the extent to
which QIOs target intensive assistance to low-performing homes and also
direct QIOs to focus intensive assistance on the quality-of-care areas on
which homes most need improvement. We are also recommending that the CMS
Administrator improve monitoring and evaluation of the QIO program by
revising program regulations to require QIOs to provide to CMS the
identities of the nursing homes they are assisting, collecting more
complete and detailed data on QIO interventions, and identifying a broader
spectrum of measures than QMs to evaluate changes in nursing home quality.
In commenting on a draft of this report, CMS concurred with but did not
indicate how it would implement our recommendations to increase the number
of homes that QIOs assist intensively and collect more complete and
detailed data on the interventions QIOs use to assist homes. CMS did not
specifically indicate if it agreed with our recommendation to revise
program regulations to allow QIOs to reveal to CMS the identity of the
nursing homes they are assisting, but did indicate that it continues to
explore options which would allow access to such data in order to
facilitate evaluation. CMS did not comment on the remaining two
recommendations.

Background

Beginning in the late 1990s, CMS took steps to broaden the mechanisms in
place intended to help ensure that nursing home residents receive quality
care. To augment the periodic assessment of homes' compliance with federal
quality requirements, CMS contracted for the development of QMs and tasked
QIOs with providing assistance to homes to improve quality. CMS used QMs
both to provide the public with information on nursing home quality of
care and to help evaluate QIO efforts to address quality-of-care issues,
such as pressure ulcers. During the 7^th SOW, organizations other than
QIOs were also working with nursing homes to improve quality.

Indicators of Nursing Home Quality

Two indicators used by CMS to assess the quality of care that nursing
homes provide to residents are (1) deficiencies identified during standard
surveys and complaint investigations and (2) QMs. Both indicators are
publicly reported on CMS's Nursing Home Compare Web site.

  Survey Deficiencies

Under contract with CMS, state agencies conduct standard surveys to
determine whether the care and services provided by nursing homes meet the
assessed needs of residents and whether nursing homes are in compliance
with federal quality standards.^13 These standards include preventing
avoidable pressure ulcers; avoiding unnecessary restraints, either
physical or chemical; and averting a decline in a resident's ability to
perform activities of daily living, such as toileting or walking.^14
During a standard survey, a team that includes registered nurses spends
several days at a home reviewing the quality of care provided to a sample
of residents. States are also required to investigate complaints filed
against nursing homes by residents, families, and others. Complaint
investigations are less comprehensive than standard surveys because they
generally target specific allegations raised by the complainants.

Any deficiencies identified during standard surveys or complaint
investigations are classified according to the number of residents
potentially or actually affected (isolated, pattern, or widespread) and
their severity (potential for minimal harm, potential for more than
minimal harm, actual harm, or immediate jeopardy). Deficiencies cited at
the actual harm and immediate jeopardy level are considered serious and
could trigger enforcement actions such as civil money penalties. We have
previously reported on the considerable interstate variation in the
proportion of homes cited for serious care problems, which ranged during
fiscal year 2005 from 4 percent of Florida's 691 homes to 44 percent of
Connecticut's 247 homes.^15 We reported that such variability suggests
inconsistency in states' interpretation and application of federal
regulations; in addition, both we and CMS have found that state surveyors
do not identify all serious deficiencies.^16

^13CMS's Survey and Certification Group is responsible for oversight of
state survey agency activities.

^14Surveys must be conducted at each home on average once every 12 months
but no less than once every 15 months.

^15This analysis excluded 13 states because fewer than 100 homes were
surveyed, and even a small increase or decrease in the number of homes
with serious deficiencies in such states could produce a relatively large
percentage-point change. In fiscal year 2005, about 17 percent of the
16,337 homes surveyed had serious deficiencies. See GAO, Nursing Homes:
Efforts to Strengthen Federal Enforcement Have Not Deterred Some Homes
from Repeatedly Harming Residents, [49]GAO-07-241 (Washington, D.C.: Mar.
26, 2007).

  Quality Measures

QMs are relatively new indicators of nursing home quality. Although survey
deficiencies have been publicly reported since 1998, CMS did not begin
posting QMs on its Nursing Home Compare Web site until November 2002. QMs
are derived from resident assessments known as the MDS that nursing homes
routinely collect on all residents at specified intervals.^17 Conducted by
nursing home staff, MDS assessments cover 17 areas, such as skin
conditions, pain, and physical functioning.

In developing QMs, CMS recognized that any publicly reported indicators
must pass a rigorous standard for validity and reliability. In October
2002, we reported that national implementation of QMs was premature
because of validity and reliability concerns.^18 Valid QMs would
distinguish between good and poor care provided by nursing homes; reliable
QMs would do so consistently. One of our main concerns about publicly
reporting QMs was that the QM scores might be influenced by other factors,
such as residents' health status. As a result, the specification of
appropriate risk adjustment was a key requirement for the validity of any
QMs. Risk adjustment is important because it provides consumers with an
"apples-to-apples" comparison of nursing homes by taking into
consideration the characteristics of individual residents and adjusting
the QM scores accordingly. For example, a home with a disproportionate
number of residents who are bedfast or who present a challenge for
maintaining an adequate level of nutrition--factors that contribute to the
development of pressure ulcers--may have a higher pressure ulcer score.
Adjusting a home's QM score to fairly represent to what extent a home does
or does not admit such residents is important for consumers who wish to
compare one home to another. Appendix II lists the 10 QMs initially
adopted and publicly reported by CMS--6 applicable to residents with
chronic care problems (long-stay residents) and 4 applicable to residents
with post-acute-care needs (short-stay residents).

^16CMS is evaluating a new survey methodology to help ensure that
surveyors do not miss serious care problems. National implementation will
depend on the outcome of the evaluation.

^17MDS assessments are conducted for all nursing home residents within 14
days of admission and at quarterly and yearly intervals unless there is a
significant change in condition. In addition, Medicare beneficiaries in a
Medicare-covered stay are assessed through MDS on or before the 5^th,
14^th, 30^th, 60^th, and 90^th day of their stays to determine if their
Medicare coverage should continue.

^18GAO, Nursing Homes: Public Reporting of Quality Indicators Has Merit,
but National Implementation Is Premature, [50]GAO-03-187 (Washington,
D.C.: Oct. 31, 2002).

MDS data are self-reported by nursing homes, and ensuring their accuracy
is critical for establishing resident care plans, setting nursing home
payments, and publicly reporting QM scores. In February 2002, we concluded
that CMS efforts to ensure the accuracy of MDS data, which are used to
calculate the QMs, were inadequate because the agency relied too much on
off-site review activities by its contractor and planned to conduct
on-site reviews in only 10 percent of its data accuracy assessments,
representing fewer than 200 of the nation's then approximately 17,000
nursing homes.^19 Although we recommended that CMS reorient its review
program to complement ongoing state MDS accuracy efforts as a more
effective and efficient way to ensure MDS data accuracy, CMS disagreed and
continued to emphasize off-site reviews.^20

Evolution of the QIO Program and the Quality Improvement Process

Over the past 24 years, the QIO program has evolved from a focus on
quality assurance in the acute care setting to quality improvement in a
broader mix of settings, including physician offices, home health
agencies, and nursing homes. Established by the Peer Review Improvement
Act of 1982^21 and originally known as Peer Review Organizations (PRO),
QIOs initially focused on ensuring minimum standards by conducting
retrospective hospital-based utilization reviews that looked for
inappropriate or unnecessary Medicare services. According to the 2006 IOM
report, as it became apparent that standards of care themselves required
attention, QIOs gradually shifted from retrospective case reviews to
collaboration with providers to improve the overall delivery of care--a
shift consistent with transformational goals set by CMS's Office of
Clinical Standards and Quality, which oversees the QIO program.^22

^19See [51]GAO-02-279 .

^20Some states that adjust nursing home payments to account for variation
in resident care needs have their own separate MDS review programs.

^21Pub. L. No. 97-248, S141-50, 96 Stat. 381-95. PROs were renamed QIOs in
2002. Under the provisions of the Peer Review Improvement Act of 1982 and
implementing regulations, a QIO can be either a physician-sponsored entity
or a physician-access entity. See 42 C.F.R. S475.101 (2005). QIOs are
allowed to be either for- or not-for-profit entities and are required to
include at least one consumer representative on the QIO governing board.
Funding for QIO activities comes from the Medicare Trust Funds.

^22IOM, Medicare's Quality Improvement Organization Program: Maximizing
Potential.

In contrast to enforcing standards, quality improvement tries to ensure
that organizations have effective processes for continually measuring and
improving quality. The goal of quality improvement is to close the gap
between an organization's current performance and its ideal performance,
which is defined by either evidence-based research or best practices
demonstrated in high-performing organizations. According to the quality
improvement literature, successful quality improvement requires a
commitment on the part of an organization's leadership and active
involvement of the staff. The 2006 IOM report notes that QIOs rely on
various mechanisms to promote quality improvement, including one-on-one
consulting and collaboratives.^23 While the former provides direct and
specialized attention, the latter relies on workshops or meetings that
offer opportunities for providers to share experiences and best practices.
Quality improvement often relies on the involvement of early adopters of
best practices--providers who are highly regarded as leaders and can help
convince others to change--for the diffusion of best practices. Key tools
for quality improvement include (1) root cause analysis, a technique used
to identify the conditions that lead to an undesired outcome; (2)
instruction on how to collect, aggregate, and interpret data; and (3)
guidance on bringing about, sustaining, and diffusing internal system
redesign and process changes, particularly those related to use of
information technology for quality improvement. Quality improvement
experts also emphasize the importance of protecting the confidentiality of
provider information, not only to protect the privacy of personal health
information but also to encourage providers to evaluate their peers
honestly and to prevent the damage to providers' reputations that might
occur through the release of erroneous information.

Section 1160 of the Social Security Act provides that information
collected by QIOs during the performance of their contract with CMS must
be kept confidential and may not be disclosed except in specific
instances; it provides the Secretary of HHS with some discretion to
determine instances under which QIO information may be disclosed. The
regulations implementing the statute limit the circumstances under which
confidential information obtained during QIO quality review studies,
including the identities of the participants of those studies, may be
disclosed by the QIO. During the 7^th SOW, QIOs submitted a list of
nursing home participants to CMS as a contract deliverable.

^23IOM defines collaboratives as interventions designed to bring together
stakeholders working toward quality improvement for the same clinical
topic. Participants usually follow the same processes to reach goals and
interact on a regular basis to share knowledge, experiences, and best
practices.

CMS Contract Funding and Requirements

During the 7^th SOW, CMS awarded a total of $117 million to QIOs to
improve the quality of care in nursing homes in all 50 states, the
District of Columbia, and the territories. The performance-based contracts
for QIO assistance to nursing homes delineated broad expectations
regarding QIO assistance to nursing homes, provided deadlines for
completing four contract deliverables, and laid out criteria for
evaluating QIO performance.^24 For contracting purposes, the QIOs were
divided into three groups with staggered contract cycles. The four
contract deliverables, however, were all due on the same dates,
irrespective of the different contract cycles. The contracts also required
QIOs to work with a QIO support contractor tasked to provide guidelines
for recruiting and selecting nursing homes as intensive participants,
train QIOs in standard models of quality improvement assistance, and
provide tools and educational materials, as well as individualized
consultation if needed, to help QIOs meet contractual requirements.^25
QIOs and nursing homes were also involved in other quality improvement
special studies with budgets separate from the QIO contracts for the 7^th
SOW. These studies varied greatly in terms of length, the clinical
issue(s) covered, the number of QIOs involved, and the characteristics of
the nursing homes that participated. Figure 1 shows the 7^th SOW contract
cycles, deliverables for the nursing home component, and the duration of
the special studies.

^24According to a CMS official, all QIO contracts prior to the 6th SOW,
which began in 2000, were considered "cost plus fixed fee" and there were
no deliverables, or set targets, that QIOs had to meet in order to obtain
payment. In the late 1990s, however, the Office of Management and Budget
instructed CMS to make QIO contracts performance-based with deliverables
and objectives that QIOs had to meet during the contract cycle. In
response, CMS changed the QIO contract so that part of QIOs' fee was based
on their performance.

^25The Rhode Island QIO was awarded the support contract for nursing homes
for the 7^th SOW. The contract defined roles for the QIO support
contractor, including (1) providing QIOs with information on clinical
topics and management systems' approaches and techniques for quality
improvement; (2) facilitating coordination and communication between QIOs;
(3) maintaining a nursing home informational clearinghouse Web site with
best practices, tools, and interventions; and (4) being available for
ongoing technical assistance.

Figure 1: Timeline for 7^th SOW Contract and Concurrent Special Studies by
QIOs to Improve the Quality of Nursing Home Care

aIn the 7^th SOW, QIOs were divided into three groups with staggered
contract cycles. The four contract deliverables, however, were all due on
the same dates, irrespective of contract cycle.

bThe term states includes the 50 United States and the District of
Columbia.

cQIOs could add--but not delete or change--QMs for their intensive
participants through September 2003.

Contract funding. The $117 million awarded to QIOs to improve the quality
of care in nursing homes during the 7^th SOW included (1) $106 million
awarded to provide statewide and intensive assistance to homes,^26 (2)
$5.6 million awarded to selected QIOs to conduct eight special studies
focused on nursing home care, and (3) $5.3 million awarded to the QIO that
served as the support contractor for the nursing home component.^27 CMS
allocated a specific amount for each component of the contracts, but
allowed QIOs to move funds among certain components.^28 Just over half of
the 51 QIOs did not spend all of the funds allocated to the nursing home
component, but on average the QIOs overspent the budget for the nursing
home work by 3 percent.

Contract requirements for quality improvement activities. Per the
contracts for the 7^th SOW, QIOs were required to provide (1) all
Medicare- and Medicaid-certified homes with information about
systems-based approaches to improving patient care and clinical outcomes,
and (2) intensive assistance to a subset of homes in each state. The
contracts directed QIOs working in states with 100 or more nursing homes
to target 10 to 15 percent of the homes for intensive assistance.^29
Figure 2 illustrates that QIOs provided two levels of
assistance--statewide and intensive--and that homes' participation was
either nonintensive or intensive. Intensive participants received both
statewide and intensive assistance. Selection of intensive participants
from among the nursing homes that volunteered was at the discretion of
each QIO, but the SOW required the QIO support contractor (the Rhode
Island QIO) to provide guidelines and criteria for QIOs to use in
determining which homes to select. Participation in the program was
voluntary, and QIOs were prohibited from releasing the names of
participating nursing homes except as permitted by statute and
regulation.^30

26The $106 million represented 13 percent of the total amount ($809
million) awarded to QIOs for their base contracts. CMS did not budget
separately for statewide and intensive assistance.

^27The QIO support contractor subcontracted with another QIO to provide
data analysis.

^28For example, QIOs could move funds between the nursing home component
and the other components under task 1, which covered clinical quality
improvement efforts with home health agencies, hospitals, physician
offices, underserved and rural beneficiaries, and Medicare Advantage
organizations.

^29QIOs working in the 13 states with fewer than 100 nursing homes were
expected to target at least 10 homes.

Figure 2: Levels of QIO Assistance and Nursing Home Participation in the
7^th SOW

aNursing homes on the official list of intensive participants submitted to
CMS by the QIOs by February 3, 2003.

Under the contracts, the quality improvement assistance provided by QIOs
focused on areas related to eight chronic care and post-acute-care QMs
publicly reported on the CMS Nursing Home Compare Web site. QIOs were
required to consult with relevant stakeholders and select from three to
five of the eight QMs on which QIOs' quality improvement efforts would be
evaluated (see table 1).^31 Intensive participant homes were also required
to select one or more QMs on which to work with the QIO. Although they
could select one QM, they were encouraged to select more than one.

^30See Social Security Act S1160; 42 C.F.R. S480.140 (2005).

^31Stakeholders may include representatives of nursing homes, trade
associations, ombudsmen, state survey agencies, medical directors,
directors of nursing, geriatric nursing assistants, other licensed
professionals, academicians, and consumers.

Table 1: Quality Measures on Which QIOs Could Focus Their Quality
Improvement Efforts in the 7^th SOW

Chronic care QMs                    Post-acute-care QMs                    
Decline in activities of daily      Failure to improve and manage delirium 
living                                                                     
Pressure ulcers                     Inadequate pain management             
Inadequate pain management          Improvement in walking                 
Physical restraints                                                        
Infections                                                                 

Source: CMS.

Note: Although CMS adopted 10 QMs, the QIOs were evaluated only on the 8
listed here (see app. II).

To improve QM scores, QIOs were expected to develop and implement quality
improvement projects focused on care processes known to improve patient
outcomes in a manner that utilized resources efficiently and reduced
burdens on providers. The QIO support contractor developed a model for
QIOs to facilitate systems change in nursing homes. This model emphasized
the importance of QIOs' statewide activities to form and maintain
partnerships, conduct workshops and seminars, and disseminate information
on interventions to improve quality. For intensive participants, the model
emphasized conducting one-on-one quality improvement assistance as well as
conferences and small group meetings. According to contract language, QIOs
were expected to coordinate their projects with other stakeholders that
were working on similar improvement efforts or were interested in teaming
with the QIO. But ultimately, each QIO determined for itself the type,
level, duration, and intensity of support it would offer to nursing homes.

Evaluation of QIO contract performance. CMS evaluated QIOs' performance on
the nursing home component of the contract using nursing home provider
satisfaction with the QIO, QM improvement among intensive participants,
and QM improvement statewide (see fig. 3).^32 Nursing home provider
satisfaction was assessed by surveying all intensive participants and a
sample of nonintensive participants around the 28^th month of each
36-month contract. CMS expected at least 80 percent of respondents to
report that they were either satisfied or very satisfied.

^32Under the 8th SOW contract, QIOs will not be held accountable for QM
improvement statewide.

Figure 3: QIO Contract Evaluation Scoring Methodology for the 7^th SOW

Note: QM improvement was calculated using the following formula: (baseline
rate minus remeasurement rate) / baseline rate. For example, if a nursing
home had a baseline rate of 20 percent for the pain management QM (e.g.,
20 percent of the home's residents had severe or moderate pain), a 10
percent improvement would mean that 18 percent of residents had moderate
or severe pain at remeasurement [(20 percent - 18 percent) / 20 percent].

aAll intensive participants and a sample of nonintensive participants were
surveyed to assess their satisfaction with the QIO.

bThe weight (percentage of total score) given to this element depended on
the proportion of the state's homes that were included in the intensive
participant group; the weight ranged from 44 percent, if 10 percent of the
homes were included, to 66 percent, if at least 15 percent of the homes
were included.

cThe weight (percentage of total score) given to this element was the
difference between 80 percent and the weight given to the intensive
participant element and ranged from 14 to 36 percent.

QIOs were also expected to achieve an 8 percent improvement in QM scores
among both intensive participants and homes statewide. The term
improvement was defined mathematically to mean the relative change in the
QM score from when it was measured at baseline to when it was remeasured.
The statewide improvement score included the QM improvement scores for
intensive participants averaged with those of nonintensive participants.

CMS established two scoring thresholds for the contracts that encompassed
scores from all components of the SOW. If a QIO scored above the first
threshold it was eligible for a noncompetitive contract renewal; if it
scored below that threshold, it was eligible for a competitive renewal
only upon providing information pertinent to its performance to a CMS-wide
panel that decided whether to allow the QIO to bid again for another QIO
contract.^33

CMS contract monitoring. CMS formally evaluated each QIO at months 9 and
18 of the 7^th SOW. If CMS found that a QIO failed to meet contract
deliverables or appeared to be in danger of failing to meet contract
goals, it could require the QIO to make a performance improvement plan or
corrective plan of action to address any barriers to the QIOs successfully
fulfilling contract requirements. In addition, CMS reviewed materials such
as QIOs internal quality control plans, which were intended to help QIOs
monitor their own progress and to document any project changes made to
improve their performance.

Other Nursing Home Quality Improvement and Assurance Initiatives

The QIO program operated in the context of other quality improvement
initiatives sponsored by federal and state governments and nursing home
trade associations. As stated earlier, CMS funded a number of special
nursing home studies involving subsets of the QIOs and nursing homes,
which addressed a variety of clinical quality-of-care issues and which are
summarized in figure 1. Under CMS's Special Focus Facility program, state
survey agencies were required to conduct enhanced monitoring of nursing
homes with histories of providing poor care. During the 7^th SOW, CMS
revised the method for selecting homes for the Special Focus Facility
program to ensure that the homes performing most poorly were included;
increased the minimum number of homes that must be included, from a
minimum of two per state to a minimum of up to six, depending on the
number of homes in the state; and strengthened enforcement for those
nursing homes with an ongoing pattern of substandard care.^34 In addition,
concurrent with the 7^th SOW, at least eight states had programs that
provided quality assurance and technical assistance to nursing homes in
their states.^35 These programs varied in terms of whether they were
voluntary or mandatory, which homes received assistance, the focus and
frequency of the assistance provided, and the number and type of staff
employed.

^33QIOs could fail to meet contract expectations for up to 2 of the 12
components and still remain eligible for noncompetitive renewal of their
contracts.

^34Initiated in January 1999, the Special Focus Facility program was
expanded by CMS in December 2004. Expansion strengthened enforcement
authority so that if homes in the program fail to significantly improve
performance from one survey to the next, immediate sanctions must be
imposed; if homes show no significant improvement in 18 months and three
surveys, they must be terminated from participation in the Medicare and
Medicaid programs.

In addition to government-operated quality improvement initiatives, three
long-term care professional associations joined together in July 2002 to
implement the Quality First Initiative.^36 This initiative was based on a
publicly articulated pledge on the part of the long-term care profession
to establish an environment of continuous quality improvement, openness,
and leadership in participating homes.

QIOs Generally Had a Choice among Homes That Volunteered but Did Not Target
Assistance to Low-Performing Homes

Although QIOs generally had a choice of homes to select for intensive
assistance because more homes volunteered than CMS expected QIOs to
assist, QIOs typically did not target the low-performing homes that
volunteered. Most QIOs reported in our Web-based survey that they did not
have difficulty recruiting homes, and their primary consideration in
selecting homes from the pool of volunteers was that the homes be
committed to working with the QIOs. In the 7^th SOW, CMS did not specify
recruitment and selection criteria for intensive participants, leaving the
development of guidelines to the QIO support contractor, which encouraged
QIOs to select homes that seemed committed to quality improvement and to
exclude homes with a high number of survey deficiencies, high management
turnover, or QM scores that were too good to improve significantly.^37 Our
analysis of state survey data showed that, nationwide, intensive
participants were less likely to be low-performing than other homes in
their state in terms of the number, scope, and severity of deficiencies
for which they were cited in standard surveys from 1999 through 2002. This
result may reflect the nature of the homes that volunteered for
assistance, the QIOs' selection criteria, or a combination of the two. The
stakeholders we interviewed--including officials of state survey agencies
and nursing home trade associations--generally believed QIOs' resources
should be targeted to low-performing homes.

^35The eight states are Florida, Maryland, Texas, Washington, Maine,
Michigan, Missouri, and North Carolina. We identified some of these states
by reviewing reports and asking officials in states that we knew had
quality assurance programs to identify other states with similar programs.
We did not attempt to determine if additional states had similar programs.

^36The organizations included the American Health Care Association, the
Alliance for Quality Nursing Home Care, and the American Association of
Homes and Services for the Aging, which are three of the largest long-term
care organizations and together represent the majority of the
approximately 16,400 nursing facilities in the United States.

^37In the 8th SOW contracts, CMS specified more selection parameters,
requiring QIOs to work with two groups of intensive participants,
including some "persistently poor-performing" homes identified in
consultation with state survey agencies; increasing the overall number of
intensive participants; and requiring geographic distribution of these
homes.

QIOs Generally Had a Choice of Which Nursing Homes to Assist Intensively

Most QIOs had a choice of which nursing homes to assist intensively, as
more homes volunteered than the QIOs could receive credit for serving
under the terms of their contracts.^38 Of the 38 QIOs in states with 100
or more homes, which were expected to work intensively with 10 to 15
percent of the homes, 30 reported in our Web-based survey that more than
15 percent of homes expressed interest in intensive assistance, and 8
reported that more than 30 percent of homes expressed interest.^39 Most
QIOs selected about as many intensive participants as needed to get the
maximum weight for the intensive participant element of their contract
evaluation score. Nationwide, the intensive participant group included
just under 15 percent (2,471) of the 16,552 homes identified by CMS at the
beginning of the 7^th SOW.^40

Most QIOs--82 percent of the 51 that responded to our survey--reported
that it was not difficult to recruit the target number of homes for
intensive assistance; the remainder reported that it was difficult (12
percent) or very difficult (4 percent) to recruit enough volunteers.^41
Among the QIOs we interviewed, personnel at two that reported difficulties
recruiting homes cited homes' lack of familiarity with QIOs as a barrier.
Personnel at one of these two QIOs commented that the QIO's first task was
to build trust among homes and address confusion about its role, as some
homes thought the QIO was a regulatory authority charged with
investigating complaints and citing homes for deficiencies.

^38QIOs could select more than 15 percent of the homes in their state for
intensive assistance. However, the weight given to this component in a
QIO's contract evaluation score could not exceed 66 percent--generally,
the weight given if the intensive participant group comprised 15 percent
of homes in the state.

^39The 13 QIOs in states with fewer than 100 homes were expected to work
intensively with at least 10 homes.

^40The 38 QIOs that were expected to work intensively with 10 to 15
percent of the homes in their state worked with an average of 15 percent.
The other 13 QIOs worked with an average of 15 homes.

^41The largest proportion of QIOs (27 percent) reported that their most
effective recruiting tactic was hosting statewide or regional conferences
for homes; however, 20 percent did not use this tactic at all. The vast
majority of QIOs (84 to 98 percent) also sent materials to homes,
contacted homes by telephone, and asked nursing home trade associations or
other groups to inform homes of the opportunity to participate.

Commitment to Working with QIOs Was QIOs' Primary Consideration in Selecting
Homes from among Those That Volunteered

QIOs that responded to our Web-based survey almost uniformly cited homes'
commitment to working with them as a key consideration in choosing among
the homes that volunteered to be intensive participants. QIOs had wide
latitude in choosing among homes because CMS did not specify the
characteristics of the homes they should recruit or select, leaving it to
the QIO support contractor to provide voluntary guidelines. The QIO
support contractor developed guidelines based on input from a variety of
sources, including QIOs that worked with nursing homes during the 6^th
SOW. Issued at the beginning of the 7^th SOW, the guidelines emphasized
the important role the selected homes would play in the QIOs' contract
performance and encouraged QIOs to select homes that demonstrated a
willingness and ability to commit time and resources to quality
improvement. The QIO support contractor also encouraged QIOs to exclude
homes with a high number of survey deficiencies, high management turnover,
and QM scores that were too good to improve significantly. With respect to
homes' survey histories, the QIO support contractor reasoned that homes
with a high number of deficiencies might be more focused on improving
their survey results than on committing time and resources to quality
improvement projects. For example, the care areas in which a home was
cited for deficiencies might not correspond with any of the eight QMs to
which CMS limited the QIOs' quality improvement activities (see table 1).
In fact, the quality of care area in which homes were most frequently
cited for serious deficiencies in surveys in 2006 was the provision of
supervision and devices to prevent accidents, which does not have a
corresponding QM.^42

42Deficiencies are deemed serious if they constitute either actual harm to
residents or actual or potential for death/serious injury.

Consistent with the guidelines, 76 percent of the 41 QIOs that reported in
our Web-based survey their considerations in selecting homes for the
intensive participant group ranked homes' commitment as their primary
consideration. Nearly all QIOs ranked commitment among their top three
considerations (see fig. 4).^43

43Although many QIOs excluded some interested homes from the official list
of intensive participants submitted as a contract deliverable, most QIOs
(75 percent) reported that they gave these homes more assistance than they
did other homes in the state, and 37 percent reported that they gave these
homes as much assistance as they gave intensive participants.

Figure 4: QIOs' Considerations in Choosing among Homes That Volunteered
for Intensive Assistance in the 7^th SOW

Note: Forty-one QIOs reported their considerations in choosing among homes
that volunteered for intensive assistance.

Homes' QM scores were also an important consideration for QIOs. QIOs were
particularly interested in including homes that had poor QM scores in
areas where the QIO planned to focus or in assembling a group of homes
that represented a mix of QM scores. With respect to homes' overall QM
scores, the QIOs that responded to our survey were more likely to seek
homes with moderate overall scores than homes with poor or good overall
scores. Similarly, personnel at most QIOs we contacted gave serious
consideration to homes' QM scores, looking for homes that appeared to need
help and could demonstrate improvement. For example, personnel at one QIO
said that they tended to select homes whose QM scores were worse than the
statewide average; personnel at another QIO said that this QIO selected
homes with scores it thought could be improved, eliminating homes with
either very high or very low scores. Personnel at one QIO acknowledged
that some QIOs might "cherry pick" homes in this way in order to satisfy
CMS contract requirements but argued that it was not possible for QIOs to
predict which homes would improve the most.

QIOs generally gave less consideration to the number of deficiencies homes
had on state surveys than to their QM scores. However, the 17 QIOs that
ranked survey deficiencies among their top three considerations in our
survey were more likely to seek homes with deficiencies in areas where
they planned to focus or homes with an overall low level (number and
severity) of survey deficiencies than homes with an overall high level.
Moreover, of the 33 QIOs that reported in our survey systematically
excluding some of the homes that volunteered from the intensive
participant group, nearly one-quarter (8) excluded homes with a high
number of survey deficiencies. None excluded homes with a low number of
survey deficiencies.^44

Personnel at the QIOs we interviewed offered several reasons for excluding
homes with a high number of survey deficiencies from the intensive
participant group. Personnel at several QIOs concurred with the QIO
support contractor that such homes were likely to be too consumed with
correcting survey issues to focus on quality improvement. Personnel at one
QIO suggested that the kind of assistance very poor-performing homes
need*help improving the basic underlying structures of operation*was not
the kind the QIO offered. Personnel at some QIOs said they considered not
just the level of deficiencies for which homes were cited on recent
surveys but the level over multiple years or the specific categories of
deficiencies. For example, personnel at one QIO said that although the QIO
excluded homes with long-standing histories of poor performance, it
actively recruited homes that had performed poorly only on recent surveys.
Personnel at another QIO stated that their concern was to avoid homes with
competing priorities. This QIO sought to include homes with deficiencies
in the areas it planned to address but to exclude homes with deficiencies
in other areas on the assumption that these homes would not benefit from
the assistance it planned to offer. Personnel we interviewed at two QIOs
said that they worked with some extremely poor-performing homes but did
not include them on the official list of intensive participants submitted
to CMS; personnel at one of these QIOs explained that they did not want to
be held responsible if these homes were unable to improve.

^44Some QIOs also considered financial status and management stability in
making their selections. Among the 51 QIOs surveyed, 8 excluded homes that
were struggling financially and 5 excluded homes with recent management
turnover. Personnel at one of the QIOs we interviewed explained that the
QIO excluded homes with known leadership instability in order to avoid
having to perform a great deal of training and retraining as
administrators came and went.

QIOs Did Not Target Intensive Assistance to Low-Performing Homes

Our analysis of homes' state survey histories from 1999 through 2002
indicates that QIOs did not target intensive assistance to homes that had
performed poorly in state surveys. Nationwide, the homes in the intensive
participant group were less likely than other homes in their state to be
low-performing in terms of the number, scope, and severity of deficiencies
for which they were cited in surveys during that time frame. As
illustrated in figure 5, the intensive participant group included
proportionately more homes in the middle of the performance spectrum and
proportionately fewer at either end. Although our analysis focused on
survey deficiencies rather than QMs, this result is generally consistent
with the results of our Web-based survey concerning QIOs' use of QM scores
as selection criteria, which showed that QIOs were more likely to select
homes with moderate overall scores than homes with poor or good overall
scores and to seek a mix of performance levels among homes in the group.
However, not knowing the composition of the pool of homes that volunteered
for assistance, we cannot determine whether the composition of the
intensive participant group*in particular, the disproportionately low
number of low-performing homes in the group*was a function of which homes
volunteered, which homes the QIOs selected from among the volunteers, or a
combination of both factors.

Figure 5: Comparison of Nonintensive and Intensive Participants'
Performance on State Surveys

Note: Homes are categorized as low-, moderately, or high-performing on the
basis of the number, scope, and severity of deficiencies for which they
were cited, relative to other homes in their state, in three standard
state surveys from 1999 through 2002. All differences are statistically
significant at p-value < 0.05 level.

On a state-by-state basis, none of the QIOs targeted assistance to
low-performing homes by including proportionately more such homes in the
intensive participant group. Most QIOs (33 of 51) worked intensively with
homes that were generally representative of the range of homes in their
state in terms of performance on state surveys from 1999 through 2002. In
these states, there was no significant difference in the proportion of
high-, moderately, or low-performing homes among intensive participants
compared with nonintensive participants. However, 18 QIOs worked
intensively with a group that differed significantly from other homes in
the state: 8 of these QIOs worked with proportionately fewer
low-performing homes, 5 worked with proportionately more moderately
performing homes, and 9 worked with proportionately fewer high-performing
homes.^45

Stakeholders Often Stated QIOs Should Target Intensive Assistance to
Low-Performing Homes

Stakeholders we interviewed who expressed an opinion about the homes QIOs
should target for intensive assistance--11 of the 16 we
interviewed--almost uniformly said that the QIOs should concentrate on
low-performing homes.^46 Survey officials in one state suggested that QIOs
should use state survey data to assess homes' need for assistance because
these data are often more current than QM data. In their emphasis on
low-performing homes, stakeholders echoed the views expressed in the 2006
IOM report, which recommended that QIOs give priority for assistance to
providers, including nursing homes, that most need improvement. Other
stakeholder suggestions regarding the homes QIOs should target are listed
in table 2. Because the QIOs were required to protect the confidentiality
of QIO information about nursing homes that agreed to work with them,
stakeholders were generally not informed which homes were receiving
intensive assistance. One exception was in Iowa, where the QIO obtained
consent from the selected homes to reveal their identities.

^45These numbers do not sum to 18 because 4 of the 5 QIOs that selected
proportionately more moderately performing homes also selected
proportionately fewer low- or high-performing homes.

^46Stakeholders included officials of state survey agencies and state
nursing home trade associations.

Table 2: Examples of Other Categories of Homes Stakeholders Suggested QIOs
Should Include as Intensive Participants

Category of home            Explanation                                    
Special focus facilities    One state survey official suggested that CMS   
                               mandate that QIOs work with the low-performing 
                               homes selected by state survey agencies for    
                               the Special Focus Facility program.^a          
Homes lacking resources for Stakeholders suggested targeting small rural   
quality improvement         facilities, "stand-alone" facilities that lack 
                               the resources of corporate chains, or          
                               facilities that are struggling financially.    
High-performing homes       Several stakeholders advocated including some  
                               high-performing homes. One stakeholder group   
                               suggested that such homes could serve as       
                               models and share their approaches with homes   
                               that were struggling. Another suggested that   
                               QIOs may include homes at varying performance  
                               levels to avoid stigmatizing the intensive     
                               participants as "bad homes."                   

Source: GAO analysis.

Note: Eleven of the 16 stakeholders we interviewed expressed an opinion
about which homes the QIOs should include as intensive participants.

aSeventeen (13 percent) of the 129 facilities in the Special Focus
Facility program as of January 2005 were also among the QIOs' 2,471
intensive participants in the 7^th SOW.

Several stakeholders said that low-performing homes can improve with
assistance. However, one suggested that QIOs might have to adapt their
approach*for example, by streamlining their training*to avoid
overburdening homes that are struggling with competing demands. Another
agreed that low-performing homes can benefit from working with a QIO but
added that real improvements in the quality of care in these homes would
require attention to staffing, turnover, pay, and recognition for staff.
The results of one special study funded by CMS during the time frame of
the 7^th SOW supported stakeholders' contention that low-performing homes
can improve, although the improvements documented in these homes cannot be
definitively attributed to the QIOs.^47 In this study, known as the
Collaborative Focus Facility project, 17 QIOs worked intensively with one
to five low-performing homes identified in consultation with the state
survey agency.^48 According to a QIO assessment of the project, the
participating homes showed improvement in areas related to the assistance
provided by the QIO in terms of both the number of serious state survey
deficiencies for which they were cited and their QM scores.^49 CMS
officials pointed out that these improvements were hard-won: one-third of
the homes that were asked to participate in the Collaborative Focus
Facility project refused, and those that did participate required more
effort and resources from the QIOs to improve than did other homes
assisted by the QIOs.

^47One reason that improvements cannot be definitively attributed to the
QIOs is that homes may have benefited from other quality improvement
efforts as well.

Overall, CMS has specifically directed only a small share of QIO resources
to low-performing homes. In the current contracts (the 8^th SOW), CMS
required QIOs to provide intensive assistance to some "persistently
poor-performing homes" identified in consultation with each state survey
agency. However, the number of such homes that the QIOs must serve is
small*ranging from one to three, depending on the number of nursing homes
in the state*and accounts for less than 10 percent of the homes the QIOs
are expected to assist intensively. Less than 17 percent of the 144
persistently poor-performing homes the QIOs selected in consultation with
state survey agencies to assist in the 8^th SOW were also special focus
facilities in 2005 or 2006.

QIOs and stakeholders tended to disagree about whether participation in
the program should remain voluntary for all homes. QIO personnel we
interviewed who expressed an opinion generally supported voluntary
participation on the theory that homes that were forced to participate
would probably be less engaged and put forth only minimal effort.
Personnel at some QIOs that opposed mandatory participation suggested that
creating incentives for homes to improve their quality of care*for
example, through pay for performance*would increase homes' interest in
working with the QIO. In contrast, most of the state survey agency and
trade association officials we interviewed who expressed an opinion about
the voluntary nature of the QIO program said that some homes should be
required to work with the QIO. Officials at one state survey agency
pointed out that the low-performing homes that really need assistance
rarely seek it; these officials believed that working with the QIO should
be mandatory for low-performing homes and voluntary for moderately to
high-performing homes. Another state survey agency official recommended
that 25 to 40 percent of the homes assisted intensively be chosen from
among the lower-performing homes in the state and required to work with
the QIO.

^48In most cases, the state survey agencies and QIOs issued joint letters
of invitation to the homes, and those that agreed to work with the QIOs
signed a participation agreement that addressed issues of confidentiality
and information sharing. The state survey agencies' role was generally
limited to identifying and helping recruit homes for the project. As with
homes in the intensive participant group, there was little overlap between
homes in the Collaborative Focus Facility project and homes selected by
state survey agencies for the Special Focus Facility program. Although the
Puerto Rico QIO participated in the Collaborative Focus Facility project,
our analysis focused on QIOs in the 50 states and the District of
Columbia.

^49Over a 1-year period, the average number of survey deficiencies the
homes received in five areas (comprehensive assessment, comprehensive care
plan, pressure sore prevention/treatment, quality of care, and physical
restraints) changed little, going from 2.59 to 2.60, but the average
number of serious deficiencies they received in these areas declined from
0.93 to 0.71. The homes' QM scores for physical restraints and high- and
low-risk pressure ulcers improved an average of 31 percent (or 38 percent
when the score with the lowest improvement was dropped from the average).

QIO Contract Flexibility Resulted in Variation in Assistance Provided to
Intensive Participants

The 7^th SOW contracts allowed QIOs flexibility in the QMs they focused on
and the interventions they used, and while the majority of QIOs selected
the same QMs and most used the same interventions to assist homes
statewide, the interventions for intensive participants and staffing to
accomplish program goals varied. Most QIOs and intensive participants
worked on the chronic pain and pressure ulcer QMs, but these were not the
QMs that some intensive participants believed matched their greatest
quality-of-care challenges. To assist all homes statewide, QIOs generally
relied on conferences and the distribution of educational materials. The
top three interventions for intensive participants included on-site visits
(87 percent), followed by conferences (57 percent), and small group
meetings (48 percent). According to nursing home staff we interviewed,
turnover and experience levels of the QIO personnel that provided them
assistance affected their satisfaction with the program and the extent of
their quality improvements.

Most Quality Improvement Efforts Focused on Chronic Pain and Pressure Ulcers

Under the terms of the contracts, both QIOs and intensive participants
could select QMs to focus on, but most chose to work on two of the same
QMs.^50 While nearly all QIOs chose to work statewide on chronic pain and
pressure ulcers, they differed on their selection of additional QMs (see
fig. 6). QIO personnel we interviewed told us they based the choice of QMs
for their statewide work on input from stakeholders and nursing homes or
QM data. For example, some stakeholders told us that specific QMs selected
addressed existing long-term care challenges and were ones on which homes
in the state ranked below the national average. Personnel from two QIOs
said they selected QMs based on input from homes in their state about
which QMs the homes were interested in working on, and personnel from
several QIOs stated that they selected QMs on which their homes could
improve. Personnel from one QIO specifically mentioned that they selected
QMs related to the quality of life for nursing home residents.

^50For their statewide assistance, three-quarters of the QIOs selected
three QMs, the minimum number contractually allowed; the remainder
selected four QMs. No QIOs selected the maximum of five.

Figure 6: QMs Selected by QIOs for Statewide Interventions and QMs
Selected by Nursing Homes for Intensive Assistance, 7^th SOW

Most intensive participants worked on a subset of the QMs selected by
their QIO--chronic pain and pressure ulcers (see fig. 6). The degree to
which intensive participants knew they had a choice of QMs was unclear. Of
the 14 intensive participants we interviewed that commented on whether
they had a choice, 9 said that they did. Staff from these homes generally
reported having selected QMs related to clinical issues on which they
could improve. However, the remaining 5 homes indicated that their QIO
selected the QMs on which they received assistance. Most of these 5 homes'
staff reported that they would have preferred to work on different QMs
from the list of eight that are publicly reported on the CMS Nursing Home
Compare Web site or other clinical issues that reflect their greatest
quality-of-care challenges.

Statewide Interventions Less Variable Than Those for Intensive Participants

The terms of the QIO contract with CMS allowed QIOs to determine the kinds
of quality improvement interventions they offered to homes, and those
selected by QIOs were consistent with an approach recommended by the QIO
support contractor: QIOs generally relied most on conferences and the
distribution of educational materials to assist homes statewide and on
on-site visits to assist intensive participants. However, there was a
greater variety of interventions frequently relied on to assist intensive
participants. In general, QIOs reported that the interventions they relied
on most were also the most effective for improving the quality of resident
care.

  Statewide Assistance

Almost three-quarters of the QIOs included conferences among the two
interventions they relied on most to provide quality improvement
assistance to homes statewide (see fig. 7). These QIOs held an average of
nine conferences over the course of the 7^th SOW, typically in various
cities throughout the state to accommodate homes from different regions.
Sixty-eight percent of these QIOs reported that more than half the homes
in their state sent staff to least one conference, and 16 percent of QIOs
reported that all or nearly all homes did so. QIO personnel reported
holding conferences to educate homes on quality improvement, discuss the
relationship between MDS assessments and the QMs, and provide QM-specific
clinical information or best practices. Some conferences included
presentations by state or national experts.

Figure 7: Statewide Interventions Most Relied on by QIOs, 7^th SOW

Almost three-quarters of QIOs also ranked the distribution of educational
materials by mail, fax, or e-mail among their top two statewide
interventions. Thirty-two percent of these QIOs sent materials four or
fewer times per year, whereas 27 percent sent materials 12 or more times
per year to all or nearly all homes in the state. For the QIOs we
interviewed, these materials included newsletters, QM-specific tools or
clinical information related to the QMs, and QM data progress reports for
the home or state, overall.

Almost one-third of the QIOs (31 percent) reported that the type or
intensity of interventions they used to assist homes statewide changed
over the course of the 7^th SOW.^51 For example, two QIOs reported that
they concentrated much of their statewide efforts into the first half of
the 3-year period; one QIO specifically reported doing so in the interest
of ensuring that any improvements in QMs were reflected in its evaluation
scores, which, as specified by the contract, were calculated near the
mid-point of the contract cycle.^52 In contrast, five other QIOs reported
that they increased the intensity of their statewide work over time, in
some cases concentrating on homes whose performance was lagging.

^51The intensity of interventions varies by type of intervention (for
example, on-site versus telephone calls) and with the frequency of use.

For the 8^th SOW, CMS has focused resources on assistance to intensive
participants by eliminating expectations for improvements in QMs
statewide. However, the contracts still contain statewide elements,
including a requirement to promote QM target-setting.

  Intensive Assistance

Fifty-one percent of QIOs ranked on-site visits as their most relied on
intervention with intensive participants and 87 percent ranked it among
their top three interventions (see fig. 8).^53 Both the number of visits
and the time spent at sites varied considerably. The median number of
visits was 5 but ranged from 1 to 20.^54 Sixty-eight percent of QIOs that
included on-site visits among their top three interventions spent an
average of 1 to 2 hours at sites each time they visited, while 20 percent
spent 3 to 4 hours. QIOs that ranked on-site visits as their number one
intervention made more and longer visits to intensive participants than
did QIOs that ranked them lower. When surveyed about a typical on-site
visit, the majority of QIO respondents reported that they generally
reviewed the homes' QM data, provided education or best practices, or
both. Approximately one-third of QIOs that conducted site visits indicated
that they had discussions with the home about their systems or processes
for care, homework assignments, or quality improvement activities.^55 Some
QIOs (26 percent) reported that they conducted team-building exercises
with the staff when on site.

^52In its 2006 report on QIOs, the IOM recommended that Congress permit
extension of the contract from 3 to 5 years to allow for measurement,
refinement, and evaluation of technical assistance efforts.

^53Because the largest component of the QIOs' contract evaluation related
to the intensive participants, we asked QIOs to rank and provide detailed
information on a greater number of interventions for intensive
participants than for statewide participants.

^54The median number of times an intervention was provided is the midpoint
of all the times that an intervention was provided, as reported by QIOs.
Half the QIOs reported a number above the median and half reported a
number below.

^55Nearly all QIOs (94 percent) also reported asking intensive
participants to complete homework assignments on their own. These
assignments most frequently involved conducting self audits, comparing
existing policies and procedures with checklists provided by the QIO, and
developing new practice protocols related to selected QMs. For example,
two homes told us they were given cause-and-effect analysis exercises to
complete to identify possible causes of and solutions to a problem. Staff
from another home told us that QIO personnel asked them to conduct a mock
survey to prepare for their next standard survey.

Figure 8: Intensive Interventions Most Relied on by QIOs and Frequency of
Interventions (Range and Median Number) during the 7^th SOW

aThe median number of times an intervention was provided is the midpoint
of all the times that an intervention was provided in the 7^th SOW, as
reported by QIOs.

QIOs varied in the interventions they used in addition to on-site visits,
with conferences, small group meetings emphasizing peer-to-peer learning,
and telephone calls being the three others most commonly used. QIOs that
included conferences among their three most relied on interventions
typically held between 3 and 10 during the 7^th SOW, but as with site
visits, some variation existed. After conferences, QIOs were most likely
to rely on small group meetings and telephone calls with individual homes.
Nearly half of the QIOs ranked these two interventions among their three
most relied on, but few ranked them highest. The number of homes that
attended small group meetings varied. An average of 6 to 10 homes was most
common, but one-fifth of QIOs reported having an average of 20 or more
homes represented at each meeting. As for telephone calls, the vast
majority of QIOs (92 percent) that ranked these calls among their three
most relied on interventions called all or nearly all of their intensive
participants, typically on a monthly basis.

Our interviews with QIOs and intensive participant homes suggested that
the small group meetings they held generally followed a similar format,
while telephone calls were used for a variety of purposes. For example,
personnel from several QIOs and intensive participant homes told us that
their small group meetings generally included a formal presentation on the
QMs or related best practices, as well as a time for less formal
information sharing and peer-to-peer learning among the attendees.
Participants shared stories about their successes and challenges
conducting quality improvement. Personnel from a number of QIOs told us
they used telephone calls to follow up after visits or meetings, discuss
the homes' progress on quality improvement, and to decide on next steps.

Almost two-thirds of QIOs indicated that the type or intensity of
interventions for intensive participants varied over time. Of these QIOs,
36 percent reduced the intensity of their interventions (substituting
small group meetings or telephone calls for on-site visits), while 33
percent did the reverse (in some cases increasing the frequency of on-site
visits or substituting small group meetings for conferences to increase
participation). For example, personnel from a few QIOs told us that while
they initially relied on on-site visits to begin the quality improvement
process, they came to rely more on telephone calls or on small group
meetings where intensive participants could share their success stories or
ways to overcome barriers to quality improvement. Seventy-nine percent of
QIOs surveyed varied their interventions based on the needs of intensive
participants. Thus, personnel from three QIOs told us they realized that
some homes did not need frequent on-site visits, while others needed more.
The two specific needs that QIOs cited most as having precipitated changes
in their interventions were nursing home staffing changes and turnover (23
percent) and poorer performance by some homes relative to others (15
percent). A few QIOs also noted that interventions varied by the
preferences or levels of readiness and participation of the homes with
which they were working.

  QIO and Nursing Home Perspectives on the Interventions

Most QIOs we surveyed deemed conferences the most effective statewide
intervention and on-site visits the most effective intensive intervention;
intensive participant homes we interviewed also found these interventions
valuable. For homes statewide, most QIOs (54 percent) reported that
conferences were their most effective intervention, followed by
distribution of educational materials and on-site visits. Of the
one-quarter of QIOs that reported they would change their statewide
approach, the largest proportion (46 percent) would make conferences their
primary intervention. Staff from several nursing homes we interviewed
tended to concur that conferences were valuable aspects of the program
because conferences included expert presenters, energized or motivated
attendees, and were free.

For intensive participants, most QIOs (63 percent) deemed on-site visits
their most effective intervention, followed by conferences and small group
meetings. Of the 15 QIOs that said they would change their approach with
these homes, most (60 percent) would make on-site visits their primary
intervention, while fewer would rely on small group meetings, conferences,
and other interventions. One QIO began conducting on-site visits and small
group meetings when it became apparent that telephone calls were less
productive than had been anticipated because of the difficulty of getting
the right staff on the telephone at the right time, the lack of speaker
phones at many homes, and the lack of staff engagement on the phone. Staff
from a number of nursing homes we interviewed agreed that visits by QIO
personnel were helpful. Some homes indicated that having someone from the
QIO visit the home maximized the number of staff that could take advantage
of the quality improvement training offered. Furthermore, the on-site
visits were motivating and kept staff on track with quality improvement
efforts. Regarding small group meetings, staff we interviewed from a few
homes stated that meeting with staff from other homes helped validate
their own efforts or facilitated the sharing of materials and experiences.
Staff from one nursing home specifically reported that they were
disappointed not to have formally participated in small group meetings
with other facilities in the state.

Homes also found particular types of assistance less helpful. Some homes'
staff reported that they did not feel they had the time or the staff
necessary to complete some of the homework assignments expected of them,
such as conducting chart reviews. Staff at some homes stated that the QIO
provided quality improvement information with which they were already
familiar.

QIO Staffing and Turnover Influenced Intensive Participants' Satisfaction with
Program

Our interviews with nursing home staff who worked intensively with the
QIOs indicated that homes' satisfaction with the program was influenced by
the training and experience of the primary QIO personnel who served as
their principal contact with the QIOs, as well as by turnover among these
individuals during the course of the 7^th SOW.^56

When a home's principal contact with the QIO was a nurse or someone with
long-term care or quality improvement experience, nursing home staff
tended to report that this person possessed the knowledge and skills
necessary to help them improve the quality of care in their home.
Interviewees also spoke appreciatively of QIO personnel who were
knowledgeable, motivating, and kept them on track with their efforts.
However, when the QIO principal contact lacked these qualifications or
characteristics, he or she was perceived as unable to effectively address
clinical topics with staff. Staff at one home said explicitly that working
with an experienced nurse, instead of a social worker who seemed to lack
knowledge of long-term care, would have led to greater improvement in
clinical quality.

The extent to which QIO primary personnel had the training or experience
that homes considered important varied. More than half (58 percent) of the
primary QIO personnel who worked with nursing homes during the 7^th SOW
were trained in nursing, and 42 percent held an advanced degree.
Nationwide, 27 percent of the primary personnel who worked with nursing
homes had less than 1 year of long-term care experience, while 30 percent
had more than 10 years of such experience.^57 Just over half of primary
QIO personnel (54 percent) working with nursing homes had 4 or fewer years
of quality improvement experience. Nine percent of QIO personnel had more
than 10 years' experience in both long-term care and quality improvement.
Few of the personnel working with nursing homes during the 7^th SOW gained
any of their experience working for the QIO during the 6^th SOW because
there was little overlap in personnel across the two periods.

^56We defined primary personnel as individuals who devoted more than 20
percent of a full-time work week to the nursing home component of the
contract. Some primary QIO personnel served as the principal contacts,
providing quality improvement assistance to homes. According to our
survey, 78 percent of QIOs also used outside experts (consultants or
subcontractors) for their quality improvement efforts. The majority of
QIOs reported using these experts to provide presentations or training at
conferences, participate in conference calls, and develop or review
materials. QIOs personnel we interviewed told us they also used outside
experts to train their primary personnel or to provide technical
assistance to intensive participant homes.

^57Among individual QIOs, the extent of long-term care experience spanned
a wide spectrum. At five QIOs, 75 percent or more of the primary personnel
who worked with nursing homes had less than 1 year of long-term care
experience, while at two QIOs, all of the primary personnel who worked
with nursing homes had more than 10 years' experience.

Our interviews with intensive participants suggested that turnover among
primary QIO personnel lowered nursing homes' satisfaction with the
program. Our survey revealed that turnover was particularly high at some
QIOs. At 24 QIOs, 25 percent or more of primary personnel who worked with
nursing homes did so for less than half of the 36-month contract, and at 6
QIOs, the proportion was 50 percent or more. When a nursing home's
principal contact with a QIO changed frequently, nursing home staff we
interviewed reported that they received inconsistent assistance that was
disruptive to their efforts to improve quality of care. For example, one
nursing home we visited had four different principal contacts over the
course of the 7^th SOW and found this to be frustrating because, just as
they were establishing a relationship with a contact, the contact would
leave. Staff at another home complained that their interaction with QIO
primary personnel turned out not to be the learning experience that the
staff thought it would be.

Staffing levels for the nursing home component also varied among QIOs. As
would be expected, given the wide variation in the number of nursing homes
per state, the number of full-time-equivalent (FTE) staff working with
nursing homes varied across the QIOs, ranging from 0.50 to 12. However,
the ratio of QIO staff FTEs to intensive participant homes also showed
significant variation. On average, the ratio was about 1 to 14; but for at
least 9 QIOs, the ratio of staff FTEs to homes was 1 to 10 or fewer, and
for at least 8 QIOs, the ratio was 1 to 18 or more.

QIOs' Impact on Quality Is Not Clear, but Staff at Homes We Contacted Attributed
Some Improvements to QIOs

Although the QIOs' impact on the quality of nursing home care cannot be
determined from available data, staff we interviewed at most nursing homes
attributed some improvements in the quality of resident care to their work
with the QIOs. Nursing homes' QM scores generally improved enough for the
QIOs to surpass by a wide margin the modest contract performance targets
set by CMS; however, the overall impact of the QIOs on the quality of
nursing home care cannot be determined from these data because of the
shortcomings of the QMs as measures of nursing home quality and because
confounding factors make it difficult to attribute quality improvements
solely to the QIOs. Multiple long-term care professionals we interviewed
indicated that QMs should not be used in isolation to measure quality
improvement, but combined with other indicators, such as state survey
data. Moreover, the effectiveness of the individual interventions QIOs
used to assist homes also cannot be evaluated with the available data. CMS
planned to enhance evaluation of the program during the 8^th SOW, but a
2005 determination by HHS's Office of General Counsel that the QIO program
regulations prohibit QIOs from providing to CMS the identities of the
homes they are assisting has hampered the agency's efforts to collect the
necessary data. Although the impact of the QIOs on the quality of nursing
home care is not known, over two-thirds of the 32 nursing homes we
interviewed attributed some improvements in care to their work with the
QIOs.

All QIOs Met Modest Targets for QM Improvement, but the Impact of the QIOs on
the Quality of Nursing Home Care Cannot Be Determined

Although all of the QIOs met the modest targets CMS set for QM improvement
among homes both statewide and in the intensive participant group, the
impact of the QIOs on the quality of nursing home care cannot be
determined because of the limitations of the QMs and because improvements
cannot be definitively attributed to the QIOs. The effectiveness of the
specific interventions used by the QIOs to assist homes also cannot be
evaluated with the available data.

  All QIOs Met CMS's Modest Targets for Improvement in Nursing Home QMs

All QIOs met the CMS performance targets for the nursing home component of
the 7^th SOW. In addition to receiving an overall passing score for this
component, nearly all QIOs surpassed expectations for each of the three
elements that contributed to the overall score: provider satisfaction,
improvement in QM scores among intensive participants, and improvement in
QM scores among homes statewide. In fact, about two-thirds of the QIOs
achieved at least five times the expected 8 percent improvement among
intensive participants, and nearly half achieved at least twice the
expected 8 percent improvement statewide.^58

CMS officials stated that the targets set for the nursing home component
of the contract were purposely modest. Because the 7^th SOW marked the
first time all QIOs were required to work with nursing homes on quality
improvement, and little data existed to predict how much improvement could
be expected, CMS deliberately designed performance criteria to limit QIOs'
chances of failing. For example, expectations for improvements in QM
scores were set no higher for intensive participants than for homes
statewide. In addition, CMS modified the evaluation plan so that if an
intensive participant worked on more than one QM, the QM that improved
least was dropped before the home's average improvement was calculated.
CMS officials told us that, based in large part on QIOs' performance in
the 7^th SOW, the agency raised its expectations for the 8^th SOW. For
example, QIOs are required to work with most intensive participants on
four specified QMs and to achieve an improvement rate of 15 to 60 percent,
depending on the QM and the homes' baseline scores. In addition, CMS will
no longer drop the QM that improved least when calculating homes' average
improvement.^59

58The improvement, or relative change, in a home's QM scores is calculated
by subtracting its score at remeasurement from its score at baseline and
dividing by its score at baseline. For example, if the number of residents
with chronic pain in a 100-bed home decreased from 20 to 12*which
translates to a change in scores from 0.20 to 0.12*the improvement in the
home's pain QM would be 40 percent ([0.20-0.12]/0.20).

  CMS's Use of QMs to Evaluate QIO Performance Is Problematic

Long-term care experts we interviewed generally agreed that CMS's use of
QMs to evaluate nursing home quality--and by extension, QIOs'
performance*is problematic because of unresolved issues related to the QMs
and the MDS data used to calculate them.

QMs. As we reported in 2002, the validity of the QMs CMS proposed to
publicly report in November 2002 was unclear.^60 Although the validation
study commissioned by CMS found that most of the publicly reported QMs
were valid and reflected the quality of care delivered by facilities,
long-term care experts have criticized the study on several grounds. For
example, a 2005 report concluded that (1) the statistical criteria for the
validity assessments were not stringent and (2) the researchers did not
attempt to determine whether QMs were associated with quality of care at
the resident level.^61 As a result, it is not clear whether a resident who
triggers a QM (e.g., is assessed as having his or her pain managed
inadequately) is actually receiving poor care.^62 The lack of correlation
among the QMs*a home may score well on some QMs and poorly on others*also
calls into question their validity as measures of overall quality. Since
2002, CMS has removed or replaced 5 of the original 10 QMs*including some
of those on which the QIOs were evaluated during the 7^th SOW--to address
limitations in the QMs, such as reliability and measurement problems. (See
app. II for a list of the QMs as of November 2002 and February 2007).

^59The four QMs specified in the contract are pressure ulcers among
high-risk patients, restraints, depression management, and chronic pain
management. With most intensive participants, QIOs are expected to work on
all four QMs and achieve a relative improvement rate of 15 to 60 percent.
With the small group of persistently poor-performing homes QIOs are now
required to assist, they are expected to work on two QMs (pressure ulcers
among high-risk patients and restraints) and achieve an improvement rate
of 10 percent.

^60 [52]GAO-03-187 .

^61Greg Arling and others, "Future Development of Nursing Home Quality
Indicators," The Gerontologist, vol. 45, no. 2 (2005).

Risk adjustment also impacts the validity of QMs. There is general
recognition that some QMs should be adjusted to account for the
characteristics of residents. However, there is disagreement about which
QMs to adjust, what risk factors should be used, or how the adjustment
should be made. For example, one expert we interviewed suggested that in
many cases pressure ulcers start in hospitals; the pressure ulcer QM does
not account for the origin of ulcers. Another expert highlighted the
difficulty of making an appropriate adjustment--noting, for example, that
improperly risk-adjusting the pressure ulcer QM could mask poor care that
contributed to the development of ulcers.

MDS. We have also previously reported concerns about MDS reliability*that
is, the consistency with which homes conduct and code the assessments used
to calculate the QMs.^63 CMS awarded a contract for an MDS accuracy review
program in 2001 but revamped the program in 2005, near the end of the
QIOs' 7^th SOW, acknowledging weaknesses--mainly its reliance on off-site,
rather than on-site, accuracy and verification reviews--that we had
previously identified.^64 Some states that sponsor on-site MDS accuracy
reviews continue to report troubling rates of errors in the data. For
example, officials of Iowa's program reported an average MDS error rate of
approximately 24 percent in 2005.

Our interviews with long-term care experts and nursing home staff
suggested that the chronic pain QM*which was selected as a focus of
quality improvement work by many QIOs and intensive participant nursing
homes*may be particularly vulnerable to error in the underlying MDS data.
Possible sources of error are systematic differences in the extent to
which facilities identify and assess residents in pain and
misunderstandings about how to accurately code MDS questions specific to
pain. For example, staff from two nursing homes told us that their pain
management QM scores improved after staff realized that they had been
mistakenly coding residents as having pain even though their pain was
successfully managed. Moreover, experts we interviewed noted that
higher-quality homes may have worse pain QM scores because they do a
better job of identifying and reporting pain in residents.

^62A resident who triggers a QM is included in both the numerator and
denominator when a facility's QM score is calculated.

^63 [53]GAO-02-279 and [54]GAO-03-187 .

^64In April 2005, CMS ended work under its data assessment and
verification contract but signed a new contract in September 2005 that
focused on on-site reviews of MDS accuracy.

The use of MDS data to measure the quality of care in nursing homes is
also problematic because the MDS was not designed as a quality measurement
tool and does not reflect advances in clinical practice. CMS is updating
the MDS now to address these limitations. For example, instead of asking
homes to classify the severity of a pressure ulcer on the basis of a
four-stage system, the draft MDS now under review includes a measurement
tool intended to more accurately classify the severity of a pressure
ulcer.^65 In addition, facilities are asked to indicate whether the
pressure ulcer developed at the facility or during a hospitalization. CMS
does not yet have an official release date for the revised MDS but
anticipates that all validation and reliability testing will be completed
by December 2007.

Other Measures of Quality. Multiple long-term care professionals we
interviewed, including stakeholders and experts on quality measurement,
recommended both that the QMs undergo continued refinement and that they
not be used in isolation to assess the quality of care in nursing homes.
They suggested a number of other sources of information as alternatives or
complements to QMs for measuring quality. For example, a representative of
the National Quality Forum (NQF), a group with which CMS contracted to
provide recommendations on quality measures for public reporting, stated
that experts do not consider the QMs sufficient in themselves to rate
homes and that the other quality markers*such as perceptions of care by
family members, residents, and staff; state survey data; and resident
complaints*also provide useful information about quality of care. Other
long-term care professionals we interviewed suggested these and other
measures, including nursing home staffing levels and staff turnover and
retention rates.

^65Stages of pressure ulcer formation are I--skin of involved area is
reddened, II--upper layer of skin is involved and blistered or abraded,
III--skin has an open sore and involves all layers of skin down to
underlying connective tissue, IV--tissue surrounding the sore has died,
exposing muscle and bone.

  Influence of Other Factors on Nursing Home Quality Makes It Difficult to
  Evaluate QIOs' Impact

Factors such as the existence of other quality improvement efforts make it
difficult to evaluate QIOs' work with nursing homes and attribute quality
improvement solely to QIOs. In an assessment of the QIO program during the
7^th SOW, CMS and QIO officials acknowledged this difficulty. The
assessment found that intensive participants improved more than
nonintensive participants on all five QMs studied, and for each QM,
intensive participants that worked on the QM improved more than intensive
participants that did not.^66 However, the authors noted that these
results could not be definitively attributed to the efforts of the QIOs
because improvements may have been influenced by a variety of factors,
including preexisting differences between intensive participants and
nonintensive participants;^67 public reporting of the QMs, which may have
focused homes' attention on improving these measures; and other quality
improvement efforts to which homes may have been exposed. As noted earlier
in this report, these other efforts included, but were not necessarily
limited to, initiatives sponsored by state governments, nursing home trade
associations, and CMS. While these other efforts varied considerably in
the intensity of technical assistance offered*ranging from a trade
association-sponsored program that homes characterized as essentially
signing a quality improvement pledge, to state-sponsored programs that
involved on-site visits by experienced long-term care nurses who provided
best-practice guidelines, educational materials, and clinical tools*the
fact that the efforts were present made it impossible to attribute quality
improvements solely to the QIOs.

In its 2006 report on all aspects of the QIO program, IOM highlighted
similar shortcomings in previous studies of the QIO program and called for
more systematic and rigorous evaluations. IOM concluded that although the
QIOs may have contributed to improvements in the quality of care, the
existing evidence was inadequate to determine the extent of their
contribution. In its response to the IOM study, CMS acknowledged the need
to strengthen its methods of evaluating the program and outlined plans to
convene an evaluation expert advisory panel to make recommendations on the
framework for the next contracts (the 9^th SOW, which will begin in 2008).
CMS also stated that it will collect information during the 8^th SOW that
will allow it to control for differences in motivation between intensive
and nonintensive participants but did not specify the nature of this
information.^68 Subsequently, HHS's Office of General Counsel determined
that QIO program regulations prohibited QIOs from providing to CMS the
identities of intensive participants.^69 CMS officials acknowledged that
this prohibition posed a considerable challenge to their evaluation plans
and said that as a short-term solution the agency might contract with one
of the QIOs to evaluate the program, with the possible stipulation that
the findings be verified by an independent auditor.

^66William Rollow and others, "Assessment of the Medicare Quality
Improvement Organization Program," Annals of Internal Medicine, vol. 145,
no. 5 (2006).

^67Because homes must volunteer and be selected by the QIOs to receive
intensive assistance, intensive participants may differ from nonintensive
participants in ways that affect their capacity to improve their QM
scores, such as differences in motivation and commitment, available
resources, and competing priorities.

  CMS Data Are Too Limited to Evaluate Effectiveness of Specific QIO
  Interventions

CMS collected little information about the specific interventions QIOs
used to assist nursing homes and acknowledged that the information it did
have was not sufficiently comprehensive or consistent to be used to
evaluate the interventions' effectiveness. In general, CMS's oversight of
QIOs' work on the nursing home component consisted of ensuring that the
QIOs produced the reports and deliverables specified in the contracts and
appeared on track to meet performance targets.

CMS's primary source of data about QIOs' interventions was the monthly
activity reports the QIOs were required to submit through the Program
Activity Reporting Tool (PARTner). In these reports, QIOs were to document
the specific interventions they provided to each home, using such activity
codes as "on-site support" and "stand-alone workshops on quality
improvement." However, with only seven activity codes for QIOs to choose
from, the level of detail in these reports was low. For example, the same
code would be used for a full-day visit as for an hour visit. Moreover,
because QIOs were not expected to enter any code more than once per month
for a home, a code for on-site support could indicate a single visit or
multiple visits. The system also captured no information about the content
of visits or other interventions. From the perspective of the QIOs, the
system was of limited use: More than half of the 52 QIOs surveyed by IOM
rated PARTner fair or poor in terms of both value and ease of use. Staff
at one QIO we interviewed reported using tracking systems they developed
themselves, rather than PARTner, to monitor their work.

^68At a meeting on October 31, 2006, of the Technical Expert Panel
convened by the contractor tasked to design an evaluation of the QIO
program for the Office of the Assistant Secretary for Planning and
Evaluation of HHS, panel members underscored the difficulty of controlling
for a subjective condition such as motivation to improve the quality of
care and noted the potential for biased assessments of the impact of the
QIOs if differences in motivation are not accounted for appropriately.

^69According to CMS guidance, the names of participants in collaborative
quality improvement projects constitute quality review study information.
See QIO Manual, S16005 (Rev. 07-11-03). Federal regulations specify that
quality review study information revealing the identities of practitioners
and institutions must be disclosed to CMS "on site" or at the QIOs' place
of operation. See 42 C.F.R. S480.140 (2005). That restriction does not
apply to disclosures to certain other federal agencies, such as HHS Office
of Inspector General or GAO. See 42 C.F.R. S480.140(b)(2005).

CMS regional offices and the nursing home satisfaction survey gathered
some additional information about the interventions used by QIOs. The CMS
regional offices gathered information through telephone calls and visits
to the QIOs and by participating in quarterly conference calls during
which QIOs and CMS regional and central offices discussed issues related
to the nursing home component of the contract. The regional office staff
also reviewed information entered into the PARTner data system by QIOs,
but they focused their evaluations on QIO contract compliance and not on
the effectiveness of specific interventions because--as some regional
staff emphasized--the contracts were performance-based, and therefore it
was not their place to "micromanage" the QIOs or to advocate for or
against specific interventions. Feedback from nursing homes was gathered
through the nursing home satisfaction survey, conducted after the midpoint
of the contract cycle by a contractor for CMS.^70 The survey collected
information about the frequency of, and homes' satisfaction with, a range
of interventions, including on-site visits, training workshops, one-on-one
telephone calls, conference calls, one-to-one e-mails, and broadcast
e-mails. However, the survey collected no information about the content of
these interventions or the aspects that contributed to providers'
satisfaction or dissatisfaction.

In its 2006 report on the QIO program, IOM emphasized the need for CMS to
gather more information about specific interventions and noted that CMS
was uniquely positioned to determine which interventions lead to high
levels of quality improvement. The agency responded that it will collect
information during the 8^th SOW to better explore the relationship between
the intensity of assistance provided by the QIO and the level of
improvement, but did not specify the type of information it will collect.
As of March 2007, CMS had not yet implemented a revamped PARTner system.
In addition, the agency cancelled its plans to conduct an initial survey
of nursing homes early in the contract period and now plans to conduct
only one, later in the contract period. CMS officials explained that the
delay and cancellation were due at least in part to the determination by
HHS's Office of General Counsel that QIOs could not provide to CMS the
identities of intensive participants to CMS.

^70For the survey conducted during the 7th SOW, the response rate for
nursing homes was 95 percent.

Homes That Received Intensive Assistance Generally Attributed Some Improvements
in Quality of Care to Work with QIOs

Although the impact of the QIOs on the overall quality of nursing home
care cannot be determined, staff we interviewed at over two-thirds of the
32 nursing homes stated that they improved the care delivered to residents
as a result of working intensively with the QIOs. Staff at 23 of the 32
homes told us that they implemented new, or made changes to existing,
policies and procedures related to pain or pressure ulcers. Of the 23
nursing homes, staff from 21 stated that they changed the way they
addressed resident pain. In general, these changes involved implementing
pain scales or new assessment forms. Staff at some facilities noted that
working with the QIO heightened staff awareness of resident pain,
including awareness of cultural differences in the expression of pain.
Staff at 8 of the 23 nursing homes stated that they changed the way they
addressed pressure ulcers. In general, these 8 homes implemented new
assessment tools, changed assessment plans, or revised facility policies
using materials provided by the QIO. (Table 3 provides examples of
resident care improvements related to pain assessment and treatment and
pressure ulcers.) Staff at 13 of the 32 nursing homes stated that the
changes they made as a result of working with the QIOs were sustainable,
but staff from some nursing homes noted that staffing turnover at their
facilities could affect sustainability.

Table 3: Examples of Resident Care Improvements Made by Homes as a Result
of Intensive Assistance Provided by QIOs, 7^th SOW

Care area       Example                                                    
Pain            Had nurses evaluate acute pain management at end of each   
                   shift with nurse aide involvement                          
                   Used interventions other than medications, such as         
                   massage, compresses, and repositioning                     
                   Recorded signs of pain when providing care for wounds such 
                   as pressure ulcers                                         
                   Began using or resumed using pain scales to assess         
                   resident pain                                              
                   Implemented pain policy that addresses both cognitively    
                   intact residents and residents who have dementia or are    
                   nonverbal                                                  
Pressure ulcers Increased skin assessments to four times a week and had    
                   nurse aides document changes on a daily basis              
                   Established a wound care team                              
                   Used a tracking tool to measure depth and width of         
                   pressure ulcers                                            
                   Conducted skin checks when a resident returned to the      
                   facility, such as after a hospitalization                  

Source: GAO interviews with staff from nursing homes assisted intensively
by the QIOs.

Of the 32 nursing homes we contacted, staff from 4 specifically stated
that working with the QIO did not change their quality of care. For
example, staff from one home stated that the QIO did not offer their
facility any new or helpful information and did not offer feedback on how
the facility's processes could improve. Staff from another home reported
that the information provided by the QIO was on techniques their facility
had already implemented. Staff at a third home noted that while the QIO
was a good resource, the home could have done as much on its own, without
assistance from the QIO. Staff at three facilities, none of which reported
making any policy or procedural changes, said the facilities experienced
worse survey results while working with their QIO; staff from two of the
three reported being cited for quality deficiencies in the specific areas
they had been addressing with the QIO. Staff at one of these facilities
believed they were cited because their work with the QIO had made the
surveyor more aware of the facility's problems in this area.

Conclusions

Although it is difficult to evaluate the impact of QIO assistance, the QIO
program does have the potential to help improve the quality of nursing
home care. CMS program improvements for the 8^th SOW, such as the agency's
decision to focus resources on intensive rather than statewide assistance
and its plans to improve evaluation, are positive steps that could result
in more effective use of available funds and provide more insight into the
program's impact. Our evaluation of assistance provided during the 7^th
SOW, however, raised two major questions about the future focus,
oversight, and evaluation of the QIO program, which we address below.

Given the available resources, which homes and quality-of-care areas
should CMS direct QIOs to target for intensive assistance? We found that
QIOs generally did not target intensive assistance to homes that performed
poorly in state surveys, partly because of concerns about the willingness
and ability of such homes to simultaneously focus on quality improvement
and cooperate with the QIOs. However, the Collaborative Focus Facility
project during the 7^th SOW demonstrated that low-performing homes could
improve their survey results and QM scores; subsequently, CMS required
that during the 8^th SOW each QIO work with up to three such homes--about
10 percent of the total number that QIOs are expected to assist
intensively. Stakeholders we interviewed believed that even more emphasis
should be placed on assisting low-performing homes. We found that there
was little overlap between homes that participated in the QIO
Collaborative Focus Facility project and in CMS's Special Focus Facility
program, which is a program involving about 130 nursing homes nationwide
that, on the basis of their survey results, receive increased scrutiny and
enforcement by state survey agencies. The limited overlap suggests that
each state has more than three low-performing facilities that could
benefit from QIO assistance.

Targeting assistance to low-performing homes could pose challenges given
the voluntary nature of the program--homes must agree to work with a QIO.
QIOs maintain that voluntary participation is critical to ensuring homes'
commitment to the program. However, the risk in this approach is that some
of the homes that need help most will not get it. Indeed, in the
Collaborative Focus Facility project, some of the low-performing homes
that were asked to participate refused QIO assistance. In addition, QIOs
expended more resources working to improve these low-performing homes than
were required to assist better-performing homes. Thus, increasing the
number of low-performing homes QIOs are required to assist above the small
number mandated for the 8^th SOW might necessitate decreasing the total
number of homes assisted. However, existing resources might be maximized
if QIOs worked with each home only on the quality-of-care areas that pose
particular challenges for that home.

Could interim steps be taken to improve oversight and evaluation of QIOs'
work with nursing homes before the contracting cycle that begins in August
2008? Currently, CMS collects data primarily on QIO
outcomes--specifically, changes in QM scores--and costs. CMS needs more
detailed data, particularly about the type and intensity of interventions
used to assist nursing homes, to improve its oversight and evaluation of
the QIO program. Without such data, CMS cannot hold QIOs fully accountable
for their performance under their contract with CMS. Some evaluation
activities are now scaled back or on hold because HHS determined early in
the 8^th SOW that program regulations prohibited the QIOs from providing
to CMS the identities of the intensive participants. Such a firewall
presents a major impediment to improved oversight and evaluation of the
QIO program and prevented CMS from implementing interim changes it planned
to make. For example, for the 7^th SOW, CMS contracted for one nursing
home satisfaction survey to be conducted near the end of the contract
period--too late to be of use in interim monitoring of the QIOs'
performance. For the 8^th SOW, CMS had planned to contract for two surveys
but was forced to cancel the one planned for early in the contract period
because it was unable to provide the names of intensive participants to
its survey contractor. Moreover, the lack of these data would preclude CMS
from independently verifying QIO compliance with such contract
requirements as the geographic dispersion of intensive participants in
each state.

CMS evaluated QIOs' work with nursing homes primarily on the basis of
changes in QM scores; given the weaknesses of QM data, the current
reliance on these data appears unwarranted. While CMS actions to improve
the MDS instrument as a quality measurement tool are important, the agency
has not yet established an implementation date. Although multiple
long-term care professionals believe that multiple indicators of quality,
including deficiencies on homes' standard and complaint surveys and
residents' and family members' satisfaction with care, should be used to
measure quality improvement, CMS is not currently drawing on these data
sources to evaluate QIOs' efforts. Recognized shortcomings in these other
data sources--such as the understatement of survey deficiencies by state
surveyors--underscore the importance of using multiple data sources to
evaluate QIO outcomes.

Recommendations for Executive Action

To ensure that available resources are better targeted to the nursing
homes and quality-of-care areas most in need of improvement, we recommend
that the Administrator of CMS take the following two actions:

           o Further increase the number of low-performing homes that QIOs
           assist intensively.

           o Direct QIOs to focus intensive assistance on those
           quality-of-care areas on which homes most need improvement.

           To improve monitoring of QIO assistance to nursing homes and to
           overcome limitations of the QMs as an evaluation tool, we
           recommend that the Administrator of CMS take the following three
           actions:

           o Revise the QIO program regulations to require QIOs to provide to
           CMS the identities of the nursing homes they are assisting in
           order to facilitate evaluation.

           o Collect more complete and detailed data on the interventions
           QIOs are using to assist homes.

           o Identify a broader spectrum of measures than QMs to evaluate
           changes in nursing home quality.
			  
			  Agency Comments and Our Evaluation

           We obtained written comments from CMS on our draft report. CMS
           addressed three of our five recommendations. It concurred with two
           of the three recommendations but did not specify how it would
           implement them, and it continues to explore options for
           implementing the third recommendation. Our evaluation of CMS's
           comments follows the order we presented each recommendation in the
           report. CMS's comments are included in app. III.

           Further increase the number of low-performing homes that QIOs
           assist intensively. CMS agreed with this recommendation but did
           not specify a time frame for addressing it or indicate how many
           low-performing homes it will expect QIOs to assist in the future.
           Although our report focused on the most recently completed
           contract period (the 7^th SOW), we acknowledged that in the
           current contract period, CMS required QIOs to provide intensive
           assistance to some "persistently poor-performing" homes identified
           in consultation with each state survey agency. However, we pointed
           out that the number of these homes the QIOs were required to serve
           was small, accounting for less than 10 percent of the homes they
           were expected to assist intensively. CMS commented that
           preliminary estimates from a special study conducted during the
           7^th SOW indicated that assisting chronically poor-performing
           homes cost the QIOs 5 to 10 times as much as assisting the "usual"
           home.^71 Our report acknowledged that additional resources were
           required for QIOs to assist low-performing homes but suggested
           that CMS could decrease the total number of homes assisted in
           order to increase the number of low- performing homes beyond the
           small number mandated for the 8^th SOW.

           Direct QIOs to focus intensive assistance on those quality-of-care
           areas on which homes most need improvement. CMS did not directly
           respond to this recommendation, but did point out that about
           one-third of QIOs were working primarily with homes on QMs on
           which the homes scored worse than the national average during the
           8^th SOW. Our recommendation was to direct all QIOs to focus
           intensive assistance on QMs that reflect homes' greatest
           quality-of-care challenges. We had reported that some nursing
           homes assisted intensively by QIOs did not have a choice of QMs on
           which to work. We concluded that having QIOs work intensively with
           homes only on the quality-of-care issues that posed particular
           challenges to them would maximize program resources.

           Revise QIO program regulations to require QIOs to provide CMS with
           the identities of the homes assisted in order to facilitate
           evaluation. CMS did not specifically indicate whether it agreed
           with this recommendation, but did indicate that it continues to
           explore options which would allow access to data on the homes
           assisted intensively in order to facilitate evaluation. However,
           CMS expressed concern that providing this access could potentially
           subject the information to laws that could afford third parties
           similar access. We believe that CMS should continue to evaluate
           how best to maintain an appropriate balance between disclosure and
           confidentiality. If CMS's evaluation indicates that it is unable
           to incorporate adequate confidentiality safeguards to promote
           voluntary participation in QIOs' quality improvement initiatives,
           the agency could seek legislation that would provide such
           safeguards.
			  
^71CMS did not provide this cost estimate during the course of our work.

           Collect more complete and detailed data on the interventions QIOs
           use to assist homes. CMS responded to this recommendation,
           although it labeled it "improve the monitoring of QIO activities,"
           and agreed with our recommendation. CMS noted that, in concert
           with HHS, it is reviewing recommendations from the IOM's 2006
           report on QIOs, which may result in redesigning the program,
           including systems for evaluating QIO activities in different care
           settings, such as nursing homes. CMS did not discuss how it
           planned to collect additional data on QIO nursing home
           interventions. Further, it stated that it may incorporate
           data-handling and -reporting features of the nursing home subtask
           into overall program improvements. We have reservations about this
           plan because we found that CMS collected little information about
           specific QIO interventions with nursing homes during the 7^th SOW,
           the information collected was not sufficiently comprehensive or
           consistent to be used to evaluate the interventions'
           effectiveness, and QIOs themselves reported that the data
           collection system was of limited use to them.

           Identify a broader spectrum of measures than QMs to evaluate
           changes in nursing home quality. CMS did not directly address this
           recommendation. However, the agency took issue with our judgment
           that the use of QMs to evaluate nursing home quality--and by
           extension, QIOs' performance--is problematic. CMS commented that
           the QMs have passed through rigorous development, testing,
           deployment, and national consensus processes. We reported that the
           study commissioned by CMS to validate the QMs has been criticized
           by experts on several grounds, including a lack of statistical
           rigor. We also noted that CMS has revised or is currently revising
           both the QMs and the MDS data used to calculate them to address
           limitations, such as reliability and measurement problems. For
           example, CMS has removed or replaced 5 of the original 10 QMs
           since 2002, including some of those on which the QIOs were
           evaluated during the 7^th SOW. In addition, CMS is currently
           updating the MDS to reflect advances in clinical practice and to
           improve its utility as a quality measure tool. While we expect
           that these efforts will improve the QMs as measures of nursing
           home quality, we believe that the QMs' current limitations argue
           for the use of a broader spectrum of measures to evaluate changes
           in nursing home quality. Multiple long-term care professionals we
           interviewed recommended that the QMs not be used in isolation to
           assess the quality of care in nursing homes; these professionals
           suggested a range of measures that could be used to supplement the
           QMs, including perceptions of care by family members, residents,
           and staff; state survey data; and nursing home staffing levels.

           As arranged with your office, unless you publicly announce its
           contents earlier, we plan no further distribution of this report
           until 30 days after its issue date. At that time, we will send
           copies to the Administrator of the Centers for Medicare & Medicaid
           Services and appropriate congressional committees. We will also
           make copies available to others upon request. In addition, the
           report will be available at no charge on the GAO Web site at
           [55]http://www.gao.gov .

           If you or your staff have any questions about this report, please
           contact me at (202) 512-7118 or [56][email protected] . Contact
           points for our Offices of Congressional Relations and Public
           Affairs may be found on the last page of this report. GAO staff
           who made major contributions to this report are listed in appendix
           IV.

           Sincerely yours,

           Kathryn G. Allen
			  Director, Health Care
			  
			  Appendix I: Scope and Methodology

           Our analysis of QIOs' work with nursing homes had three major
           components: (1) site visits to five QIOs, (2) analysis of state
           survey data to compare homes that were assisted intensively with
           homes that were not, and (3) a Web-based survey of 51 QIOs.
			  
			  Site Visits

           We visited a QIO in each of five states to gather detailed
           information about QIOs' work with nursing homes from the
           perspective of the QIOs, nursing homes in the intensive
           participant group, and stakeholders; we used this information to
           address all three objectives.^1 We selected the states-*and by
           extension, the QIOs that worked in those states--on the basis of
           six criteria described in the following section. After selecting
           the QIOs, we identified nursing homes that received intensive
           assistance and stakeholders to contact for interviews. We
           conducted most of our site visit interviews in March and April
           2006.
			  
           Selection of QIOs

           We based our selection of QIOs on the following criteria:

           o Number of nursing home beds in the state. We divided the states
           into three groups of 17 states each based on the number of nursing
           home beds at the beginning of the 7^th SOW (2002). We over-sampled
           states with high numbers of nursing home beds by selecting one
           state with a low number of beds, one state with a medium number,
           and three states with a high number.

           o Evaluation score for the nursing home component of the 7^th SOW
           relative to scores of other QIOs. We divided the states into three
           groups of 17 based on the QIOs' evaluation scores for the 7^th
           SOW. To help us identify the possible determinants of scores, we
           selected more states at each end of the spectrum than in the
           middle: two states with scores in the bottom third, one state with
           a score in the middle third, and two states with scores in the top
           third.

^1To assist in the development of our site visit interview protocols, we
interviewed personnel from three additional QIOs (Massachusetts, Rhode
Island, and Washington) and staff from one nursing home in each of four
other states (Maryland, Massachusetts, New Hampshire, and Virginia).

           o State survey performance of homes selected for intensive
           assistance relative to homes not selected. We also considered the
           extent to which the homes selected for intensive assistance by a
           given QIO at the beginning of the 7^th SOW differed from the homes
           that were not selected, in terms of serious deficiencies cited on
           state surveys (both the proportion of homes cited in each group
           and the average number of serious deficiencies per home). We chose
           one QIO that selected worse homes, three QIOs that selected homes
           that were neither better nor worse, and one QIO that selected
           better homes.

           o Presence of a state-sponsored nursing home quality improvement
           program. At the time we selected QIOs for site visits, we were
           aware of four states that had state-sponsored quality improvement
           initiatives in place during the 7^th SOW.^2 To learn more about
           these efforts and how they interacted with and compared with
           efforts by the QIOs, we included one state (Florida) with its own
           quality improvement initiative.^3 After we made our selection, we
           learned that another state we had selected (Maine) had a
           state-sponsored quality improvement program.

           o QIO participation in the Collaborative Focus Facilities project.
           CMS has funded QIOs to conduct several special studies with
           nursing homes, including one in which the 17 participating QIOs
           each worked intensively with up to five nursing homes identified
           by their state survey agencies as having significant quality
           problems. To learn more about the challenges involved in working
           with low-performing homes, we selected two states whose QIOs
           participated in this project.

           o Census region. We selected states from four different regions of
           the country: Northeast, Midwest, South, and West.

           Using these criteria, we selected the following five states:
           Colorado, Florida, Iowa, Maine, and New York. Together these
           states represented 15 percent of nursing home beds nationwide at
           the beginning of the 7^th SOW (2002).

^2The four states were Florida, Maryland, Texas, and Washington. We
subsequently learned that four other states, (Maine, Michigan, Missouri,
and North Carolina) also had state-sponsored quality improvement programs.

^3We contacted officials of programs in six states: Florida, Maryland,
Michigan, North Carolina, Texas, and Washington.

           Selection of Nursing Homes

           Overall, we interviewed staff from 32 nursing homes in nine
           states. To assist in the development of our site visit protocols,
           we interviewed staff from 4 homes in four states. During the site
           visits to five states, we interviewed staff from 28 nursing homes.
           In each state, we interviewed staff from 4 to 8 nursing homes that
           received intensive assistance from the QIO, for a total of 28
           homes in these five states. The number of homes we selected in
           each of the five states visited varied depending on the number of
           homes the QIO was expected to select for intensive assistance, an
           expectation based on the number of homes in the state.
           Specifically, we selected either four homes or 7 percent of the
           maximum number of homes that each of the five QIOs was expected to
           assist intensively, whichever was greater.^4

           We chose homes on the basis of four characteristics: number of
           serious deficiencies in the standard state survey at the beginning
           of the 7^th SOW (2002), improvement in QM scores during the 7^th
           SOW, distance from the QIO (in order to include homes that were
           more difficult for QIOs to visit), and urban versus rural
           location. Specifically, we sought to include (1) at least one home
           that had one or more serious deficiencies and that finished in the
           top third of the intensively assisted homes in their state in
           terms of improvement on QM scores, and (2) at least one home that
           had one or more serious deficiencies and that finished in the
           bottom third of the intensively assisted homes in their state in
           terms of improvement on QM scores. For the remaining homes, we
           sought a group whose state survey deficiency levels and QM
           improvement scores were representative of the range among
           intensive participants in their state. However, the experiences of
           this sample of 32 homes cannot be generalized to the entire group
           of homes that received intensive assistance from the QIOs
           nationwide.
			  
			  Selection of Stakeholders

           In each state we also interviewed officials from three stakeholder
           groups: (1) the state survey agency; (2) the local affiliate of
           the American Health Care Association, which generally represents
           for-profit homes; and (3) the local affiliate of the American
           Association of Homes and Services for the Aging, which represents
           not-for-profit homes.
			  
^4QIOs working in states with at least 100 nursing homes were expected to
target 10 to 15 percent of all homes in the state for intensive
assistance. In the state we selected that had the highest number of homes
(Florida), 7 percent of the homes in the state equaled approximately 8
homes.

           Analysis of State Survey Data

           To assess the characteristics of the nursing homes that were
           selected by the QIOs for intensive assistance from among the homes
           that volunteered, we analyzed 3 years of standard state survey
           data on deficiencies cited at nursing homes and compared the
           results for homes that were assisted intensively with results for
           homes that were not; we used this information to address our first
           objective.^5 The analysis involved three steps:
			  
			  Identifying Homes with Three Standard Surveys

           To identify homes whose performance was consistently lower or
           higher than other homes in their state prior to the selection of
           homes by the QIOs, we included in our analysis only homes for
           which we were able to identify three standard surveys from January
           1, 1999, through November 1, 2002. Using the state survey calendar
           year summary files for 1999 through 2002 for the 50 states and the
           District of Columbia, we obtained 3 years of deficiency data from
           standard surveys for 16,303 homes.^6
			  
			  Classifying Homes as Low-, Moderately, or High-Performing

           CMS classifies deficiencies according to their scope and severity.
           For each of the three surveys, we ranked all of the nursing homes
           in each state based on the number of deficiencies in two
           categories: (1) actual harm or immediate jeopardy and (2)
           potential for more than minimal harm.^7 Deficiencies in the first
           category are considered serious deficiencies. We gave more weight
           to the serious deficiencies by sorting the homes first on the
           number of deficiencies in the first category and then on the
           number of deficiencies in the second category. Homes with the same
           number of deficiencies in each category were assigned the same
           rank. Based on these rankings, we identified homes in the bottom
           and top quartile in each state in each survey.^8
			  
^5This analysis drew on data from the On-line Survey, Certification, and
Reporting system (OSCAR), a database maintained by CMS that compiles the
results of every state survey conducted at Medicare- and
Medicaid-certified facilities nationwide.

^6We eliminated from the analysis 1,946 homes that had a standard survey
in the year prior to November 1, 2002, but for which we were unable to
identify two additional surveys during the period we specified. The homes
that we eliminated represented a larger proportion of the group of homes
not selected by the QIOs (11.8 percent) than of the group of homes that
were selected by the QIOs (3.4 percent).

^7CMS defines immediate jeopardy as actual or potential for death/serious
injury.

^8Because homes with the same number of deficiencies were assigned the
same rank, in some cases the top and bottom quartiles included more than
25 percent of the homes in the state. We based our classification of homes
on their performance level relative to other homes in the state to take
into account the inconsistency in how states conduct surveys, a problem we
have reported on since 1998. An alternative approach, which would not take
into account the inconsistency in how states conduct surveys, would be to
classify homes based on the absolute number of deficiencies they had
received*for example, to classify all homes with five or more serious
deficiencies as low-performing homes. For data on inconsistencies, see
[57]GAO-06-117 and [58]GAO-07-241 .			  

           We classified homes as low-performing if they ranked in the bottom
           quartile in the most recent of the three surveys and in at least
           one of the two preceding surveys. We classified homes as
           high-performing if they ranked in the top quartile in the most
           recent of the three surveys and in at least one of the two
           preceding surveys. We classified homes as moderately performing if
           they did not meet the criteria for inclusion in either the low- or
           high-performing group. Of the 16,303 homes with three standard
           state surveys during the period we specified, we classified 15
           percent as low-performing, 65 percent as moderately performing,
           and 20 percent as high-performing.

           To assess the stability of our categorization of homes as low- (or
           high-) performing, we ran a logistic regression model to predict
           the probability of a home being categorized as low- (or high-)
           performing in the most recent of the three surveys given its
           categorization in the two prior surveys. The regression results
           showed that homes that were categorized as low- (or high-)
           performing in one survey were significantly more likely to be
           categorized as low- (or high-) performing in the other surveys as
           well.
			  
			  Determining Statistically Significant Differences between Homes
			  Assisted Intensively and Homes Not Assisted Intensively by the QIOs

           Our final step was to determine, on both a nationwide and
           state-by-state basis, whether there was a statistically
           significant difference in the proportion of (1) low-performing
           homes, (2) moderately performing homes, and (3) high-performing
           homes in the group assisted intensively by the QIOs compared with
           the group not assisted intensively.^9
			  
^9We used the Satterthwaite t-test because it does not require the
variances of the two groups to be equal. We rejected the null hypothesis
that the proportions of two groups were equal when the p-value from the
Satterthwaite t-test was less than 0.05.
			  
			  Web-Based Survey of QIOs

           To gather information about the characteristics of the QIOs,
           including their process for selecting homes for intensive
           assistance from the pool of volunteers and the interventions they
           used, on July 19, 2006, we launched a two-part Web-based survey of
           QIOs in all 50 states and the District of Columbia; we used this
           information to address objectives one and two.^10 We achieved a
           100 percent response rate. The first part of the survey gathered
           information about the primary personnel who worked with nursing
           homes during the 7^th SOW, including information about their
           employment with the QIO, and their relevant credentials and
           experience.^11 The second part of the survey gathered information
           on a range of other topics, including information about
           stakeholder involvement with the QIO, recruitment and selection of
           nursing homes for intensive assistance, interventions used with
           intensive participants, interventions used with homes statewide,
           and QIOs' communication with CMS. We specifically inquired about
           QIOs' use of six interventions: (1) mailings, faxes, and e-mails;
           (2) conferences; (3) small group meetings; (4) conference calls
           and video or Web conferences with multiple homes; (5) telephone
           conversations with individual homes; and (6) on-site visits.^12 We
           asked QIOs to rank and provide information on the two
           interventions they relied on most to assist homes statewide and on
           the three interventions they relied on most to assist homes in the
           intensive participant group.^13 We also asked QIOs to rank the
           effectiveness of the interventions they used and to identify the
           interventions they would use if they could do the 7^th SOW over
           again.
			  
^10We asked the QIOs to complete a separate survey for each state in which
they worked during the 7th SOW.

^11We defined primary personnel as employees, subcontractors, or
consultants who worked with nursing homes or provided direct oversight of
those individuals, excluding administrative support staff and individuals
who worked less than 20 percent of a full-time work week on the nursing
home component.

^12QIOs were also given the option of specifying other interventions they
used.

^13Because QM improvement among intensive participants constituted the
largest part of the QIOs' contract evaluation score, we asked QIOs to rank
and provide detailed information on a greater number of interventions for
intensive participants than for statewide participants.

           Appendix II: Publicly Reported Quality Measures
			  
           In November 2002, CMS began a national Nursing Home Quality
           Initiative that included the development of QMs that would be
           publicly reported on the CMS Web site called Nursing Home Compare.
           CMS has continued to refine the QMs and, as shown in table 4, has
           dropped some QMs and added others.

                        1. identifying nursing homes that had three standard
                        state surveys from 1999 through 2002;
                        2. ranking nursing homes in each state in each year,
                        based on the number of serious and other
                        deficiencies, and then classifying homes as
                        consistently low-, moderately, or high-performing;
                        and
                        3. identifying on a nationwide and state-by-state
                        basis any statistically significant differences
                        between homes selected and not selected by the QIO,
                        in terms of the proportion of low-, moderately, or
                        high-performing nursing homes.

Table 4: QMs as of November 2002 and as of February 2007

                                          QM as of November QM as of February 
QM                                           2002              2007        
Chronic care QM                                                            
Decline in activities of daily living                                      
Pressure ulcers                                                            
Pressure ulcers^a                                                          
Pressure ulcers in high-risk residents                                     
Pressure ulcers in low-risk residents                                      
Inadequate pain management                                                 
Physical restraints                                                        
Infections                                                                 
Weight loss                                                                
Urinary tract infection                                                    
Catheter insertion                                                         
Depression                                                                 
Bowel or bladder control in low-risk                                       
residents                                                                  
Bedfast                                                                    
Worsening ability to move about room                                       
Administration of influenza                                                
vaccination during flu season                                              
Assessment for and administration of                                       
pneumococcal vaccination                                                   
Post-acute-care QM                                                         
Failure to improve and manage delirium                                     
Failure to improve and manage delirium                                     
(facility-adjusted)^a                                                      
Inadequate pain management                                                 
Improvement in walking                                                     
Pressure ulcers                                                            
Administration of influenza                                                
vaccination during flu season                                              
Assessment for and administration of                                       
pneumococcal vaccination                                                   

Source: CMS.

aFacility-level risk adjustment was intended to take into account the fact
that some homes may admit frailer, sicker residents, or may specialize in
a particular area of care that may account for a larger proportion of
residents for a particular measure. CMS reported the delirium measure both
with and without facility adjustment.

Appendix III: Comments from the Centers for Medicare & Medicaid Services

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Kathryn G. Allen, (202) 512-7118 or [59][email protected]

Acknowledgments

In addition to the contact named above, Walter Ochinko, Assistant
Director; Nancy Fasciano; Sara Imhof; Elizabeth T. Morrison; Colbie
Porter; and Andrea Richardson made key contributions to this report.

Related GAO Products

Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not Deterred
Some Homes from Repeatedly Harming Residents. [60]GAO-07-241 . Washington,
D.C.: March 26, 2007.

Nursing Homes: Despite Increased Oversight, Challenges Remain in Ensuring
High-Quality Care and Resident Safety. [61]GAO-06-117 . Washington, D.C.:
December 28, 2005.

Nursing Home Deaths: Arkansas Coroner Referrals Confirm Weaknesses in
State and Federal Oversight of Quality of Care. [62]GAO-05-78 .
Washington, D.C.: November 12, 2004.

Nursing Home Fire Safety: Recent Fires Highlight Weaknesses in Federal
Standards and Oversight. [63]GAO-04-660 . Washington D.C.: July 16, 2004.

Nursing Home Quality: Prevalence of Serious Problems, While Declining,
Reinforces Importance of Enhanced Oversight. [64]GAO-03-561 . Washington,
D.C.: July 15, 2003.

Nursing Homes: Public Reporting of Quality Indicators Has Merit, but
National Implementation Is Premature. [65]GAO-03-187 . Washington, D.C.:
October 31, 2002.

Nursing Homes: Quality of Care More Related to Staffing than Spending.
[66]GAO-02-431R . Washington, D.C.: June 13, 2002.

Nursing Homes: More Can Be Done to Protect Residents from Abuse.
[67]GAO-02-312 . Washington, D.C.: March 1, 2002.

Nursing Homes: Federal Efforts to Monitor Resident Assessment Data Should
Complement State Activities. [68]GAO-02-279 . Washington, D.C.: February
15, 2002.

Nursing Homes: Sustained Efforts Are Essential to Realize Potential of the
Quality Initiatives. [69]GAO/HEHS-00-197 . Washington, D.C.: September 28,
2000.

Nursing Home Care: Enhanced HCFA Oversight of State Programs Would Better
Ensure Quality. [70]GAO/HEHS-00-6 . Washington, D.C.: November 4, 1999.

Nursing Home Oversight: Industry Examples Do Not Demonstrate That
Regulatory Actions Were Unreasonable. [71]GAO/HEHS-99-154R . Washington,
D.C.: August 13, 1999.

Nursing Homes: Proposal to Enhance Oversight of Poorly Performing Homes
Has Merit. [72]GAO/HEHS-99-157 . Washington, D.C.: June 30, 1999.

Nursing Homes: Complaint Investigation Processes Often Inadequate to
Protect Residents. [73]GAO/HEHS-99-80 . Washington, D.C.: March 22, 1999.

Nursing Homes: Additional Steps Needed to Strengthen Enforcement of
Federal Quality Standards. [74]GAO/HEHS-99-46 . Washington, D.C.: March
18, 1999.

California Nursing Homes: Care Problems Persist Despite Federal and State
Oversight. [75]GAO/HEHS-98-202 . Washington, D.C.: July 27, 1998.

(290500)

GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( [76]www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
[77]www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: [78]www.gao.gov/fraudnet/fraudnet.htm
E-mail: [79][email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [80][email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [81][email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

[82]www.gao.gov/cgi-bin/getrpt?GAO-07-373 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Kathryn G. Allen, (202) 512-7118,
[email protected].

Highlights of [83]GAO-07-373 , a report to the Ranking Member, Committee
on Finance, U.S. Senate

May 2007

NURSING HOMES

Federal Actions Needed to Improve Targeting and Evaluation of Assistance
by Quality Improvement Organizations

In 2002, CMS contracted with Quality Improvement Organizations (QIO) to
help nursing homes address quality problems such as pressure ulcers, a
deficiency frequently identified during routine inspections conducted by
state survey agencies. CMS awarded $117 million over a 3-year period to
the QIOs to assist all homes and to work intensively with a subset of
homes in each state. Homes' participation was voluntary. To evaluate QIO
performance, CMS relied largely on changes in homes' quality measures
(QM), data based on resident assessments routinely conducted by homes. GAO
assessed QIO activities during the 3-year contract starting in 2002,
focusing on (1) characteristics of homes assisted intensively, (2) types
of assistance provided, and (3) effect of assistance on the quality of
nursing home care. GAO conducted a Web-based survey of all 51 QIOs,
visited QIOs and homes in five states, and interviewed experts on using
QMs to evaluate QIOs.

[84]What GAO Recommends

GAO recommends that the CMS Administrator (1) further increase the number
of low-performing homes that QIOs work with intensively, (2) improve
monitoring and evaluation of QIO activities, and (3) require QIOs to share
with CMS the identity of homes assisted intensively in order to facilitate
evaluation. CMS agreed with the first two recommendations, but did not
specifically indicate if it agreed with the third.

Although more homes volunteered to work with the QIOs than CMS expected
them to assist intensively, QIOs typically did not target their assistance
to the low-performing homes that volunteered. Most QIOs' primary
consideration in selecting homes was their commitment to working with the
QIO. CMS did not specify selection criteria for intensive participants but
contracted with a QIO that developed guidelines encouraging QIOs to select
committed homes and exclude those with many survey deficiencies or QM
scores that were too good to improve significantly. Consistent with the
guidelines, few QIOs targeted homes with a high level of survey
deficiencies, and eight QIOs explicitly excluded these homes. GAO's
analysis of state survey data confirmed that selected homes were less
likely than other homes to be low-performing in terms of identified
deficiencies. Most state survey and nursing home trade association
officials interviewed by GAO believed QIO resources should be targeted to
low-performing homes.

QIOs were provided flexibility both in the QMs on which they focused their
work with nursing homes and in the interventions they used. Most QIOs
chose to work on chronic pain and pressure ulcers, and most used the same
interventions*conferences and distribution of educational materials*to
assist homes statewide. The interventions used to assist individual homes
intensively varied and included on-site visits, conferences, and small
group meetings. Just over half the QIOs reported that they relied most on
on-site visits to assist intensive participants. Sixty-three percent said
such visits were their most effective intervention. Of the 15 QIOs that
would have changed the interventions used, most would make on-site visits
their primary intervention. Homes indicated that they were less satisfied
with the program when their QIO experienced high staff turnover or when
their QIO contact possessed insufficient expertise.

Shortcomings in the QMs as measures of nursing home quality and other
factors make it difficult to measure the overall impact of the QIOs on
nursing home quality, although staff at most of the nursing homes GAO
contacted attributed some improvements in the quality of resident care to
their work with the QIOs. The extent to which changes in homes' QM scores
reflect improvements in the quality of care is questionable, given the
concerns raised by GAO and others about the validity of the QMs and the
reliability of the resident assessment data used to calculate them. In
addition, quality improvements cannot be attributed solely to the QIOs, in
part because the homes that volunteered and were selected for intensive
assistance may have differed from other homes in ways that would affect
their scores; these homes may also have participated in other quality
improvement initiatives. Ongoing CMS evaluation of QIO activities for the
contract that began in August 2005 is being hampered by a 2005 Department
of Health and Human Services decision that QIO program regulations
prohibit QIOs from providing to CMS the identities of homes being assisted
intensively.

References

Visible links
  48. http://www.medicare.gov/NHCompare/home.asp
  49. http://www.gao.gov/cgi-bin/getrpt?GAO-07-241
  50. http://www.gao.gov/cgi-bin/getrpt?GAO-03-187
  51. http://www.gao.gov/cgi-bin/getrpt?GAO-02-279
  52. http://www.gao.gov/cgi-bin/getrpt?GAO-03-187
  53. http://www.gao.gov/cgi-bin/getrpt?GAO-02-279
  54. http://www.gao.gov/cgi-bin/getrpt?GAO-03-187
  55. http://www.gao.gov/
  56. mailto:[email protected]
  57. http://www.gao.gov/cgi-bin/getrpt?GAO-06-117
  58. http://www.gao.gov/cgi-bin/getrpt?GAO-07-241
  59. mailto:[email protected]
  60. http://www.gao.gov/cgi-bin/getrpt?GAO-07-241
  61. http://www.gao.gov/cgi-bin/getrpt?GAO-06-117
  62. http://www.gao.gov/cgi-bin/getrpt?GAO-05-78
  63. http://www.gao.gov/cgi-bin/getrpt?GAO-04-660
  64. http://www.gao.gov/cgi-bin/getrpt?GAO-03-561
  65. http://www.gao.gov/cgi-bin/getrpt?GAO-03-187
  66. http://www.gao.gov/cgi-bin/getrpt?GAO-02-431R
  67. http://www.gao.gov/cgi-bin/getrpt?GAO-02-312
  68. http://www.gao.gov/cgi-bin/getrpt?GAO-02-279
  69. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-00-197
  70. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-00-6
  71. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-99-154R
  72. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-99-157
  73. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-99-80
  74. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-99-46
  75. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-98-202
  76. http://www.gao.gov/
  77. http://www.gao.gov/
  78. http://www.gao.gov/fraudnet/fraudnet.htm
  79. mailto:[email protected]
  80. mailto:[email protected]
  81. mailto:[email protected]
  82. http://www.gao.gov/cgi-bin/getrpt?GAO-07-373
  83. http://www.gao.gov/cgi-bin/getrpt?GAO-07-373
*** End of document. ***