[Federal Register Volume 70, Number 140 (Friday, July 22, 2005)]
[Notices]
[Pages 42331-42336]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 05-14505]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Centers for Medicare & Medicaid Services

[CMS-3142-FN]


Medicare Program; Evaluation Criteria and Standards for Quality 
Improvement Program Contracts

AGENCY: Centers for Medicare & Medicaid Services (CMS), HHS.

ACTION: Final notice.

-----------------------------------------------------------------------

SUMMARY: This final notice describes the evaluation criteria we will 
use to evaluate the Quality Improvement Organizations (QIOs) under 
their contracts with us, for efficiency and effectiveness in accordance 
with the Social Security Act. These evaluation criteria are based on 
the tasks and related subtasks set forth in the QIO's Scope of Work 
(SOW). The current 7th SOW includes Tasks 1 through 4, with subtasks 
included under all tasks,

[[Page 42332]]

excluding Task 4. QIOs were awarded contracts for the 7th SOW, or 7th 
Round, for 3 years, with staggered starting dates beginning August 
2002, November 2002, and February 2003. This final notice also responds 
to the public comments received regarding the evaluation criteria 
published in July 2004.

DATES: Effective August 22, 2005.

FOR FURTHER INFORMATION CONTACT: Maria Hammel, (410) 786-1775.

SUPPLEMENTARY INFORMATION:

I. Background

    The Peer Review Improvement Act of 1982 (Title I, Subtitle C of 
Public Law 97-248) amended Part B of Title XI of the Social Security 
Act (the Act) to establish the Peer Review Organization (PRO) program. 
The PRO program (now called the Quality Improvement Organization (QIO) 
program) was established to redirect, simplify, and enhance the cost-
effectiveness and efficiency of the medical peer review process. 
Sections 1152 and 1153 of the Act define the types of organizations 
eligible to become QIOs, and establish certain limitations and 
priorities regarding QIO contracting.
    The Secretary enters into contracts with QIOs to perform three 
broad functions:
     Improve quality of care for beneficiaries by ensuring that 
beneficiary care meets professionally recognized standards of health 
care;
     Protect the integrity of the Medicare Trust Fund by 
ensuring that Medicare only pays for services and items that are 
reasonable and medically necessary and that are provided in the most 
economical setting;
     Protect beneficiaries by expeditiously addressing 
individual cases such as beneficiary quality of care complaints, 
contested hospital issued notices of noncoverage (HINNs), alleged 
Emergency Medical Treatment and Labor Act (EMTALA) violations (patient 
dumping), and other statutory responsibilities.
    Section 1154 of the Act requires that QIOs review those services 
furnished by physicians; other health care practitioners; and 
institutional and non-institutional providers of health care services, 
including health maintenance organizations and competitive medical 
plans. Section 109 of the Medicare Prescription Drug, Improvement, and 
Modernization Act of 2003 (MMA), Pub. L. 108-173, amended section 
1154(a)(1) of the Act to expand the scope of review of QIOs to include 
Medicare Advantage Organizations and prescription drug sponsors. 
Section 109 of the MMA also created a new section 1154(a)(17) of the 
Act, which requires QIOs to offer to providers, practitioners, Medicare 
Advantage Plans, and prescription drug sponsors quality improvement 
assistance pertaining to prescription drug therapy. We will not 
evaluate QIOs on these provisions in the current Scope of Work (SOW) 
because these provisions of sections 1154(a)(1) and (a)(17) of the Act 
were not included in the contract.
    Section 1153(h)(2) of the Act requires the Secretary to publish in 
the Federal Register the general criteria and standards that would be 
used to evaluate the efficient and effective performance of contract 
obligations by QIOs and to provide the opportunity for public comment. 
The QIO contracts for the 7th SOW were awarded for 3 years with 
starting dates staggered into three approximately equal groups (rounds) 
starting August 2002, November 2002, and February 2003, respectively.

II. Provisions of the Notice With Comment

    On July 23, 2004, we published a notice with comment in the Federal 
Register titled ``Medicare Program; Evaluation Criteria and Standards 
for Quality Improvement Organizations.'' The comment period for this 
notice closed on August 23, 2004. The evaluation criteria published in 
the notice are currently being used to evaluate QIO performance on the 
7th SOW. The evaluation criteria is listed here for the reader's 
convenience. No modifications were made to the evaluation criteria 
based on comments provided in response to the notice.

A. Measuring QIO Performance

    Under the 7th Round contracts, QIOs are responsible for completing 
tasks in the following four areas, with additional subtasks contained 
in the first three areas:
Task 1--Improving Beneficiary Safety and Health Through Clinical 
Quality Improvement

a. Nursing Home
b. Home Health
c. Hospital
d. Physician Office
e. Underserved and Rural Beneficiaries
f. Medicare+Choice Organizations (M+COs), now called Medicare Advantage 
Organizations (MAs)
Task 2--Improving Beneficiary Safety and Health Through Information and 
Communications
a. Promoting the Use of Performance Data
b. Transitioning to Hospital-Generated Data
c. Other Mandated Communications Activities
Task 3--Improving Beneficiary Safety and Health Through Medicare 
Beneficiary Protection Activities
a. Beneficiary Complaint Response Program
b. Hospital Payment Monitoring Review Program
c. All Other Beneficiary Protection Activities

    Task 4--Improving Beneficiary Safety and Health Through 
Developmental Activities (Special Studies defined as work that we 
direct a QIO to perform or work that a QIO elects to perform with our 
approval that is not currently defined in the Tasks, but falls within 
the scope of the contract and section 1154 of the Act).
    Under this contract, to merit having its contract renewed non-
competitively, the QIO must meet the performance criteria (including a 
score of 1.0 or greater for Tasks 1a through 1e and 2b) on 10 of 12 
subtasks (9 of 11 for States with no MA plans) of Tasks 1 through 3 of 
the 7th SOW. To renew the QIO's contract non-competitively for both of 
the subtasks that do not meet the criteria, the QIO must have: (1) 
Achieved a score of 0.6 or better on all quantitative subtasks, and (2) 
for the remaining subtasks only, in the judgment of the Project 
Officer, the QIO expended a reasonable effort to address these 
subtasks, and developed and implemented an appropriate initial work 
plan. The work plan must have been assessed by the Project Officer 
during the contract period to determine if it was achieving results 
likely to lead to success in meeting contractual performance 
expectations and had made appropriate adjustments to its work plan 
based on these results.
    To be considered successful (that is, meeting the criteria outlined 
in the J-7 found at http://www.cms.hhs.gov/qio/2.asp), though not 
meriting a non-competitive renewal, the QIO must meet the performance 
criteria (including a score of 1.0 or greater for Tasks 1a through 1e 
and 2b) on 9 of 12 subtasks (8 of 11 for States with no MA plans) of 
Tasks 1 through 3 of the 7th Round Contract. For the subtasks that do 
not meet the criteria, the QIO must--
     Achieve a score of 0.6 or better on all quantitative 
subtasks;
     For the remaining subtasks only, in the judgment of the 
Project Officer, the QIO has expended a reasonable effort to address 
these subtasks, developed and implemented an appropriate initial work 
plan that was assessed by the Project Officer during the contract 
period to determine if it was achieving

[[Page 42333]]

results likely to lead to success in meeting contractual performance 
expectations, and had made appropriate adjustments to its work plan 
based on these results; and
     Failed to meet the criteria in no more than two subtasks 
of any one task.
    For Task 4, except as provided in Task 3b that is evaluated by the 
Task Leader, all special studies approved under this task will be 
evaluated individually, based on study-specific evaluation criteria. 
The QIO's success or failure on a special study will not be factored 
into the evaluation of the QIO's work under Tasks 1 through 3.
    However, meeting the minimum performance standards does not 
guarantee a noncompetitive renewal of the QIO's contract. For example, 
an organization within a particular State meeting the definition of a 
QIO may express interest in competing for a contract currently held by 
a QIO from outside that State, according to section 1153(i) of the Act. 
In this case, we will compete the contract despite acceptable 
performance by the current QIO. We will make a final decision on 
renewal/non-renewal by the end of the 30th month of the 7th Round 
contract. We will issue a ``Notice of Intent to Non-renew the QIO 
Contract'' letter to all QIOs that do not meet the minimum performance 
standards no later than the end of the 33rd month of the contract. The 
QIO will be considered to have met minimum performance standards if the 
QIO had demonstrated acceptable performance in each Task area as 
specified in section III of this notice, Standards for Minimum 
Performance.
    If the QIO has not met the criteria to merit a noncompetitive 
renewal, it will be notified of our intention not to renew its contract 
and will be informed of its right to request an opportunity to provide 
information about its performance under the contract to a CMS-wide 
panel. The panel includes representatives from each of the four QIO 
Regional Offices and the Central Office. The QIO's Project Officer will 
not be eligible to represent the Regional Office on the panel when it 
reviews the work of his or her QIO. However, the Project Officer will 
be available to answer any questions. Also, the QIO will be given the 
opportunity to provide additional information. The panel will have the 
right to create its own procedures, but must apply them consistently to 
all QIOs. At a minimum, the panel will use the criteria listed below 
for all Tasks:
     The degree of collaboration the QIO exhibited with the 
Quality Improvement Organization Support Centers (QIOSCs) and other 
QIOs, both by sharing the lessons and tools it developed and by 
adopting practices and tools developed by other QIOs.
     Whether the QIO was a new contractor in the 7th SOW.
     Whether specific identifiable circumstances uniquely 
interfered with the QIO's efforts.
     Evidence suggesting that the QIO has done exceptional work 
in one or more of the other Task areas.
     Any other issues that the panel may deem relevant.
    Upon completion of its review, the panel will recommend a final 
disposition of the QIO's contract renewal to the Director of CMS' 
Office of Clinical Standards and Quality (OCSQ).

B. Standards for Minimum Performance

General Criteria
    We will evaluate the QIO's performance on each sub-task by some 
combination of the following elements:
     Statewide improvement on the quality measure(s).
     Improvement on the quality of care measure(s) among a 
group of identified participants as defined within each subtask.
     Satisfaction among providers and practitioners regarding 
their interaction with the QIO.
    Satisfaction will be assessed using a survey, the purpose of which 
will be to:
     Measure satisfaction as one component of the QIO's 
evaluation.
     Identify opportunities where the QIO can improve 
satisfaction.
    Task 1 (including subtasks a through e) and subtask 2b will be 
evaluated quantitatively. The QIO's success will be measured by 
assessing its relative improvement on each evaluation criterion. The 
term ``improvement'' as used in the 7th Round Contract will be defined 
mathematically to mean the relative reduction in the failure rate. The 
expected minimum improvement level, as determined by our management and 
defined in the J-7 at http:www.cms.hhs.gov/qio/2.asp, will serve as the 
reference point for each calculated relative improvement.
    In a number of the Task 1 subtasks, statewide improvement will be 
averaged with the improvement among a set of identified participant 
providers. In these cases, we have set a target percentage of 
identified participant providers. The relative weights of the statewide 
improvement and of identified participants' improvement will combine to 
equal 80 percent of the subtask's weight, and will be a function of the 
percentage of the target percentage (up to 150 percent) that the QIO 
identifies as participants. Tasks 1f, 2a, 2c and all of Task 3 will be 
evaluated by the Project Officer using qualitative measures based on 
information provided in reports developed from data provided by the 
QIOs on the QIO's status to date.

C. Task Specific Standards

1. Task 1--Improving Beneficiary Safety and Health Through Clinical 
Quality Improvement
a. Task 1a--Nursing Home Quality Improvement
    The QIO will be held accountable for improvement in the quality of 
care measure rates for all nursing homes in the State and for 
identified participant nursing homes. QIOs will be evaluated based on 
the following components: Statewide improvement on the set of three to 
five publicly reported quality of care measures that the QIO has 
selected in consultation with stakeholders, improvement in the selected 
nursing home publicly reported quality of care measures for identified 
participants, and nursing home satisfaction based on a survey of 
identified participating nursing homes. To view the weighting criteria 
for each component, go to http://www.cms.hhs.gov/qio/2.asp for a copy 
of the J-7.
b. Task 1b--Home Health Quality Improvement
    The QIO will be held accountable for improvement in the Outcome 
Based Quality Improvement (OBQI) quality of care measure rates for a 
set of home health agencies that are identified participants. The QIOs 
will be evaluated based on the following components: The extent to 
which the number of participating home health agencies, with 
significant improvement in a targeted outcome, equals or exceeds 30 
percent of the total number of home health agencies in the State, and 
the identified participant satisfaction that will be measured by a 
survey of identified participant home health agencies using a composite 
measure of satisfaction that reflects the type of activities that QIOs 
are expected to have undertaken with these providers.
c. Task 1c--Hospital Quality Improvement
    QIOs will be evaluated on the following criteria: Statewide 
improvement on the quality of care measures listed in the 7th Round 
Contract, and hospital satisfaction based on feedback from the 
hospitals in the State. To view the specific criteria, go to http://www.cms.hhs.gov/qio/2.asp for a copy of the J-7.

[[Page 42334]]

d. Task 1d--Physician Office Quality Improvement
    QIOs will be evaluated based on the following general criteria: 
statewide improvement on quality of care measures, improvement on 
diabetes and cancer screening quality of care measures for identified 
participant physicians, and physician satisfaction based on feedback 
from physician designees in the State who participated with the QIO. To 
view the specific criteria for this task, go to http://www.cms.hhs.gov/qio/2.asp for a copy of the J-7.
e. Task 1e--Underserved and Rural Beneficiaries Quality Improvement
    The QIO's work on this task will be primarily evaluated on the 
success of the QIO's efforts to reduce disparity between the targeted 
underserved group and their geographically relevant non-underserved 
reference group from baseline to re-measurement. To be judged to have 
performed minimally successful on this task, the QIO must demonstrate 
disparity reduction. QIOs will also be evaluated on three factors that 
collectively demonstrate knowledge generated by the QIO about the 
underserved target group, the interventions planned upon the basis of 
that knowledge, the use of literature on effective interventions, and 
by demonstrating the effectiveness of their interventions through 
analyses comparing the intervention group and a contrast group. To view 
the specific criteria for this task, go to http://www.cms.hhs.gov/qio/2.asp for a copy of the J-7.
f. Task 1f--Medicare + Choice Organizations (M+COs) (Now Called 
Medicare Advantage Organizations (MAs)) Quality Improvement
    QIOs will be expected to have demonstrated appropriate activity to 
include MAs in Tasks 1a to 1e as determined by the Project Officer. We 
will survey MAs that have worked with the QIO using a composite measure 
of satisfaction that reflects the types of activities that QIOs are 
expected to have undertaken with these organizations. We will further 
use the results of the Medicare+Choice Quality Review Organizations 
(M+CQRO) or accreditation organization evaluation of the Quality 
Assessment and Performance Improvement (QAPI) projects to determine if 
expected improvement was demonstrated.
2. Task 2--Improving Beneficiary Safety and Health Through Information 
and Communications
a. Task 2a--Promoting the Use of Performance Data
    QIO success will be assessed on the timely completion and 
submission of a project work plan, timely completion and submission of 
all required reports and deliverables, and the extent to which the QIO 
uses information we have provided as well as any other feedback the QIO 
receives to refine its project activities to achieve the desired 
outcome.
b. Task 2b--Transitioning to Hospital-Generated Data
    The evaluation for this task will be based on the following 
elements:
     We will determine the completeness of the assessment 
survey information for each hospital.
     We will review hospital data submitted to the national 
repository via QualityNet Exchange to determine the proportion of 
hospitals within the State that have implemented a data abstraction 
system to abstract quality of care measures.
     We will review hospital satisfaction with the QIO data 
abstraction support. To view specific criteria for this task, go to 
http://www.cms.hhs.gov/qio/2.asp for a copy of the J-7.
c. Task 2c--Other Mandated Communication Activities
    QIO success on this task will be assessed on the following 
elements: The establishment and use of a Consumer Advisory Council to 
advise and provide guidance regarding consumer related activities, the 
QIO's success at broadening consumer representation on the QIO Board of 
Directors, the successful operation of a beneficiary helpline, and the 
publication and distribution of an annual report.
3. Task 3--Improving Beneficiary Safety and Health Through Medicare 
Beneficiary Protection Activities
a. Task 3a--Beneficiary Complaint Response Program
    QIO success will be assessed by the timeliness of completed 
reviews, quality improvement activities as the result of beneficiary 
complaints, reliability of the review of cases as determined by QIO 
assessment of the review determinations, and beneficiary satisfaction 
with the complaint process.
b. Task 3b--Hospital Payment Monitoring Review Program
    The QIO must complete reviews within the prescribed timeframes. The 
QIO must also meet one of the following criteria: with respect to the 
absolute payment error rate, the follow-up payment error rate must be 
no greater than 1.5 standard errors above the baseline error rate, or 
the QIO must have made acceptable progress in improving provider 
performance in relation to all projects approved or directed by us.
c. Task 3c--Other Beneficiary Protection Activities
    The QIO will be assessed on the timeliness of reviews for HINN/
NODMAR, EMTALA review, other case review activities and post review 
activities.

III. Analysis of and Responses to Public Comments and Provisions of the 
Final Notice

    We received several public comments on the 2004 Federal Register 
notice with comment period.
    Comment: One commenter expressed concern over the hospital 
satisfaction survey in Task 1c. The commenter noted that some hospitals 
have changed to acute care hospitals late in the SOW. The commenter 
believes this does not provide the QIO ample opportunity to work with 
the hospital before the hospital completes the satisfaction survey. The 
commenter recommended that we establish a cut-off date for new entries 
as acute care hospitals participating in the satisfaction survey.
    Response: While we understand the concern that hospitals with only 
recent experience in acute care could have an impact on the hospital 
satisfaction survey, we do not believe that it would be a significant 
impact for the 7th SOW. The Task 1c satisfaction scores from the first 
two rounds appear to support our position. All QIOs in the first two 
rounds received scores that met or exceeded the 80 percent passing 
threshold. The suggestion to include a cut-off date is a reasonable one 
that we can consider for subsequent Scopes of Work. We intend to 
evaluate all rounds for the current SOW identically.
    Comment: One commenter expressed concern about the project plan 
requirements in Task 2a. Specifically, the commenter stated that the 
task only required a project plan for the Nursing Home Quality 
Initiative. The commenter requested more specific language in the 
evaluation criteria to address this issue.
    Response: For the 7th SOW, we are requiring only one formal project 
plan for the Nursing Home Quality Initiative. A deliverable has not 
been added for subsequent plans. QIOs will not be held accountable for 
failing to deliver project plans that are not required deliverables for 
the task.

[[Page 42335]]

    Comment: One commenter stated that there are no historical data to 
demonstrate that nursing homes' thresholds and home health thresholds 
are achievable or realistic.
    Response: We believe that the thresholds are achievable for most 
QIOs. The results of the 1st Round 28-month evaluations show that the 
majority of the QIOs (87 percent) achieved or exceeded the target 
performance. Therefore, there is no indication that these thresholds 
should be changed.
    Comment: One commenter stated that the tasks to be evaluated 
subjectively would be less ambiguous if the components of the 
evaluation were known before the start of the SOW.
    Response: We agree with this comment. However, QIOs were provided a 
copy of the J-7 before the start of the SOW. The tool used to do the 
actual evaluation was based on the materials provided in the J-7, and 
did not include any criteria or standards not in the SOW. We will 
produce the tool for the 8th SOW early in the contract period. It will 
be distributed to QIOs as soon as it is available.
    Comment: Three commenters questioned how statistical significance 
could be calculated for home health agencies with a small number of 
episodes of care.
    Response: We use the Fisher's exact test to calculate statistical 
significance for agency outcomes with 10 to 30 episodes of care. This 
test does not require a large sample to estimate statistical 
significance. More information on this test can be found in Categorical 
Data Analysis by Alan Agresti. Additionally, we tested the impact of 
small HHAs by recalculating evaluation results. Excluding all HHAs with 
fewer than 30 episodes of care did not substantively improve the 
overall evaluation results. Based on this information, we decided not 
to modify the 1b evaluation criteria.
    Comment: One commenter questioned how we determined the home health 
task denominator for the 30 percent.
    Response: The home health denominator is made up of two components. 
It includes identified participants and non-identified participants. 
Identified participants are defined as all home health agencies that 
submitted an OBQI plan of action (POA) and have at least a one 3-bar 
OBQI report for any reporting period ending at least 12 months after 
the POA submission date. A 3-bar report allows the HHA to compare 
current outcome rates to prior year outcome rates and national outcome 
rates. Non-identified participants are defined as having no OBQI plan 
of action submitted, but with a 3-bar OBQI report for the reporting 
period ending in the 24th month of the contract. This definition 
recognizes the dynamic nature of the home health industry, and counts 
only agencies with sufficient caseload during the 24 months included in 
the 3-bar report. We believe that this definition provides QIOs with 
the best opportunity to successfully pass the evaluation, while 
including all agencies operating with a sufficient caseload during a 
large part of the SOW.
    Comment: One commenter stated that many of the Task 1c hospital 
indicators will have a small number in the denominator. The commenter 
stated that by collecting the same number of cases for all States, the 
precision and confidence interval is much smaller for a large State, 
thereby making the evaluation of the QIO less accurate.
    Response: The assumption on the part of the commenter is not 
completely accurate. The evaluation score equally weights the four 
conditions for hospital public reporting (see http://www.cms.hhs.gov/quality/hospital for list of conditions) to provide a more robust 
estimate of quality improvement. Three of the four conditions have 
large enough samples so that sample size (not population size) is the 
primary determinant driving the precision of the estimates. Acute 
Myocardial Infarction measures, one of the four conditions, with 
systematically small samples are weighted accordingly to minimize the 
impact of any unreliable estimates on the overall evaluation. AMI is 
the only one of the four conditions with systematically small samples. 
It is weighted accordingly to minimize the impact of any unreliable 
estimates on the overall evaluation.
    Comment: One commenter stated that the Task 2b evaluation should 
not be considered under the quantitative evaluation criteria. The 
commenter stated that the largest weighted criterion for this task is 
related to the Reporting Hospital Quality Data for Annual Payment 
Update (RHQDAPU), which does not have a quantitative measurement.
    Response: The RHQDAPU criterion for this subtask is dichotomous in 
nature and requires that QIOs contact all hospitals in their State and 
assist them in their data submission into the Standard Data Processing 
System Clinical Warehouse. QIOs must also document their communication 
and assistance with all hospitals, participating and non-participating. 
Although this task does involve some activities that may be evaluated 
in a qualitative manner, the majority of the activities are 
quantitative in nature. Therefore, we have chosen to evaluate this task 
quantitatively.
    Comment: One commenter expressed concern over the lateness of data 
for Task 1d. The commenter believes that this has made it nearly 
impossible to assess the effectiveness of the QIO interventions, or to 
identify other areas for intervention.
    Response: We recognize that time lags can hinder the QIO's 
technical assistance to providers in the outpatient setting. We have 
set the baseline period to allow QIOs to work with providers during the 
transition period between SOWs. Much of this work is reflected in the 
next SOW's evaluation results. The relative stability of QIOs in their 
States lessens the impact of the time lag.
    Comment: One commenter suggested that we change the evaluation 
criteria in the J-7 for Task 1e to make them the same as the evaluation 
criteria that were originally developed for their QIO's improvement 
project.
    Response: We assume the commenter is referring to the use of sub-
county targeting in the evaluation of this Task. We have already 
modified the evaluation on this Task to allow sub-county targeting. 
This modification to the evaluation was approved by the Project 
Officers in the beginning of the SOW. We do not anticipate any further 
changes at this point.
    Comment: One commenter suggested that Task 3 activities be elevated 
to a higher position in the SOW. This commenter believes the current 
Task 3 should be Task 1 or Task 2 to increase its importance in the 
contract.
    Response: We agree that all of the Tasks performed by the QIOs are 
important to foster quality improvement in the health care delivered to 
Medicare beneficiaries. The evaluation criteria reflect this belief. 
Task 3 comprises 3 out of 12 subtasks evaluated by us. QIOs must 
successfully perform Task 3 work in order to be granted non-competitive 
contract renewal. We believe that the stringent evaluation criteria in 
place for this task reflect the importance of the work.
    Comment: One commenter asked about the provider satisfaction survey 
and how we plan to use the survey if the QIO does not have a sufficient 
sample size.
    Response: Identifying opportunities for improvement is part of a 
quality improvement feedback cycle. We believe that the results of the 
satisfaction survey are useful to QIOs in identifying quality 
improvement opportunities. CMS and its statistical contractor have 
provided all QIOs with detailed information about their satisfaction 
survey results. The

[[Page 42336]]

statistical contractor will also write a national analysis of the 
survey results to identify opportunities for QIO program improvement as 
a whole. In the few instances with insufficient sample size, we use the 
actual satisfaction rate to evaluate QIO performance. However, we grant 
QIOs a passing evaluation score when the overall evaluation status 
(that is, pass vs. fail) is sensitive to this potentially unreliable 
rate. Usually this rate does not affect a QIO's overall evaluation 
status on a particular subtask, since its relative weight is small in a 
subtask's evaluation.
    Comment: One commenter stated that, with the development of the 
Excel spreadsheet to evaluate the qualitative tasks, these tasks are no 
longer qualitative. They are now being evaluated in a quantitative way.
    Response: The Excel tool allows Project Officers to subjectively 
evaluate QIO performance in the qualitative tasks. It was developed in 
response to concerns from QIOs about inter-region variation in the 6th 
SOW. It uses the same evaluation criterion provided in the J-7, and is 
not intended to make the evaluation quantitative in nature. Rather, it 
gives some consistency to the subjective review by the Project 
Officers. We agree that this tool should be provided to QIOs as early 
as possible in the contract cycle. We will strive to provide this tool 
to the QIOs as early as possible for the 8th SOW.
    Comment: One commenter stated that a great deal of effort was put 
into the National Voluntary Hospital Reporting Initiative (NVHRI), but 
this effort was not included in the evaluation criteria.
    Response: We appreciate the fact that the NVHRI did require some 
additional effort on the part of the hospitals. However, participation 
could not be included in the evaluation criteria because this was a 
voluntary program on the part of hospitals. The voluntary nature of the 
program requires a different approach by the QIO than is required by 
the other subtasks and deliverables of the contract.
    Comment: One commenter stated that for those States with 100 
percent participation in hospital public reporting, the Hospital 
Generated Data (HGD) Survey is redundant. The commenter stated that the 
same information may be obtained through both sources.
    Response: We have been careful to avoid redundant activities for 
both providers and QIOs. The HGD Survey does not determine if a 
hospital is a reporting hospital. Instead, it assesses the hospital's 
ability to collect data. Therefore both the survey and the actual 
hospital reporting are necessary and provide different information to 
us.
    Comment: One commenter questioned the evaluation criteria for Task 
3b. In the J-7, the term ``reliability'' is used. The guidance document 
states that the QIO will be evaluated based on both ``reliability'' and 
``validity of review.'' This commenter also requested clarification as 
to why Tasks 3a and 3b require reliability while Task 3c does not 
require validity for evaluation.
    Response: The reliability of the review is the primary criterion 
for evaluating this component of the task. We will ensure consistency 
in documents released for the 8th SOW. The evaluation criteria were 
chosen for each subtask in Task 3 based on the appropriateness for the 
task.
    Comment: One commenter expressed concern over using Medicare 
physician billing as the method to measure the rate of statewide and 
identified participants' improvement in quality care measures for Task 
1d.
    Response: We are investigating this method of measuring improvement 
for the Round 1 evaluations, and have so far found nothing large-scale 
or systematic that would alter evaluation results for Task 1d. We 
believe that the evaluation measures are relatively stable and reliable 
estimates, and that billing issues as a whole do not contribute 
significant bias to these estimates. We understand the limitations of 
using billing information to estimate quality improvement, and are 
working to minimize its impact by identifying these problems and 
reporting questionable billing issues to the appropriate parties.
    We are adopting the provisions of the notice with comment as final.

IV. Executive Order 12866 Statement

    In accordance with the provisions of Executive Order 12866, this 
notice with comment period was not reviewed by the Office of Management 
and Budget.

    Authority: Section 1153 of the Social Security Act (42 U.S.C. 
1320c-2).

(Catalog of Federal Domestic Assistance Program No. 93.774, 
Medicare--Supplementary Medical Insurance Program)

    Dated: March 14, 2005.
Mark B. McClellan,
Administrator, Centers for Medicare & Medicaid Services.
[FR Doc. 05-14505 Filed 7-21-05; 8:45 am]
BILLING CODE 4120-01-P