Defense Health Care: Appointment Timeliness Goals Not Met; Measurement
Tools Need Improvement (Letter Report, 09/30/1999, GAO/HEHS-99-168).

Pursuant to a congressional request, GAO reviewed the Department of
Defense's (DOD) administration of its health maintenance organization,
TRICARE Prime, focusing on: (1) DOD's performance in scheduling
appointments; and (2) possible reasons why Prime enrollees might not
obtain appointments within the appointment timeliness goals.

GAO noted that: (1) after correcting definitional discrepancies in DOD
data, GAO found DOD has not achieved its goal of scheduling 98 percent
of acute and routine appointments within the timeliness standards it
established; (2) about 70 percent of appointments for a routine visit at
military treatment facilities (MTF) were scheduled within the standard,
while between 80 and 97 percent of the appointments for acute care,
preventive care, or specialists were scheduled within the relevant
standards; (3) DOD's analysis of appointment timeliness is consistent
with GAO's findings, and DOD has reported that the MTFs' performance has
fallen short of its expectations; (4) there are several reasons why
active duty members and other enrollees may not obtain appointments
within the standards; (5) GAO found that about 16 percent of the
appointment slots were given to nonenrolled beneficiaries; (6) DOD
permits nonenrollees to make appointments and obtain care in MTFs
because it believes treating these beneficiaries is necessary to support
medical readiness and training requirements; (7) DOD has made no
analysis, however, of the extent to which this policy adversely affects
the ability of the enrolled population to obtain care and treatment or
the effect of any resulting shortfall on readiness and training; (8)
research by the Congressional Budget Office has shown that instituting a
copayment for care provided in MTFs could reduce demand for care and
improve appointment timeliness by freeing up appointments for
active-duty members and Prime enrollees; (9) DOD's data tools are
inadequate for measuring appointment timeliness against the access
standards; (10) survey weaknesses include reliance on the beneficiaries'
ability to correctly recall details of the appointments, a low response
rate, and no analysis of the beneficiaries who do not respond--all of
which affect the accuracy of the information on how well appointment
standards were met; (11) DOD also has several efforts under way to
improve the data contained in the Customer Satisfaction Survey and
Composite Health Care System (CHCS) appointment scheduling system; and
(12) once implemented, CHCS promises to become a good source of the
appointment timeliness information DOD needs to effectively manage and
monitor access to care.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-99-168
     TITLE:  Defense Health Care: Appointment Timeliness Goals Not Met;
	     Measurement Tools Need Improvement
      DATE:  09/30/1999
   SUBJECT:  Customer service
	     Health services administration
	     Health care programs
	     Performance measures
	     Military personnel
	     Health maintenance organizations
IDENTIFIER:  DOD TRICARE Program
	     DOD Composite Health Care System
	     DOD TRICARE Prime Program
	     DOD Customer Satisfaction Survey

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

Cover
================================================================ COVER

Report to the Chairman and Ranking Minority Member, Subcommittee on
Military Personnel, Committee on Armed Services, House of
Representatives

September 1999

DEFENSE HEALTH CARE - APPOINTMENT
TIMELINESS GOALS NOT MET;
MEASUREMENT TOOLS NEED IMPROVEMENT

GAO/HEHS-99-168

Access to Defense Health Care

(101613)

Abbreviations
=============================================================== ABBREV

  CBO - Congressional Budget Office
  CHCS - Composite Health Care System
  DOD - Department of Defense
  MTF - military treatment facility

Letter
=============================================================== LETTER

B-279619

September 30, 1999

The Honorable Steve Buyer
Chairman
The Honorable Neil Abercrombie
Ranking Minority Member
Subcommittee on Military Personnel
Committee on Armed Services
House of Representatives

Over the past decade, the Department of Defense (DOD) has faced the
same challenges in delivering health care to its beneficiaries as the
nation's health care system has for the general population, including
increasing costs and uneven access to care.  In 1993, after years of
demonstration programs designed to explore options to manage the
delivery of health care more effectively, DOD restructured its health
care system into TRICARE, its managed care program.  Today, about 8.2
million active-duty personnel, their dependents, and retirees are
eligible to receive care in this $15.6 billion-per-year health-care
system.  Care for eligible beneficiaries is provided mostly in
military treatment facilities (MTFs), supplemented by networks of
contracted civilian providers.  To help ensure timely access to care,
TRICARE established appointment timeliness standards and goals
similar to those of private health plans for the beneficiaries who
choose to enroll in TRICARE's health maintenance organization option,
called Prime. 

While TRICARE was designed in part to improve beneficiaries' access
to health care, beneficiaries have complained about the difficulties
they encounter obtaining care, including the length of time needed to
get an appointment.  As you requested, this report provides
information on DOD's performance in scheduling appointments, and
possible reasons why Prime enrollees might not obtain appointments
within the appointment timeliness goals.  We also provide information
on improvements needed to DOD's measurement tools.  We conducted our
work between April 1998 and June 1999 in accordance with generally
accepted government auditing standards.  See appendix I for our scope
and methodology. 

   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

After correcting definitional discrepancies in DOD data, we found DOD
has not achieved its goal of scheduling 98 percent of acute and
routine appointments within the timeliness standards it established. 
About 70 percent of appointments for a routine visit at MTFs were
scheduled within the standard, while between 80 and 97 percent of
appointments for acute care, preventive care, or specialists were
scheduled within the relevant standards.  DOD's analysis of
appointment timeliness is consistent with our findings, and the
Department has reported that the MTFs' performance has fallen short
of its expectations. 

There are several reasons why active duty members and other enrollees
may not obtain appointments within the standards.  For example, Prime
beneficiaries sometimes request an appointment date later than the
one offered that was within the standard, although DOD does not have
the data needed to identify the actual number of these requests. 
Another factor is the extent to which MTFs provide care to
nonenrolled beneficiaries.  We found that about 16 percent of the
appointment slots were given to nonenrolled beneficiaries.  DOD
permits nonenrollees, including retirees over age 65, to make
appointments and obtain care in MTFs because it believes treating
these beneficiaries is necessary to support medical readiness and
training requirements.  DOD has made no analysis, however, of the
extent to which this policy adversely affects the ability of the
enrolled population to obtain care and treatment or the effect of any
resulting shortfall on readiness and training.  Another factor
affecting appointment availability is that military beneficiaries
traditionally utilize health care at a higher rate than do
private-sector beneficiaries.  Research by the Congressional Budget
Office (CBO) has shown that instituting a copayment for care provided
in MTFs could reduce demand for care and improve appointment
timeliness by freeing up appointments for active-duty members and
other Prime enrollees. 

As currently configured, DOD's data tools--its Customer Satisfaction
Survey and Composite Health Care System (CHCS) appointment scheduling
system--are inadequate for measuring appointment timeliness against
the access standards.  Survey weaknesses include reliance on the
beneficiaries' ability to correctly recall details of the
appointments, a low response rate, and no analysis of the
beneficiaries who do not respond--all of which affect the accuracy of
the information on how well appointment standards were met.  CHCS
also has weaknesses.  In particular, the appointment names used in
the MTF's appointment scheduling system do not directly relate to the
access standards.  Although DOD has some efforts under way to improve
its Survey, the efforts will not overcome its inherent weaknesses,
such as its reliance on beneficiary recall.  DOD also has several
efforts under way to improve the data contained in the CHCS
appointment scheduling system, including standardizing the
appointment names across the military health-care system and
associating them with the timeliness standards.  Once implemented,
CHCS promises to become a good source of the appointment timeliness
information DOD needs to effectively manage and monitor access to
care.  This report makes recommendations to the Secretary of Defense
to improve appointment timeliness measurement and access to care for
active-duty members and other Prime enrollees. 

   BACKGROUND
------------------------------------------------------------ Letter :2

DOD's primary medical mission is to maintain the health of 1.6
million active-duty service personnel and to provide health care to
them during military operations.  DOD additionally offers health care
to 6.6 million nonactive-duty beneficiaries, including dependents of
active-duty personnel, military retirees, and dependents of retirees. 
Under TRICARE, most care is provided in MTFs worldwide and is
supplemented by civilian providers.  TRICARE is a triple-option
health benefit program designed to give beneficiaries a choice among
a health maintenance organization (Prime), a preferred provider
organization (Extra), and a fee-for-service benefit (Standard).\1
TRICARE Prime is the only option for which beneficiaries must enroll;
active-duty members are automatically enrolled in Prime.  Active-duty
family members and retirees and their dependents under age 65 are
also eligible to enroll in Prime.  Retirees and their dependents and
survivors over age 65 are not eligible to enroll in Prime, but can
still obtain care in MTFs if space or resources are available. 
Beneficiaries can also obtain care from civilian providers. 
Beneficiaries who obtain care within the MTFs, including Prime
enrollees, pay nothing for their outpatient visits.  However,
beneficiaries obtaining care from civilian providers are subject to
out-of-pocket costs ranging from 25 percent of the allowable charge
for a TRICARE Standard office visit to a copayment of $6 or $12 for a
Prime enrollee visiting a provider outside the MTF but in the TRICARE
network.\2

Under section 712 of the National Defense Authorization Act for
Fiscal Year 1996 (P.L.  104-106), DOD was required to establish
priorities for accessing care within MTFs.\3 Under DOD's implementing
policy, active-duty personnel have highest priority, followed by
active-duty family members enrolled in Prime; retirees, their family
members, and survivors enrolled in Prime; nonenrolled active-duty
family members; and nonenrolled retirees, their family members, and
survivors.  In addition, DOD policy specifies that MTF commanders
have the discretion to grant exceptions to the access priority rules
for various reasons, such as giving groups or individuals higher
priority to meet requirements of graduate medical education programs. 

To better ensure timely access to health care, DOD established
appointment timeliness standards for Prime enrollees similar to the
standards used in private-sector managed care programs.  DOD's
standards, which apply to MTFs and the civilian network, established
the following maximum wait times between the day a Prime enrollee
requests an appointment with his or her primary-care physician and
the actual date of the visit: 

  -- 1 day for acute illness care, defined as visits requiring
     physician intervention and urgent in nature;

  -- 1 week for routine visits, defined as requiring physician
     intervention but nonurgent in nature;

  -- 4 weeks for well visits, defined as health maintenance and
     prevention, and nonurgent in nature; and

  -- 4 weeks for specialty care referrals from a primary-care
     physician to a specialist. 

In June 1998, DOD established a goal that at least 98 percent of
acute and routine primary-care appointments for Prime enrollees
should be scheduled within the time allowed by the standards.  In
March 1999, DOD lowered its 98-percent goal to what DOD considers a
more achievable goal of 90 percent because most of the MTFs failed to
meet the 98-percent goal.  According to a DOD official, lowering the
target to 90 percent provides more opportunity for MTFs to achieve
DOD's established goal. 

Section 713 of the Strom Thurmond National Defense Authorization Act
for Fiscal Year 1999 (P.L.  105-261) established requirements for DOD
to collect data on the timeliness of appointments in order to measure
performance in meeting the primary-care access standards established
under TRICARE.\4 This requirement is consistent with the Government
Performance and Results Act of 1993, which requires agencies to
define their missions clearly, set goals, measure performance, and
report on their accomplishments.  DOD uses information from a
Customer Satisfaction Survey to meet this legislative requirement. 
The Survey asks a sample of patients a number of questions about a
specific visit with a particular medical provider in an MTF,
including the severity of the need for the visit (such as whether the
visit was urgent or routine), the number of days between requesting
the appointment and the actual appointment date, and their
satisfaction with the care they received.  DOD aggregates the results
of selected questions as a measure of how well or poorly MTFs as a
group performed in meeting the access standards.  Information on
appointment timeliness is also contained in the
appointment-scheduling module of the CHCS system.  CHCS is considered
the primary health-care information system of the military
health-care system, and is used by all MTFs to capture patient
demographic information, schedule appointments, and to order
prescriptions and ancillary services.  It also contains information
on the timeliness of scheduled appointments for virtually all clinics
in the MTFs. 

--------------------
\1 DOD previously provided health care under the Civilian Health and
Medical Program of the Uniformed Services, a fee-for-service program. 

\2 Dependents of lower-rank active-duty members pay $6 for an
outpatient visit, while dependents of higher-rank active-duty
members, and retirees and their dependents and survivors, pay $12. 

\3 10 U.S.C.  section 1097(c). 

\4 10 U.S.C 1073. 

   DOD EXPERIENCING DIFFICULTY
   ACHIEVING APPOINTMENT
   TIMELINESS GOALS
------------------------------------------------------------ Letter :3

Available data do not permit us or other analysts to precisely
measure the extent to which DOD is meeting its access standards. 
However, after correcting the DOD data for definitional
discrepancies, we were able to develop an assessment of appointment
timeliness.  Our analysis shows that appointments obtained by Prime
enrollees, including active-duty members, were not always scheduled
within the timeliness standards.  Furthermore, about the same
percentage of appointments for nonenrolled beneficiaries were
scheduled within the standards as were those for active-duty and
Prime enrollees.  These findings are consistent with DOD's own
analysis, which concluded that its performance in appointment
timeliness has not met its expectations and goals. 

According to DOD officials, there are several reasons why
appointments for active-duty and other Prime beneficiaries are not
scheduled within the standard.  These include beneficiaries turning
down an appointment that was offered to them within the standard, and
nonenrolled beneficiaries being scheduled for appointments that
otherwise would have been available for active-duty and Prime
beneficiaries.  Several options exist to increase the percentage of
appointments scheduled within the standards and improve access for
active-duty and other enrollees.  These options include DOD
conducting an assessment of the extent to which medical readiness and
training needs can be met without treating nonenrolled beneficiaries,
and stricter enforcement of the access to care priorities based on
this assessment.  Also, requiring a copayment for care provided in
the MTFs could reduce the traditionally higher usage of military
health-care (as compared to utilization in private health plans) and
help DOD achieve its appointment timeliness goals. 

      DOD HAS NOT ACHIEVED
      TIMELINESS GOAL FOR
      ACTIVE-DUTY AND PRIME
      ENROLLEES
---------------------------------------------------------- Letter :3.1

Our analysis of Customer Satisfaction Survey and CHCS appointment
data indicates that DOD fell short of its original goal that 98
percent of acute and routine primary-care appointments for Prime
enrollees, including active-duty members, be scheduled within the
period of time set in the standards.  For example, both data sources
show that only about 70 percent of routine appointments for Prime
enrollees were scheduled within the required 1 week of the request
for the appointment.  While the 98-percent goal was in place for the
time period we analyzed, the performance for scheduling acute and
routine appointments was even below DOD's lowered goal of 90 percent. 
Table 1 summarizes DOD's appointment standards, goals, and the
percentage of appointments within the standards for active-duty
members and other Prime enrollees. 

                                          Table 1
                          
                           DOD Appointment Scheduling Standards,
                          Goals, and Appointments Scheduled Within
                            Standards for Active-Duty and Other
                                      Prime Enrollees

                                                                       Prime enrollee
                                                                   appointments scheduled
                                               Active-duty            within standard
                                          appointments scheduled  (excluding active-duty)
                                           within standard (%)              (%)
                                          ----------------------  ------------------------
                            Goal for
               Appointment  appointments
               scheduling   to be           Customer                  Customer
               standard     scheduled     Satisfacti              Satisfaction
Appointment    for Prime    within         on Survey        CHCS        Survey        CHCS
type           enrollees    standard (%)      data\a      data\b        data\a      data\b
-------------  -----------  ------------  ----------  ----------  ------------  ----------
Primary care   1 day        98                    84          91            80          92
acute

Primary care   1 week       98                    81          81            71          65
routine

Primary care   4 weeks      No goal               96          91            97          81
well

Specialty      4 weeks      No goal               94          96            94          91
referral
------------------------------------------------------------------------------------------
\a Data for 117 MTFs with clinics that had more than 200 visits per
month for the 5-month period of May 1, 1998, to September 30, 1998. 
Sampling errors are no greater than +/-3 percentage points. 

\b Data for appointments scheduled between October 1, 1997, and
September 30, 1998, at five MTFs and between January 1, 1998, and
December 31, 1998, at one MTF. 

Source:  GAO analysis of DOD data. 

      TIMELINESS FOR PRIME
      ENROLLEES SIMILAR TO THAT
      FOR NONENROLLED
      BENEFICIARIES
---------------------------------------------------------- Letter :3.2

Among the beneficiaries who obtained appointments, the percentage of
appointments scheduled for active-duty and other Prime enrollees
(those with the highest priority) within the standards was similar to
the percentage of appointments within the standards for nonenrolled
beneficiaries (who have the lowest priority).  For example, the
Customer Satisfaction Survey indicates that the percentage of acute
and well primary-care appointments scheduled for active-duty members
within the standards (84 and 96 percent, respectively) was similar to
the percentage for nonenrolled appointments (81 percent and 95
percent, respectively).  The CHCS data show that the appointment
timeliness for other enrollees and the nonenrolled for all
appointment types was also similar.  Table 2 summarizes the
appointment timeliness for active-duty members, other Prime
enrollees, and nonenrolled beneficiaries. 

                                         Table 2
                         
                           Comparison of Appointments Scheduled
                         Within Standards for Active-Duty, Other
                             Prime Enrollees, and Nonenrolled
                                      Beneficiaries

                                            Prime enrollee
                                        appointments scheduled
             Active-duty appointments      within standard       Nonenrolled appointments
                 scheduled within      (excluding active-duty)       scheduled within
                   standard (%)                  (%)                   standard (%)
             ------------------------  ------------------------  ------------------------
                 Customer                  Customer                  Customer
             Satisfaction              Satisfaction              Satisfaction
Appointment        Survey        CHCS        Survey        CHCS        Survey        CHCS
type               data\a      data\b        data\a      data\b        data\a      data\b
-----------  ------------  ----------  ------------  ----------  ------------  ----------
Primary                84          91            80          92            81          88
 care acute
Primary                81          81            71          65            69          69
 care
 routine
Primary                96          91            97          81            95          78
 care well
Specialty              94          96            94          91            90          93
 referral
-----------------------------------------------------------------------------------------
\a Data for 117 MTFs with clinics that had more than 200 visits per
month for the 5-month period of May 1, 1998, to September 30, 1998. 
Sampling errors are no greater than +/-3 percentage points. 

\b Data for appointments scheduled between October 1, 1997, and
September 30, 1998, at five MTFs and between January 1, 1998, and
December 31, 1998, at one MTF. 

Source:  GAO analysis of DOD data. 

While the data show similarities in the timeliness of appointments
for enrolled and nonenrolled beneficiaries, it is important to note
that the majority of the appointments--84 percent--were for enrolled
beneficiaries.  Also, there are no data showing the number of
nonenrolled beneficiaries who were unable to obtain an appointment. 

      DOD REPORTS PERFORMANCE IN
      MEETING ACCESS STANDARDS HAS
      NOT MET ITS EXPECTATIONS
---------------------------------------------------------- Letter :3.3

In October 1998, DOD reported that it had a serious problem providing
timely access to care, based on its analysis of Customer Satisfaction
Survey data for the May to July 1998 period.  According to DOD, less
than 15 percent of the 115 MTFs included in its analysis were able to
schedule acute appointments within the standard, and DOD
characterized the performance of many of the MTFs as "dismal." Over
the next 5 months, DOD said that although it had noticed some
improvements, the achievement of the access standards continued to
fall below its goal.  In March 1999, the Executive Director of the
TRICARE Management Activity stated that access must improve and
tasked the Surgeons General and regional TRICARE management offices
to work with MTFs to identify access problems and make needed
improvements. 

      REASONS WHY APPOINTMENTS ARE
      NOT SCHEDULED WITHIN THE
      STANDARDS
---------------------------------------------------------- Letter :3.4

According to DOD, appointments may not be scheduled within the
standards either because the beneficiary requests a later appointment
for personal convenience, or because there are no appointment slots
available.  DOD does not have data to identify the actual number of
personal convenience requests, but is planning some revisions to the
CHCS appointment system to capture information on whether the
beneficiary accepts the first offered appointment or requests a later
one. 

Appointment availability at the MTFs is also affected by the extent
to which care is provided to nonenrolled beneficiaries.  Our review
of CHCS appointment data at six MTFs shows that about 16 percent of
the appointments were for beneficiaries who were not enrolled in
TRICARE Prime.\5 According to DOD, providing medical care to other
beneficiaries, including those over age 65, provides medical
proficiency training that supports military medical readiness and
training requirements.  DOD has made no analysis of the extent to
which providing care to these beneficiaries adversely affects the
ability of the enrolled population to obtain care or the effect of
any resulting shortfall on readiness and training. 

Another factor that affects the availability of appointments for
active-duty and other Prime enrollees is the extent to which care in
the MTFs is overutilized by beneficiaries.  Studies have shown that
the per-capita utilization of DOD health care services by military
beneficiaries has historically been much higher than in civilian
health plans, due in part to the lack of a cost-sharing requirement
in MTFs.  As we have previously reported, research has shown that the
lack of a cost-sharing requirement leads to a higher utilization of
health care.\6 CBO has reported that sharing costs with beneficiaries
reduces health-care utilization.  In its April 1999 report, CBO
concluded that requiring a copayment from beneficiaries who use MTFs
would help curb excessive use.\7 Furthermore, according to CBO,
concerns that increasing cost-sharing requirements could discourage
beneficiaries from seeking necessary care are not well founded,
especially for the military health-care beneficiaries.  CBO reports
that cost-sharing requirements do not prevent beneficiaries at ages
and income levels typical of military beneficiaries from seeking
needed care. 

--------------------
\5 Although we obtained appointment data from eight MTFs, we were
only able to use data from six due to limitations and discrepancies
in the data that could not be corrected. 

\6 Defense Health Care:  Challenges Facing DOD in Implementing
Nationwide Managed Care (GAO/T-HEHS-94-145, Apr.  19, 1994), and
Addressing the Deficit:  Budgetary Implications of Selected GAO Work
for Fiscal Year 1998 (GAO/OCG-97-2, Mar.  14, 1997). 

\7 Maintaining Budgetary Discipline:  Spending and Revenue Options,
CBO (Washington, D.C., Apr.  1999). 

      OPTIONS TO IMPROVE
      APPOINTMENT TIMELINESS AND
      ACCESS
---------------------------------------------------------- Letter :3.5

Several options could improve appointment timeliness for active-duty
and other Prime enrollees.  One is to more rigorously implement the
access priorities.  Some of the MTFs we visited had procedures to
give appointment priority to active-duty and other enrollees, such as
specifying certain times of day for active-duty and Prime enrollees
to request appointments, after which appointments were available for
all beneficiaries, whether enrolled or not.  However, once
beneficiaries are booked into appointments, the appointment priority
no longer exists.  One option is to "bump" a nonenrollee who has an
appointment when an enrolled beneficiary needs an appointment and
none is available within the required time frame.  Second, if each
MTF identified what percentage of the care provided to nonenrollees
was necessary to achieve their medical readiness and training
requirements, the rest of the care could be reallocated to
active-duty and other enrollees to improve their access.  Third,
establishing a beneficiary copayment for care in the MTF could reduce
the demand for care in the MTF and free up more appointments for
active-duty members and other Prime enrollees.  A standard practice
used by commercial managed care plans to bring about more appropriate
utilization is requiring enrolled beneficiaries to pay a copayment
for care.  Commercial plan copayments for outpatient physician visits
range from about $5 to $15, with most beneficiaries paying $10 per
visit.  DOD's civilian copayment requirement of $6 or $12 per visit
is consistent with commercial plans. 

While these options are intended to improve appointment timeliness
and availability for enrolled beneficiaries, it is possible that the
options may cause some nonenrolled beneficiaries to seek care
elsewhere and experience higher out-of-pocket costs.  The options may
also affect the timeliness of the care they receive.  However, we did
not evaluate the extent to which this might occur. 

   WEAKNESSES IN DATA TOOLS
   PREVENT ACCURATE ASSESSMENT OF
   APPOINTMENT TIMELINESS
------------------------------------------------------------ Letter :4

While DOD has been measuring appointment timeliness, the tools it
uses have several weaknesses that limit their usefulness in providing
management information on the extent to which beneficiaries are
obtaining appointments within the prescribed standards.  For example,
problems associated with the design and administration of DOD's
Customer Satisfaction Survey, such as the accuracy of
beneficiary-reported data and the small number of visits included in
the sample, prevent using the Survey to measure MTFs' performance
against the appointment timeliness standards.  CHCS appointment
system data can provide information on appointment timeliness at each
MTF, but cannot be used to compare the data against the standards or
across the military health care system unless certain modifications
are made.  Although DOD has efforts under way to improve the Survey,
its reliance on beneficiaries' recall of their appointment experience
is an inherent weakness that fundamentally limits the Survey's
usefulness in this area.  However, the efforts under way to improve
CHCS should address the weaknesses and make CHCS a good source of
data to measure and monitor MTF performance in scheduling
appointments within the standards. 

      WEAKNESSES ASSOCIATED WITH
      SURVEY DATA AFFECT
      USEFULNESS
---------------------------------------------------------- Letter :4.1

While DOD's Customer Satisfaction Survey provides information on how
beneficiaries perceive their health care experiences, it has
weaknesses when used for measuring the performance of MTFs in meeting
access standards.  For example, the quality of data is entirely
dependent on the beneficiary's ability to remember the number of days
it took to get a specific appointment.  However, because
beneficiaries can receive the survey up to 45 days after the
appointment, they may have difficulty accurately recalling their
experience, thus calling into question the validity of the Survey
results. 

Another weakness is that DOD relies on the respondents to correctly
classify their appointment types in their survey responses.  DOD uses
the respondents' classifications to determine which access standard
should be related to the appointments.  The Survey asks each
respondent to classify the purpose of his or her visit as one of the
following: 

  -- Care for illness or injury which the patient felt required him
     or her to see a doctor right away;

  -- Routine care for a nonurgent condition;

  -- Well-patient visit for preventive care (checkup); or

  -- Specialty care, referral visit. 

However, our discussions with beneficiaries revealed that they were
uncertain about the correct category for their visits.  Beneficiaries
did not understand the difference between a routine visit for a
nonurgent condition and a well-patient visit for preventive care, and
were unsure about how to categorize a follow-up visit with either a
primary-care physician or specialist.  Also, the design of the
question intends that primary-care visits would be identified by
responses to the first three categories and specialty care by
selecting the fourth choice.  Beneficiaries said they would select
the first option if they felt that they needed care from a specialist
right away.  Our analysis of Survey responses confirmed the potential
for this error.  We found that about two-thirds of the responses from
beneficiaries who received care in a specialty clinic marked one of
the first three categories, and thus were misclassified as primary
care. 

Even if beneficiaries were able to interpret the questions and report
their experiences accurately, the sample size of the Survey is too
small to provide precise estimates of clinic performance.  Each
month, DOD randomly selects 35 visits from each clinic that receives
at least 200 visits per month.  Given the survey response rate of 40
percent, this sample size yields about 14 responses per month for
each clinic sampled.  Even if data were aggregated and analyzed every
6 months, a sample size of only around 85 for the period could be
expected, which would provide information only on very large changes
in performance at the clinic level.  For example, an increase in the
appointments scheduled within the standards from 70 percent to 80
percent would not represent a statistically significant change based
on a sample size of 85. 

The response rate of the Survey also calls into question the validity
of the Survey results.  While the Survey results provide information
on those who responded, DOD knows little about the experiences of the
60 percent or more of surveyed beneficiaries who did not respond.\8
Without conducting a nonrespondent analysis, DOD cannot determine the
extent to which their health-care experiences were similar to or
different from experiences of patients who did complete the survey.\9
Because the group of nonrespondents is so large, their experiences,
if different from the experiences of the respondents, could
dramatically change the survey results. 

In regard to measuring civilian provider appointment timeliness, DOD
is developing a survey modeled after the Customer Satisfaction
Survey.  However, the limitations of the MTF Customer Satisfaction
Survey would also apply to the civilian survey.  Thus, while the
civilian survey might provide some general indications about
beneficiaries' experiences with civilian providers, it would not
capture precise data needed to assess how well the access standards
are being met in the civilian network. 

--------------------
\8 A DOD official involved with the Survey told us that the response
rate has historically been 40 percent or less. 

\9 One way to assess the extent to which nonrespondents differ from
respondents is to conduct a nonresponse analysis.  A nonresponse
analysis is a technique used to determine the difference between
those who responded and those who did not respond to a survey, and
the extent to which the respondents represent the overall population. 
A nonresponse analysis for a mail survey is usually conducted by
administering the survey over the telephone. 

      CHCS APPOINTMENT DATA NEED
      MODIFICATION TO BE A VIABLE
      MEASUREMENT TOOL
---------------------------------------------------------- Letter :4.2

Appointment data taken directly from the CHCS appointment scheduling
system used by all MTFs potentially could be the best data DOD has
available to measure the performance of MTFs in meeting the access
standards.  While CHCS data are not vulnerable to the limitations
inherent in the Customer Satisfaction Survey, the CHCS has other
shortcomings that limit its current usefulness as a tool to measure
appointment timeliness. 

A critical weakness of the CHCS data for appointment-measuring
purposes is that the appointment names used in the MTF's appointment
scheduling system do not directly relate the types of visits to the
standards.  We found that four of the eight MTFs in our study used
appointment names within their scheduling systems that could not be
linked to only one appointment timeliness standard.  For example, at
one MTF the appointment name "PRIME" was used to book acute, routine,
and well primary-care appointments, which are each subject to
different access standards.  At another MTF, the appointment name
"PACU" was used to book acute and routine appointments, while the
name "ROUP" was used to book routine, follow-up, and well
appointments.  In these cases, more than one timeliness standard
would be applicable and the MTF would not know which standard to use
to measure its performance in making timely appointments.  Unless
MTFs link their appointment names to a single standard, they will be
unable to determine the extent to which their appointments are in
compliance with the appointment timeliness standards. 

In addition, the lack of standard appointment names among the MTFs
prevents DOD from consolidating individual facility CHCS data into
regional or systemwide data.  Under DOD's current procedures, each
MTF has the flexibility to design a unique appointment system.  This
practice hampers DOD's ability to collect and monitor appointment
data across the military health-care system.  For example, DOD would
have to know exactly which names were used in every MTF's appointment
system and which of the access standards applied to the appointment
name.  In our review of appointment names in use at eight MTFs, we
found 14 different names for appointments associated with the
timeliness standard for acute appointments, 18 different names for
appointments associated with the timeliness standard for routine
appointments, and 35 different names for appointments associated with
the timeliness standard for well visits.  Even though some of these
MTFs could associate the appointment names they used with the
applicable timeliness standard, the lack of consistent and standard
appointment names across a system of more than 450 MTFs with
potentially thousands of appointment names would make any effort to
collect, monitor, and regularly report on systemwide appointment data
a complex and complicated undertaking. 

      DOD ATTEMPTING TO RESOLVE
      DATA WEAKNESSES
---------------------------------------------------------- Letter :4.3

DOD officials told us they recognize the weaknesses of the Survey and
CHCS data and have some efforts under way or planned to address some
of the weaknesses.  With regard to the Survey, they acknowledged that
beneficiaries are confused when trying to categorize their
appointments, especially because the Survey does not define the
categories.  According to DOD officials, providing some general
definitions with examples of appointments could help beneficiaries
responding to the Survey.  However, they believe it is not possible
to provide sufficient examples to completely eliminate the confusion
and ensure correct categorization, and are not planning any revisions
to that Survey question.  The officials also agreed that memory
recall about the number of days it took to get an appointment was a
concern.  DOD officials said that sending the Survey closer to the
appointment date might improve memory recall, but the administrative
tasks associated with selecting the sample and mailing the Survey
could not be hastened.  Related to the lack of a nonrespondent
analysis, DOD recently decided to conduct the first analysis of
nonrespondents in fiscal year 2000. 

With regard to the CHCS data, DOD has efforts under way or planned
that should address the critical weaknesses that affect the
usefulness of the data in measuring appointment timeliness.  DOD
officials told us that a policy is being developed requiring MTFs to
correlate their primary-care appointments to the three timeliness
standards and to standardize the appointment names across the
military health-care system.  Other enhancements are also planned
that will improve the accuracy of CHCS appointment timeliness data. 
Officials estimate that CHCS data would be reliable for monitoring
and measuring access in MTFs by March 2000 after these changes and
improvements are tested and implemented throughout the military
health-care system.  If successful, DOD could rely on CHCS and cease
using the Survey as a means of measuring compliance with the
timeliness standards at the MTF, regional, and systemwide level. 

   CONCLUSIONS
------------------------------------------------------------ Letter :5

Active-duty and other Prime enrollees have not been able to obtain
appointments within the prescribed timeliness standards to the extent
that DOD expected when it first established goals for TRICARE. 
Moreover, the performance in meeting standards is about the same for
active-duty members, who have the highest priority, and nonenrolled
beneficiaries, who have the lowest priority.  In some cases,
appointments are scheduled outside the standards due to the
beneficiary's request for a later appointment to meet personal needs. 
However, appointments within the standards for enrolled beneficiaries
may not be available because nonenrolled beneficiaries have filled
available appointment slots ahead of them.  Providing care to
nonenrollees, especially those who are eligible to enroll in Prime,
counters the program's intention that eligible beneficiaries enroll,
and reinforces some beneficiaries' view that they can still obtain
care in the MTFs without enrolling. 

There are several options DOD could test to improve the availability
of appointments for active-duty and other enrolled beneficiaries. 
These include more vigorously enforcing systemwide access priorities,
to the extent of giving appointments booked for nonenrollees to
enrolled beneficiaries in need of an appointment within the standard. 
Also, eliminating care to nonenrolled beneficiaries that exceeds
medical training requirements could result in more available
appointments.  Lastly, instituting a copayment in the MTFs could lead
to more appropriate utilization of care in MTFs, thereby opening up
additional appointment slots for enrollees.  While copayments could
help improve appointment timeliness, potential benefits actually go
well beyond this.  Copayments would also serve to equalize the
cost-sharing for all beneficiaries, regardless of whether they
receive care from military or civilian providers, by eliminating the
inherent inequity of providing more generous health benefits to those
who live near an MTF.  It would also allow physicians to refer
beneficiaries to the most appropriate provider--whether military or
civilian--without regard to the financial implications of the
referral for the beneficiary. 

The two data tools that provide information on appointment timeliness
in MTFs have significant weaknesses that affect the accuracy and
sufficiency of the data.  DOD is undertaking efforts to address the
weaknesses of the CHCS appointment system by requiring MTFs to
associate appointment names with the access standards and by
establishing standard appointment names across the system.  Regarding
the Customer Satisfaction Survey, we agree that it provides DOD with
meaningful information on how beneficiaries feel about their
health-care experiences and can be used for this purpose. 
Furthermore, the planned analysis of nonrespondents will further
improve the data.  However, two remaining weaknesses in the
Survey--beneficiary categorization of appointments and reliance on
memory recall--are sufficiently significant to continue to call into
question the validity of the results as a measure of either civilian
providers' or MTFs' performance against the standards.  While we
recognize the challenges DOD faces in obtaining comprehensive
information on civilian providers' performance, accurate and
appropriate data to measure how well MTFs are meeting the standards
can and should be obtained from MTF data sources, not from
beneficiaries.  Therefore, the CHCS system should be the primary data
source for determining MTF compliance with the access standard.  In
our view, it is imperative that DOD implement changes to the CHCS
system as soon as possible so that it can meet its responsibilities
to beneficiaries and more effectively manage access to the MTFs. 

   RECOMMENDATIONS
------------------------------------------------------------ Letter :6

We recommend that the Secretary of Defense direct the Assistant
Secretary of Defense (Health Affairs) to measure and monitor
beneficiaries' access to health care in a more comprehensive and
accurate manner by directing that CHCS be used in lieu of the
Customer Satisfaction Survey to measure compliance with the
appointment timeliness standards in the MTFs and that the necessary
modifications be made to CHCS so that appointment names are linked to
the appropriate access standard and standardized across the military
health-care system.  The Secretary should direct that the results be
reported at all levels--individual facility, service- and
system-wide, and by the various beneficiary categories. 

The Secretary should also direct a test of a policy that appointments
scheduled for nonenrolled beneficiaries are subject to cancellation
if an active-duty member or other Prime enrollee requests care and no
other appointment is available within the access standard.  This test
could be implemented in those MTFs having the greatest difficulty
scheduling active-duty members and other Prime enrollees within the
access standards. 

The Secretary should also test the option of instituting copayments
within the MTFs comparable to those in the civilian networks to help
bring about more appropriate utilization of military care and thus
free up appointment space. 

   AGENCY COMMENTS
------------------------------------------------------------ Letter :7

We provided a draft of this report to DOD for review and comment, but
DOD has not provided comments. 

---------------------------------------------------------- Letter :7.1

As agreed with your offices, we are sending copies of this report to
the Honorable William C.  Cohen, Secretary of Defense, and will make
copies available to others upon request.  Please contact me on (202)
512-7111 or Michael T.  Blair, Jr., Assistant Director, on (404)
679-1944 if you or your staff have any questions.  Other major
contributors to this report are listed in appendix II. 

Stephen P.  Backhus, Director
Veterans' Affairs and Military Health
 Care Issues

SCOPE AND METHODOLOGY
=========================================================== Appendix I

SCOPE

To obtain information on the Department of Defense's (DOD) access
policies and measures, we met with officials in the Office of the
Assistant Secretary of Defense (Health Affairs) and the TRICARE
Management Activity, who are responsible for managing the military
health-care program.  We also spoke with staff of the three Services'
Surgeons General and three of the TRICARE managed care support
contractors.  We discussed local access policies and appointment
procedures with officials at 15 military treatment facilities (MTF)
and visited 12 of the facilities.  We also reviewed DOD standards for
primary- and specialty-care appointment timeliness and DOD's policy
on priority for access to care in MTFs. 

We researched the access standards used by commercial managed care
plans and how they measure their performance against these standards. 
We reviewed accreditation standards related to access to care from
two health-care industry accreditation bodies--the Joint Commission
on Accreditation of Healthcare Organizations and the National
Committee for Quality Assurance.  We also gathered information on
appointment timeliness and access standards used by individual
private-sector health care plans, as well as cost-sharing
requirements. 

METHODOLOGY

For the purpose of this study, we defined access as appointment
timeliness--measuring the number of elapsed days between the date
that the beneficiary requests an appointment and the scheduled
appointment date.  We selected this measure because (1) it was the
access measure for which DOD had established criteria or standards
against which its performance could be measured, and (2) appointment
data were available throughout the military health care system from
the Customer Satisfaction Survey and the Composite Health Care System
(CHCS).  However, because of limitations in DOD's data from both
sources, we could not use the data as they existed in DOD's systems,
and designed a methodology and analysis approach (discussed below) to
minimize the effect of the limitations. 

      CUSTOMER SATISFACTION SURVEY
------------------------------------------------------- Appendix I:0.1

We made several adjustments to the Survey data to minimize the
weaknesses and correct discrepancies.  Because we had concerns about
whether beneficiaries had correctly classified their visits, we did
not use their classification from the Survey.  Instead, we used the
sample selection data that provided information on the clinics the
beneficiaries visited.  From these data, we identified primary-care
visits, which we defined as a visit to one of the four clinics DOD
now considers to be a primary-care clinic throughout the military
health care system--family practice, primary care, flight medicine,
or pediatrics.  In consultation and agreement with DOD officials
responsible for administering the Survey, we considered all
appointments not associated with one of the four primary clinics to
be a specialty appointment.  We did not include information on visits
from certain clinics for which the access standards were not
applicable, such as mental health clinics and emergency departments. 
Our analysis covered survey responses for appointments in the 5-month
period from May 1, 1998, to September 30, 1998.  We selected the
beginning date of May 1, 1998, so that our analysis contained only
responses since the survey instrument was revised in May 1998. 
September 30, 1998, was the latest date for which data were available
at the time of our request.  To determine appointment timeliness and
compliance with the primary-care standards, we relied on the
beneficiaries' response to the question asking them how many days it
took to obtain the appointment and compared it to the relevant access
standard determined by the clinic of their visit, as discussed above,
and the beneficiaries' categorization as to the urgency of the visit
(for those determined to be primary-care visits).  In analyzing the
data by beneficiary category and enrollment status, we found and
corrected discrepancies.  We considered all active-duty respondents
as enrolled in Prime, and retirees over age 65 as not enrolled in
Prime, regardless of how they responded to the survey question about
their enrollment status. 

      CHCS APPOINTMENT DATA
------------------------------------------------------- Appendix I:0.2

As a result of our discussions with MTF officials, we confirmed that
the CHCS appointment system could provide the information we needed
to assess the appointment timeliness at the MTF level.  This
information included beneficiary category and enrollment status, the
date the beneficiary requested the appointment, the date of the
scheduled appointment, and appointment type or name.  Our analysis of
CHCS data also confirmed that the key limitation with these data was
determining for each appointment name in the system which appointment
timeliness standard was relevant. 

We asked eight MTFs to provide the appointment names used in their
scheduling system that were subject to each of the access standards
for their primary care.\10

We also asked them to provide the same information on appointment
names for selected specialty clinics.  After obtaining the
information on the appointment name used, we asked each of the eight
MTFs to provide us with 12 months of appointment data for the
identified primary-care and specialty clinics.  Table I.1 shows the
eight MTFs and the clinics for which we obtained data. 

                                        Table I.1
                         
                          Primary and Specialty Care Clinics at
                          Eight MTFs Providing Appointment Data

MTF                     Primary care clinics    Specialty care clinics
----------------------  ----------------------  -----------------------------------------
Fort Benning/Martin     Internal Medicine,      Gynecology, Internal Medicine, Optometry,
Army Community          Family Practice (2      Podiatry, Allergy, Physical Therapy,
Hospital                clinics), Aviation      Orthopedics, Ophthalmology, Dermatology,
                        Medicine                Nutrition, Obstetrics,
                                                Otorhinolaryngology, Urology

Davis-Monthan AFB/      Family Practice (2      Gynecology, Internal Medicine, Optometry,
355\th Medical Group    clinics)                Physical Therapy, General Surgery,
                                                Orthopedics, Dermatology

Fort Hood/Darnall Army  Family Care (3          Gynecology, Internal Medicine, Optometry,
Community Hospital      clinics), Pediatrics    Podiatry, Allergy, Physical Therapy,
                        (3 clinics)             General Surgery, Orthopedics,
                                                Ophthalmology, Dermatology, Nutrition,
                                                Neurology, Obstetrics, Women's Health,
                                                Urology

Naval Hospital Oak      Family Practice (3      Gynecology, Internal Medicine, Optometry,
Harbor                  clinics), Primary Care  Physical Therapy, General Surgery,
                        (3 clinics),            Obstetrics
                        Pediatrics (3
                        clinics), Aviation
                        Medicine (3 clinics)

Mountain Home AFB/      Primary Care (3         Gynecology, Internal Medicine, Optometry,
366\th Medical Group    clinics)                Physical Therapy, General Surgery,
                                                Obstetrics

Wilford Hall Medical    Family Medicine,        Gynecology, Obstetrics, Optometry,
Center                  Internal Medicine,      Podiatry, Allergy, Physical Therapy,
                        Women's Health,         Cardiology, General Surgery,
                        General Pediatrics      Ophthalmology, Dermatology, Nutrition,
                                                Neurology, Otorhinolaryngology,
                                                Orthopedics, Urology

Fort Rucker/Lyster      Aviation Medicine,      Gynecology, Internal Medicine, Optometry,
Army Community          Ambulatory Care,        Physical Therapy, General Surgery,
Hospital                Family Practice         Orthopedics, Ophthalmology, Dermatology

Naval Hospital          Primary Care, Family    Allergy, Dermatology, General Surgery,
Jacksonville            Practice (2 clinics),   Gynecology, Internal Medicine, Neurology,
                        Pediatrics              Nutrition, Ophthalmology, Orthopedics,
                                                Otorhinolaryngology, Physical Therapy,
                                                Urology
-----------------------------------------------------------------------------------------
From the clinic appointment data, we calculated the number of days
between the date the beneficiary requested the appointment and the
date of the scheduled appointment.  We analyzed these data by
different variables, including beneficiary category and whether the
beneficiary was enrolled in TRICARE Prime.  For our analysis, we
assumed that the patient's visit to the provider was in fact for the
type of visit indicated by the appointment name and timeliness
standard.  We could not correct the data to reflect any instances in
which patients were scheduled into appointment types that were
different from the type they requested.  Another discrepancy we found
was that in some cases active-duty personnel were recorded as not
enrolled in Prime, when they are actually considered automatically
enrolled.  We corrected for this by recoding all active-duty as
enrolled in Prime, regardless of the enrollment status field in the
CHCS data.  Similarly, we recoded all retirees over age 65 as not
enrolled despite the enrollment status shown in the data.  We
ultimately had to exclude data from two MTFs--Fort Hood and Wilford
Hall Medical Center--because some of their appointment types were
associated with more than one standard, which precluded comparing the
data against the access standards. 

--------------------
\10 We selected the MTFs to represent the different Services, TRICARE
contractors, areas of the country, and size of medical facility. 

GAO CONTACTS AND STAFF
ACKNOWLEDGMENTS
========================================================== Appendix II

GAO CONTACTS

Michael T.  Blair, Jr., (404) 679-1944
Nancy T.  Toolan, (404) 679-1983

STAFF ACKNOWLEDGMENTS

In addition to those mentioned above, Sylvia D.  Jones, Linda S. 
Lootens, Deborah L.  Edwards, and Beverly Brooks-Hall made key
contributions to this report. 

*** End of document. ***