VA Health Care: Adequacy of Resident Supervision Is Not Assured, 
but Plans Could Improve Oversight (02-JUL-03, GAO-03-625).	 
                                                                 
The Department of Veterans Affairs (VA) provides graduate medical
education (GME) to as many as one-third of U.S. resident	 
physicians, but oversight responsibilities spread across VA's	 
organizational components and multiple affiliated hospitals and  
medical schools could allow supervision problems to go undetected
or uncorrected. GAO was asked to examine VA's procedures for (1) 
monitoring VA medical centers' adherence to VA's requirements for
resident supervision, (2) using evaluations of supervision by GME
accrediting bodies and residents, and (3) using information about
resident supervision drawn from VA's programs for monitoring the 
quality and outcomes of patient care.				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-625 					        
    ACCNO:   A07397						        
  TITLE:     VA Health Care: Adequacy of Resident Supervision Is Not  
Assured, but Plans Could Improve Oversight			 
     DATE:   07/02/2003 
  SUBJECT:   Graduate education 				 
	     Health care personnel				 
	     Internal controls					 
	     Medical education					 
	     Medical schools					 
	     Monitoring 					 
	     Physicians 					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-625

Report to Congressional Requesters

United States General Accounting Office

GAO

July 2003 VA HEALTH CARE Adequacy of Resident Supervision Is Not Assured,
but Plans Could Improve Oversight

GAO- 03- 625

VA cannot assure that the resident physicians who provide care in its
facilities receive adequate supervision because its procedures for
monitoring supervision are insufficient. VA does not know whether medical
centers have adopted VA*s national requirements for supervision of
residents* diagnosis, treatment, or discharge of patients. VA officials
require a review of only one specific requirement that is intended to
ensure availability of supervision when a supervising physician does not
need to be in the operating or procedural suite while a resident performs
a diagnostic or therapeutic procedure. Four of 11 network officials we
interviewed had not conducted this review, and the requirement at one
medical center in one of these four networks was less stringent than VA*s
national requirement. To obtain more complete information about adherence
to its national supervision requirements, VA plans to have external peer
reviewers examine documentation of supervision in patients* medical
records. VA*s plans for

this review have not been finalized. For example, as of May 2003, VA had
not decided whether reviewers would examine records from VA*s new
outpatients. Without records from new patients, reviewers will not be able
to assess documentation of residents* supervision during a veteran*s first
outpatient visit.

To improve its oversight, VA is making efforts to obtain information from
accrediting bodies and residents about the quality of resident
supervision. For example, VA has taken steps to obtain direct access to
letters from accrediting bodies that contain evaluations of the GME
programs in which its medical centers participate. To solicit feedback
from residents, VA implemented a national survey, but was unable to send
this survey to a representative sample of residents from each VA medical
center because it does not have a complete central list of its residents.
VA is taking action to obtain this information.

In addition, VA uses information from its broader programs for monitoring
the quality and outcomes of patient care, such as its patient safety and
surgical quality improvement programs, to identify and correct problems
with resident supervision. Information from these programs has served as
the basis for corrective actions by VA officials. The Department of
Veterans Affairs

(VA) provides graduate medical education (GME) to as many as one- third of
U. S. resident physicians, but oversight responsibilities spread across
VA*s

organizational components and multiple affiliated hospitals and medical
schools could allow supervision problems to go undetected or uncorrected.
GAO

was asked to examine VA*s procedures for (1) monitoring VA medical
centers* adherence to VA*s requirements for resident supervision, (2)
using evaluations

of supervision by GME accrediting bodies and residents, and (3) using
information about resident supervision drawn from VA*s programs for
monitoring the quality and outcomes of patient care.

GAO recommends that VA  ensure that VA medical centers that provide GME
adopt and adhere to VA*s national requirements for resident supervision
and

 ensure that external peer review of documentation of resident
supervision includes

records from VA*s new outpatients. VA concurred with the

recommendations.

www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 625. To view the full product,
including the scope and methodology, click on the link above. For more
information, contact Cynthia A. Bascetta at (202) 512- 7101. Highlights of
GAO- 03- 625, a report to

congressional requesters

July 2003

VA HEALTH CARE

Adequacy of Resident Supervision Is Not Assured, but Plans Could Improve
Oversight

Page i GAO- 03- 625 VA*s Oversight of Resident Supervision Letter 1
Results in Brief 3 Background 6 VA Lacks Adequate Procedures to Monitor
Implementation of Its

Supervision Requirements 10 VA Is Acting to Obtain Information about
Supervision from Accrediting Bodies and Residents 20 VA Uses Its Programs
for Monitoring Patient Care to Identify and Correct Problems with Resident
Supervision 24 Conclusions 27 Recommendations for Executive Action 28
Agency Comments 28 Appendix I Scope and Methodology 30

Appendix II Comments from the Department of Veterans Affairs 34

Appendix III GAO Contact and Staff Acknowledgments 37 GAO Contact 37 Staff
Acknowledgments 37 Tables

Table 1: Examples of Requirements from VA*s Resident Supervision Handbook
Issued on October 25, 2001, by Domain of Residents* Health Care Activities
10 Table 2: Examples of Questions about Monitoring Processes from

the Annual Report on Residency Training Programs Completed by Medical
Centers and Networks 13 Table 3: Number of VA Medical Centers That
Reported Monitoring Some Aspect of the Documentation of Resident
Supervision, by Domain of Residents* Health Care Activities 15 Table 4: VA
Networks Included in Our Sample 31 Table 5: VA Medical Centers Included in
Our Sample 32 Contents

Page ii GAO- 03- 625 VA*s Oversight of Resident Supervision Abbreviations

ACGME Accreditation Council for Graduate Medical Education GME graduate
medical education NSQIP National Surgical Quality Improvement Program OAA
Office of Academic Affiliations VA Department of Veterans Affairs

This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.

Page 1 GAO- 03- 625 VA*s Oversight of Resident Supervision

July 2, 2003 The Honorable Bob Graham Ranking Minority Member Committee on
Veterans* Affairs United States Senate

The Honorable John D. Rockefeller IV United States Senate

The Veterans Health Administration of the Department of Veterans Affairs
(VA) is the largest single provider of graduate medical education (GME)
training sites in the United States, with as many as one- third of the
nation*s resident physicians receiving part or all of their training in VA
health care facilities. Residents are medical school graduates who receive
supervised training in a medical specialty (such as internal medicine or
surgery) prior

to providing care without supervision. 1 As a provider of GME, VA faces
the dual challenges of ensuring the safety and quality of the health care
its patients receive from residents while simultaneously providing
residents with appropriate educational opportunities. Supervision of
residents by qualified physicians is central to balancing these patient
care and educational goals, and responsibility for the care provided by a
resident to any patient belongs to the licensed physician who supervises
that

resident. 2 Through observation and direction, supervising physicians are
to impart knowledge and skills to residents while making sure that
patients receive appropriate, timely, and effective care. Effective
oversight is necessary if VA is to assure the adequacy of resident
supervision. Key components of oversight include procedures to assess the
supervision residents receive and to initiate corrective action when there
is a problem. Information from multiple, complementary sources can be used
to assess supervision; such information includes evidence of whether
residents receive required supervision, evaluations of the

1 In this report, the term *residents* also refers to fellows, physicians
who have already completed a residency and are obtaining additional
training in an advanced specialty or subspecialty.

2 VA requires that the supervisor be a licensed physician who has been
credentialed and privileged as a member of the staff of the medical
facility in which the care is provided.

United States General Accounting Office Washington, DC 20548

Page 2 GAO- 03- 625 VA*s Oversight of Resident Supervision

adequacy of supervision by organizations that accredit GME programs and by
residents, and analyses of the quality and outcomes of care provided by
residents. In 1986 and 1992, we reported that VA headquarters officials
had

not adequately overseen resident supervision and that the documentation of
resident supervision at some medical centers was inadequate. 3 Although
documentation does not fully communicate the extent or quality of
supervision, it is an important record of whether a supervising physician
was involved in a patient*s care.

Responsibilities for resident supervision and its oversight are
distributed across multiple VA organizational components and are shared by
VA*s affiliated medical schools and teaching hospitals. VA headquarters
established national requirements for supervision of residents* health
care activities* including diagnosis, treatment, and discharge of
patients* and for oversight of supervision. Responsibilities for
implementing these requirements are assigned to the administrators of its
regional networks 4 of medical facilities, medical center managers, and
supervising physicians.

Most residency training within VA medical centers is conducted through GME
programs run by medical schools or other teaching hospitals, which are
known as sponsoring institutions. GME accrediting bodies hold the
sponsoring institutions responsible for the quality of the GME program in
each medical specialty. As a result, VA medical centers share
responsibility for ensuring the adequacy of residents* supervision with
these affiliated sponsoring institutions.

Concerned that overlapping authority for residents* activities could allow
problems with resident supervision to go undetected or uncorrected, you
asked us to examine the adequacy of VA*s oversight of resident
supervision. In response to your request, we examined VA*s procedures for
(1) monitoring VA medical centers* adherence to VA*s requirements for
resident supervision, (2) using evaluations of supervision by GME
accrediting bodies and residents, and (3) using information about resident

3 See U. S. General Accounting Office, VA Hospitals: Surgical Residents
Need Closer Supervision, GAO/ HRD- 86- 15 (Washington, D. C.: Jan. 13,
1986) and VA Health Care: Medical Centers Are Not Correcting Identified
Quality Assurance Problems,

GAO/ HRD- 93- 20 (Washington, D. C.: Dec. 30, 1992). 4 VA health care
facilities are organized into 21 regional networks, known as Veterans
Integrated Service Networks, which are to coordinate the activities of and
allocate funds to VA health care facilities. VA had 22 networks until
January 2002, when it merged two of them.

Page 3 GAO- 03- 625 VA*s Oversight of Resident Supervision

supervision drawn from VA*s programs for monitoring the quality and
outcomes of patient care. To address these objectives, we examined VA*s
policy for resident supervision and reviewed relevant documents from VA
headquarters offices, networks, and medical centers. We analyzed annual
reports on residency training submitted to VA headquarters for the 2000/
2001 academic year by VA*s regional networks and 114 of the approximately

130 VA medical centers that provide GME training. 5 We interviewed VA
officials, as well as GME experts and officials of accrediting bodies,
medical associations, and other stakeholder groups. We also interviewed
GME managers from 11 of VA*s 21 regional networks and 11 medical

centers. The sample included one medical center from each sampled network
and was designed to cover a range in total number of residency positions
and number of medical specialties in which training occurred. We reviewed
information from three additional medical centers involved in GME programs
that, as of May 2002, had been placed on probationary

accreditation or for which accreditation was to be withdrawn. 6 We visited
two of those medical centers. Our work covered VA*s oversight of resident
supervision and did not include an evaluation of the quality of care
provided by residents or the quality of the supervision provided to
residents. We conducted our work from September 2001 through June 2003 in
accordance with generally accepted government auditing standards. See
appendix I for a more detailed discussion of our scope and methodology.

VA cannot assure that the residents who provide care in its facilities
receive adequate supervision because its procedures for monitoring
supervision are insufficient. VA does not know whether medical centers
have adopted resident supervision policies that are consistent with VA*s
national requirements for supervision of residents* health care
activities, such as diagnosis, treatment, and discharge of patients. VA
officials require a review of only the requirement that is intended to
ensure

5 The number of VA medical centers that provide GME varies slightly over
time, in part because at facilities with only a few allocated residency
slots, it is possible that no slot might be filled at a particular time. 6
In addition to these three programs, one program at 1 of the 11 medical
centers in our sample was on probationary accreditation. Of the four
programs that had received adverse accreditation decisions, two are no
longer in an adverse accreditation status. Reevaluation of the other two
was not complete as of May 2003. Results in Brief

Page 4 GAO- 03- 625 VA*s Oversight of Resident Supervision

availability of supervision when a supervising physician does not need to
be in the operating or procedural suite while a resident performs a
diagnostic or therapeutic procedure. Network GME officials are to review
this requirement in medical centers* policies, and 4 of 11 network GME
managers we interviewed had not conducted this review. Moreover, we found
that the requirement at one medical center in one of these four networks
was less stringent than VA*s national requirement. To learn which aspects
of resident supervision medical centers and networks monitor, VA requires
medical centers and networks to submit annual reports on residency
training. Medical centers* annual reports for the 2000/ 2001 academic year
indicate that most medical centers monitor some documentation of
supervision, but few conduct comprehensive reviews.

About half of the 91 medical centers that reported having a review process
also reported finding inadequate documentation of supervision and then
taking steps to improve it. To obtain more complete information about

adherence to its national requirements for supervision, VA plans to have
external peer reviewers examine the documentation of supervision in
patients* medical records. External peer review could allow assessment of
whether most of VA*s key documentation requirements are being met. VA*s
plans for this review, however, have not been finalized, and as of May
2003, VA had not decided whether reviewers would examine records from VA*s
new outpatients. Without a sample of records from new patients,

reviewers would not be able to assess the required documentation of
supervisory involvement during a veteran*s first outpatient visit.

VA is making efforts to obtain information from accrediting bodies and
residents about the quality of supervision provided to its residents. For
example, VA has taken steps to gain copies of accreditation letters
directly from GME accrediting bodies. These letters contain evaluations of
the GME programs in which VA medical centers participate and are sent to
the institutions that sponsor GME programs, but not to medical centers
that participate in those programs. As a result, VA medical centers must
generally rely on affiliated sponsoring institutions to inform them of
problems identified by the accrediting body. We found that most VA medical
center managers indicated that their affiliated sponsors had shared this
information, and when problems with VA were identified, VA managers
reported taking action to solve them. We also found, however, that one
sponsoring institution did not provide a participating VA medical center
with timely information about an impending withdrawal of accreditation for
a GME program. To obtain standardized feedback from residents about their
educational experiences, including the quality of

their supervision, VA implemented a national survey in 2001. In 2001 and
2002, VA could not send the survey to a representative sample of residents

Page 5 GAO- 03- 625 VA*s Oversight of Resident Supervision

from each VA medical center because it lacked a complete list of its
residents. VA is taking action to obtain this information so that it can
send the survey to a sample of residents at each medical center. Medical

centers* annual reports provide network and headquarters officials with
additional information about concerns expressed by residents and steps
taken to address those concerns.

VA also uses information from its broader programs for monitoring the
quality and outcomes of patient care, such as its patient safety and
surgical quality improvement programs, to identify and correct problems
with resident supervision. Although too new to evaluate, the patient
safety program VA implemented in January 2002 established a process for
determining the causes of events that led to or could have led to patient
harm and for taking steps to eliminate or minimize identified risks, such
as inadequate resident supervision. In addition, VA*s program for
monitoring and improving surgical outcomes allows VA to examine residents*
performance of surgical procedures. Information generated by this program
has prompted medical center and network officials to take steps to improve
the supervision of surgical residents. For example, a team of

experts from this program noted inadequate supervision of surgeries
performed by urology residents at one medical center they visited in 2002.
The medical center responded by arranging for urologists to spend more

time at the medical center and ensuring that they understood VA*s
supervision requirements. Tort claim review is another way VA monitors the
possible role of resident supervision in problems with patient care.
Review of paid tort claims led VA in 2001 to clarify its requirements for

supervision in inpatient settings by adding an explicit reference to
weekends and holidays to the requirement that each inpatient must be seen
by the supervising physician within 24 hours of admission.

To improve VA*s oversight of resident supervision and help ensure the
quality of both health care and GME, we are making recommendations to the
Secretary of Veterans Affairs to ensure that VA medical centers adopt and
adhere to VA*s national requirements for resident supervision and to

ensure that external peer review of documentation of resident supervision
includes records from VA*s new outpatients. VA concurred with our
recommendations.

Page 6 GAO- 03- 625 VA*s Oversight of Resident Supervision

Education is one of VA*s four core missions, 7 and in fiscal year 2002, VA
paid approximately $383 million to residents training at about 130 VA
health care facilities. 8 For the 2002/ 2003 academic year, VA supported
almost 8,800 residency slots, about 9 percent of all residency training
positions in the United States. Moreover, because several residents
typically rotate through each slot, VA estimates that it provides graduate
medical training to more than 28,000 residents each year, or as many as

one- third of the nation*s residents. The number of residency slots VA
allocates to individual medical centers involved in GME ranges from less
than 1 to more than 200. 9 Although about half of VA*s residency positions
are in primary care, VA supports GME in 45 recognized medical specialties
and subspecialties; individual medical centers provide training in from 1
to more than 30 specialties.

VA headquarters officials have ultimate oversight responsibility for the
activities of residents within VA medical centers, and several different
headquarters offices have monitoring functions that relate to resident
supervision. VA*s Office of Academic Affiliations (OAA) has responsibility
for developing and overseeing policies for resident supervision,
monitoring VA*s GME activities, and allocating residency slots. Under the
Patient Safety Program VA implemented in January 2002, VA*s National
Center for Patient Safety collects and analyzes information from VA
medical centers about patient risk events and their causes. Medical
centers are required to report all patient safety events* including
adverse events and close calls 10 *to the National Center for Patient
Safety. In addition, medical centers are required to determine the root
causes of patient safety incidents with severe or potentially severe
outcomes and develop plans to prevent them in the future. The success of
this program will depend on the extent to which VA is able to establish a
culture in

7 VA*s four core missions are patient care, education, research, and
medical backup to the Department of Defense in the event of a national
security emergency. 8 These expenditures included stipends and benefits
for residents training in accredited medical specialties and
subspecialties and an additional 150 special fellows training in emerging,
as yet nonaccredited fields of medicine, such as geriatric neurology and
palliative care. 9 Allocations of fractions of slots are possible because
residents might obtain only a part of their training at a VA medical
center.

10 Adverse events include adverse drug events and procedural errors or
complications that are associated with care. Close calls are events or
situations that could have resulted in an adverse event but did not,
either by chance or through timely intervention. VA specifies that
alternative procedures are to be used for reporting intentionally unsafe
acts. Background

Page 7 GAO- 03- 625 VA*s Oversight of Resident Supervision

which employees feel safe to make these reports. 11 VA*s Office of Patient
Care Services establishes and monitors health care programs. For example,
its National Surgical Quality Improvement Program (NSQIP) examines
postoperative outcomes. 12 Additional oversight of resident supervision is
provided by VA*s Office of Inspector General. 13 Because VA*s health care
system is decentralized, responsibilities for

implementing VA*s national policy for resident supervision are assigned to
networks and medical centers. Network officials are to provide medical
centers with the resources necessary to ensure that residents are
supervised in accordance with VA*s national policy and are to evaluate the
strengths and weaknesses of medical centers* GME activities. Medical
center directors are responsible for establishing facility policies for
resident supervision that fulfill the requirements of VA*s national
policy, 14 and medical center chiefs of staff are responsible for the
educational and patient care activities of all residents within the
facility. In addition, a

physician in each medical specialty is responsible for ensuring that the
residents training in that specialty are supervised as required. VA
medical centers typically also share responsibility for the oversight of
residents with affiliated institutions that sponsor GME programs. VA

participates in more than 1,900 distinct GME programs, 29 of which are
sponsored by VA medical centers. 15 The rest are sponsored by about 120
medical schools and teaching hospitals with which VA medical centers are
affiliated. The majority of VA medical centers work with one GME

11 VA arranged for the National Aeronautics and Space Administration to
provide an independent external system for reporting patient safety
concerns. This system allows anyone who feels uncomfortable reporting an
event to VA*s internal patient safety managers to file a voluntary,
confidential report to an outside agency. Reports entered in this database
are anonymous. Nationwide implementation of this second reporting system
began in March 2002.

12 NSQIP is housed administratively in VA*s Office of Patient Care
Services. It exercises its monitoring and advisory functions through the
chief medical officers of VA*s networks. 13 In April 2003, VA*s Office of
Inspector General reported that part- time physicians were

not always present in the clinics where the residents they supervised
provided care. See VA Office of Inspector General, Audit of the Veterans
Health Administration*s Part- Time Physician Time and Attendance, 02-
01339- 85 (Washington, D. C.: April 2003). 14 Medical centers must adopt
their own policies to ensure that local requirements, such as those
established by affiliated GME sponsors, are included. 15 These 29 GME
programs are sponsored by seven VA medical centers, each of which also
participates in GME programs that are sponsored by affiliated
institutions.

Page 8 GAO- 03- 625 VA*s Oversight of Resident Supervision

sponsoring institution, but individual VA medical centers participate in
the GME programs of up to four different sponsors. When a VA medical
center serves as a training site for residents, but is not the sponsoring
institution, it is known as a participating institution. GME accrediting
bodies hold

sponsoring institutions responsible for all aspects of their educational
programs, including aspects conducted within participating institutions.
GME accrediting bodies do not separately accredit participating
institutions and do not evaluate the extent to which supervision that
occurs within participating institutions, such as VA medical centers,
meets requirements set by those participating institutions.

VA requires accreditation of each GME program through which its residents
obtain training. More than 98 percent of VA*s residency slots are filled
by residents in GME programs that are subject to accreditation review by
the Accreditation Council for Graduate Medical Education (ACGME); the
remaining slots are filled by residents in osteopathic programs that are
subject to accreditation review by the American Osteopathic Association.
GME accreditation status indicates an overall assessment of the quality of
an educational program in a particular

medical specialty. Accrediting bodies evaluate several aspects of each GME
program, including provisions for the supervision and safety of residents,
the adequacy of institutional resources, educational curriculum, and the
extent to which the program meets that specialty*s specific training
requirements. A program can be fully accredited, or a program can be
granted accreditation with notification of problems that must be
corrected. Accreditation can also be withdrawn. A program*s accreditation
status is made public, but to safeguard confidential information, 16
specific problems with the program or its training sites are described in
letters sent only to the sponsoring institution. Accrediting bodies have
not been sending these letters to participating institutions.

Accrediting bodies state that the quality of patient care must remain the
highest priority of GME programs. Health care organizations that provide
GME must ensure that qualified staff physicians supervise residents and
that the same standards for the quality and safety of patient care apply
when residents are involved in health care delivery as when they are not.
GME accrediting bodies require that supervising physicians adjust the
level of supervision to meet the educational goal of increasing residents*

16 ACGME classifies certain records as confidential to foster candor by
residency programs, residents, and others as they submit information
during the accreditation process.

Page 9 GAO- 03- 625 VA*s Oversight of Resident Supervision

competence by giving them appropriate opportunities to assume greater
independence in their patient- care activities, that is, allowing
residents to assume graduated responsibilities. The supervising physician
relies on his or her professional judgment and knowledge of the patient*s
medical condition and the resident*s level of mastery to determine the
degree of independence of the resident*s patient- care responsibilities.

VA*s national policy on resident supervision is detailed in a handbook
that establishes specific requirements for (1) the involvement of
supervising physicians in the care provided by residents who diagnose,
treat, or discharge patients and (2) the documentation of that
involvement. 17 These specific requirements apply to four domains of
residents* clinical activity* inpatient care, outpatient care, diagnostic
and therapeutic procedures, and consultations* and provide guidelines for
putting into practice GME accrediting bodies* principles of resident
supervision and graduated levels of responsibility. (See table 1 for an
example of VA*s requirements for supervision in each of the four domains.)
Experts on GME told us that the requirements in VA*s handbook are
reasonable and appropriately consider the role of supervision in ensuring
the quality of patient care and of resident education. Some of these
experts described it as a best practice model. 17 The most recent revision
of the handbook was issued on October 25, 2001.

Page 10 GAO- 03- 625 VA*s Oversight of Resident Supervision Table 1:
Examples of Requirements from VA*s Resident Supervision Handbook Issued on
October 25, 2001, by Domain of Residents* Health Care Activities

Domain Examples of requirements for supervision and its documentation
Inpatient care The supervising physician must meet each new inpatient

within 24 hours of admission (including weekends and holidays) and
personally document that encounter in the patient*s medical record.
Concurrence with, or modifications to, the resident*s diagnosis and
treatment

plan must be documented in the supervising physician*s progress note.
Outpatient care The supervising physician must supervise the initial visit
of

each new patient to an outpatient clinic, either by seeing the patient or
discussing the patient with the resident at that initial visit.
Involvement of the supervising physician must be documented in the medical
record. Diagnostic and therapeutic procedures When a resident is involved
in the care of a patient who is

to undergo an elective or scheduled procedure, the supervising physician
is to write a preprocedural note that indicates the diagnosis and
treatment plan. Consultations The supervising physician must meet with
each patient

who was seen by a resident for a consultation and document his or her
personal evaluation in the patient*s medical record.

Source: VA.

VA does not have adequate procedures to determine whether residents at VA
medical centers are supervised in accordance with its national
requirements. For example, VA does not check whether each medical center
involved in GME has adopted policies that are consistent with VA*s
requirements for resident supervision. To learn what medical centers and
networks do to monitor whether supervision is consistent with VA*s
national requirements, VA requires that medical centers and networks
submit annual reports on residency training. Medical centers* reports
filed for the 2000/ 2001 academic year indicate that most medical centers
review some documentation of resident supervision, but few conduct
comprehensive reviews. To obtain more complete information about the
supervision residents receive, VA is planning to use external peer review
to assess adherence to its requirements for documenting resident
supervision. These plans have not been finalized. For example, as of May
2003, VA had not decided whether reviewers would examine records from VA*s
new outpatients. VA Lacks Adequate

Procedures to Monitor Implementation of Its Supervision Requirements

Page 11 GAO- 03- 625 VA*s Oversight of Resident Supervision VA does not
know whether all its medical centers have adopted policies that are
consistent with the specific requirements in its resident

supervision handbook for the supervision of residents* diagnosis,
treatment, and discharge of patients. The director of each medical center
involved in GME is to establish facility policies for resident supervision
that fulfill the requirements in VA*s handbook, but VA requires a review
of only one requirement involving the supervision of diagnostic and

therapeutic procedures* the medical centers* requirements for the minimal
acceptable level of supervision for diagnostic and therapeutic procedures.
Specifically, in situations in which the supervising physician is not in
the operating or procedural suite, VA requires that the supervisor

must, at a minimum, be immediately available in the facility or campus to
provide direct supervision of the procedure if necessary. 18 Network GME
managers 19 are supposed to review and approve this requirement; they are
not required to report the results of their reviews to OAA. There is no
separate OAA review of any of the requirements in medical centers*
supervision policies.

We found that not all networks have completed the one required review and
that medical centers* policies are not always consistent with VA*s
national policy. Of the 11 network GME managers we interviewed, 7 told us
that they had completed this required review of the minimal requirements
for supervision of procedures in medical center policies, but 4 told us
that they had not. We found that the requirement of a medical center in
one of the four networks that had not conducted this review was less
stringent than the requirement in VA*s handbook for supervision of
diagnostic and therapeutic procedures. The written policy at this medical
center stated that the supervising physician can be immediately available
by telephone rather than requiring him or her to be immediately available

in the facility or on campus. 20 One network GME manager who did review
this requirement for supervision of diagnostic and therapeutic procedures
told us that in 2002, he identified three medical centers that had written
18 VA*s requirements for the minimum level of supervision for diagnostic
and therapeutic procedures do not apply to procedures performed in
emergency situations, in which immediate action is necessary to save a
patient*s life or prevent serious impairment of the patient*s health, or
to procedures that are elements of routine and standard patient care, such
as drainage of superficial abscesses.

19 These managers are known as network academic affiliations officers. 20
An official of this medical center told us in September 2002 that there
had been no adverse patient outcomes associated with resident supervision
during the preceding 2 years. VA Does Not Determine

Whether VA Medical Centers* Policies Are Consistent with Its National
Requirements for Resident Supervision

Page 12 GAO- 03- 625 VA*s Oversight of Resident Supervision requirements
for supervision of these procedures that were less stringent than the
requirement in VA*s handbook and that he instructed each of

these facilities to change its policy to be consistent with VA*s national
requirement.

To learn what medical centers and networks do to monitor whether
supervision is consistent with VA*s resident supervision handbook, VA has
required annual reports on residency training programs beginning with the
1999/ 2000 academic year. Medical center managers are to provide narrative
answers to specific open- ended questions about their monitoring processes
as well as about the problems they identified and actions they took to
address them for each of three areas of oversight. (See table 2.) These
medical center reports are channeled through VA*s networks to OAA. Network
officials are to review them and summarize the strengths and weaknesses of
the medical centers* GME programs in network- level annual reports, which
are also submitted to OAA. VA Headquarters Monitors

Medical Center and Network Oversight of Resident Supervision through
Annual Reports

Page 13 GAO- 03- 625 VA*s Oversight of Resident Supervision Table 2:
Examples of Questions about Monitoring Processes from the Annual Report on
Residency Training Programs Completed by Medical Centers and

Networks Area of oversight

Examples of questions to be completed by medical center managers

Examples of questions to be completed by network managers

Supervision requirements Describe your process for reviewing and
monitoring medical center data collected for assessing resident
supervision in the following areas: (1) inpatient admission, continuing
care, and discharge supervision; (2) outpatient visit

supervision; and (3) supervision of diagnostic and therapeutic procedures
and consultations. a Describe any network- level

process for review of medical center data collected for assessing
adherence to VA*s educational supervision requirements and the results of
such review in the following areas: (1) inpatient admission, continuing
care, and discharge supervision; (2) outpatient visit

supervision; and (3) supervision of diagnostic and therapeutic procedures
and consultations. Evaluations of resident supervision Describe concerns
of the

accrediting bodies specific to VA clinical rotations. b Describe your
process for

obtaining and reviewing resident comments related to their VA clinical
training experience. a Describe any network- level

process for review of residents* comments related to their VA clinical
training experience and the results of such review.

Patient care Describe your process for reviewing and monitoring all
incidents and risk events c with complications to ensure that the
appropriate level of resident supervision occurred. a Describe any
network- level

review process for assessing incidents and risk events c to ensure that
the appropriate level of resident supervision occurred and the results of
that review.

Source: VA. a Medical centers are also asked to describe results of their
reviews and action plans for correction or remediation of problems found.
b Medical centers are also asked to note each program*s accreditation
status, summarize affiliate and

VA responses to accrediting body concerns, and describe any corrective
actions. c Risk events include events that did result, or could have
resulted, in an adverse outcome.

These annual reports can provide managers with limited, but useful,
information about the extent and quality of monitoring performed by
medical centers, including whether medical centers monitor documentation
or some other indication of supervision. Some medical centers and networks
provided little detail in response to the annual reports* open- ended
questions. For example, not all medical centers

Page 14 GAO- 03- 625 VA*s Oversight of Resident Supervision described
which specific aspects of resident supervision they monitored. OAA used
open- ended questions in part to accommodate differences

among medical centers in the number and type of residents they train. VA
officials have used information from annual reports to monitor medical
center oversight of resident supervision. For example, one network GME
manager followed up on a problem identified through a medical center
annual report by requiring the medical center to submit an action plan for
improving supervision of ophthalmology residents by the beginning of the
2002/ 2003 academic year. An OAA official told us that analysis of these
annual reports not only helped identify areas of vulnerability with
residency programs, but also pointed to possible best practices.

VA does not require its medical centers or networks to conduct systematic
reviews of the documentation of resident supervision, 21 and medical
centers differ in the extent to which they monitor adherence to VA*s
requirements for supervision. More than three- fourths of medical centers*
annual reports included a description of an independent review of the
documentation of supervision of at least one aspect of care provided by
residents, but most medical centers did not describe reviews of all four

domains of residents* health care activities. 22 For each of three
domains* inpatient care, outpatient care, and diagnostic and therapeutic
procedures* over half the medical centers described a process for an

independent review of at least one element of the documentation of
resident supervision, that is, a review by someone other than a physician
with related supervisory responsibilities (see table 3). For example, the
quality management office at one medical center reviews medical records
each month to determine whether documentation indicates that inpatients
were seen by supervising physicians within 24 hours of admission. As

21 An OAA official told us that OAA does not require medical centers or
networks to conduct comprehensive documentation reviews to avoid
duplicating the cost and effort VA headquarters is expending to develop a
plan for systemwide external peer review of supervision documentation.
This plan will be addressed in the next section of this report. 22 OAA
provided us with annual reports from 114 of the approximately 130 medical
centers

that were allocated VA- funded residency slots during the 2000/ 2001
academic year. These were all the medical center annual reports for the
2000/ 2001 academic year OAA had received as of June 18, 2002. Before
giving these reports to us, OAA redacted them to remove identifying
information such as the names of medical centers and sponsoring

institutions. We analyzed these reports to determine whether the medical
centers described a systematic, independent review of the documentation of
resident supervision in a sample of medical records. Most VA Medical
Centers Monitor Some

Documentation of Resident Supervision, but Few Conduct Comprehensive
Reviews

Page 15 GAO- 03- 625 VA*s Oversight of Resident Supervision shown in table
3, few medical centers, however, described such a process for review of
supervisory documentation when residents provide

consultations to patients* primary physicians.

Table 3: Number of VA Medical Centers That Reported Monitoring Some Aspect
of the Documentation of Resident Supervision, by Domain of Residents*
Health Care Activities

Inpatient care Outpatient

care Diagnostic

and therapeutic procedures Consultations

Explicit independent review process described a 77 58 65 21

No explicit independent review process described b 26 47 42 86

Blank, missing, or reported to be not applicable 11 9 7 7

Total 114 114 114 114

Source: VA. Notes: GAO analysis of VA medical center 2000/ 2001 academic
year annual reports on resident supervision submitted to VA by June 18,
2002. We considered the review process to be independent if the
description indicated that documentation is reviewed by someone other than
a physician with related supervisory responsibilities. a Includes all
descriptions of systematic independent review processes of one or more
aspects of

documentation, as well as less systematic review processes and reviews of
only some services provided by residents. b Includes medical centers that
stated that they had no process for that domain or for which the
description included insufficient information to determine whether the
process was independent and systematic.

In addition, medical centers* annual reports did not always include clear,
detailed descriptions of the documentation requirements they monitor. Few
specifically mentioned monitoring particular VA- wide requirements, such
as the requirement for documentation of supervisory involvement at the
time of each new outpatient*s first visit. In some instances, medical
centers described a less systematic review process or one that was used
for only some services provided by residents. For diagnostic and
therapeutic procedures, for example, some medical centers described

Page 16 GAO- 03- 625 VA*s Oversight of Resident Supervision processes for
reviewing only selected procedures, such as endoscopies or major
surgeries. 23 About half of the 91 medical centers that reported having an
independent review process indicated they found deficiencies with the
documentation of resident supervision, and all but one discussed actions
they took to

correct these problems. 24 For example, officials from one medical center
told us that they implemented a program to discipline individual
physicians who consistently do not meet the medical center*s requirements
for documenting supervision. The acting chief of staff there told us that
during the 2001/ 2002 academic year, three physicians had each been
suspended without pay for 1 day for not consistently meeting

documentation requirements and that there had been significant improvement
in the documentation of resident supervision since this disciplinary
program went into effect. This medical center has also developed a
strategy for linking contract physicians* pay to their provision and
documentation of supervision. 25 Documentation reviews have proven useful
in identifying inadequate supervision. We identified three medical centers
that described in their

annual reports finding evidence of inadequate resident supervision through
their documentation reviews. In their annual reports, two of these three
medical centers stated that there were no adverse patient events involving
resident supervision. The third did not state whether there had been any
adverse patient outcomes. In the first instance, the medical center
reported that its review of documentation indicated that some staff
physicians provided a *low level* of supervision to residents in the
inpatient surgical setting. Medical center officials responded by meeting
with those physicians and conducting a follow- up review to monitor the

23 Seven of the 11 medical center GME managers we interviewed told us that
since preparing their 2000/ 2001 annual reports, their medical centers
have implemented or are developing additional reviews of the documentation
of supervision. For example, one medical center that had not reviewed
documentation of resident supervision in inpatient settings during the
2000/ 2001 academic year began reviewing that documentation on a quarterly
basis during the 2001/ 2002 academic year.

24 Insufficient documentation does not necessarily indicate a lack of
supervision. For example, some medical centers reported that supervision
was documented, but not in a way that met VA*s requirements, and others
reported that interviews with staff indicated

that appropriate supervision had occurred, although documentation was
lacking. 25 In addition to employing salaried physicians, VA medical
centers sometimes use contracts to obtain the services of medical
specialists.

Page 17 GAO- 03- 625 VA*s Oversight of Resident Supervision level of
supervision. In the second instance, the medical center reported that its
supervision of residents was generally satisfactory, but that it had

found through its documentation review one episode in which the attending
surgeon had left the city during a procedure that he was supposed to be
supervising. This medical center reported that the surgeon was formally
reprimanded. In the third instance, a medical center reported that through
its documentation review, it identified two specialties* urology and
plastic surgery* for which it wanted to increase the number of procedures
performed with the staff physician physically present and directly
involved in the surgery. The medical center reported that its management
was working with the surgery service chief to achieve this goal.

We also identified a few medical centers that described independent
processes for monitoring resident supervision that went beyond reviewing
documentation. One medical center, for example, reported that staff in its
intensive care unit are required to report to the nurse manager any
situation they observe in which the supervision of a resident was
inappropriate.

In addition to monitoring processes established by medical centers, five
of VA*s networks indicated in their 2000/ 2001 annual reports that they
had a networkwide process for assessing adherence to one or more VA
requirements for documentation of resident supervision. For example, two
networks stated that they monitor the documentation of supervising
physicians* involvement in the care of inpatients within 24 hours of
admission and another network assesses documentation of the supervision of
high- risk procedures. Two other networks reported they are developing
networkwide monitoring processes.

Page 18 GAO- 03- 625 VA*s Oversight of Resident Supervision To obtain more
complete information about the extent to which its requirements for
supervision are being followed, VA has begun to test its

plans to monitor adherence through external peer review of the
documentation of supervision. External peer reviewers would examine a
sample of medical records from each medical center involved in GME to
determine whether they include required documentation of supervision. 26
Although documentation does not provide full information about the

extent or quality of supervision, it can provide VA oversight officials
with important information about whether supervisors were involved in
patient care. We compared the instructions that external reviewers would
follow with the requirements for supervision in VA*s handbook and found
that the instructions would allow reviewers to assess adherence to most of
VA*s key documentation requirements in the four domains of residents*
health care activities. For example, if a resident participated in the
care of an inpatient or an outpatient during the current academic year,
the external reviewer is to determine whether documentation of supervision
in the

patient*s medical record met the requirements in VA*s national handbook.
Reviewers are also to assess documentation of the supervision of residents
who performed diagnostic or therapeutic procedures or provided
consultations to other physicians. Results from each medical center are to
be provided to that medical center, as well as to headquarters managers.

External peer review of documentation of supervision in medical records
will be facilitated by features of VA*s computerized patient record
system. 27 For example, the system automatically records the date and time
of notes; it also has the capacity to require that notes written by a
resident

be co- signed by the supervising physician, in which case the note is not
considered complete until the required co- signature has been entered. In
addition, supervising physicians with whom we spoke noted that

26 As part of its broader quality management process, VA began its
External Peer Review Program in 1995. Through this program, trained
reviewers from outside VA examine documentation from a sample of medical
records from each medical center to determine whether specific health care
activities, such as influenza immunization, have occurred. These data have
allowed VA to monitor its progress in meeting specific health care
objectives.

27 The core features of VA*s computerized patient record system, which was
developed to support its health care mission, have been installed at all
VA medical centers, although medical centers differ in the extent to which
it is used. External reviewers will review either electronic or paper
records, whichever are available. VA*s Plans to Use External

Peer Review to Monitor Documentation of Supervision Have Not Been
Finalized

Page 19 GAO- 03- 625 VA*s Oversight of Resident Supervision immediate and
easy access to legible information facilitates supervisors* review of
residents* activities. 28 VA is in the early stages of testing its
procedures for external peer review

of the documentation of resident supervision, and a VA official told us
that this effort is a high priority. A pilot test of portions of the
inpatient assessment methodology was conducted from October 2001 through
June 2002 on a sample of almost 10,000 medical records. That pilot test
indicated that the central database used to select the sample of medical
records does not include information about which patients were seen by
residents. As a result, reviewers were unable to select an appropriate
sample of medical records. Until this problem is resolved, VA cannot
implement its plans for external peer review of resident supervision. OAA
has worked with other headquarters offices to revise VA*s information
technology software to ensure that this database contains information
about whether patients* physicians were residents. VA expects to implement
this revision to its software by July 2003. The pilot test did not
indicate any other obstacles to implementing the portion of the plan for
reviewing documentation of resident supervision in inpatient settings.
Pilot tests of methods for assessing documentation of outpatient care,
diagnostic and therapeutic procedures, and consultations will not begin
until patients seen by residents can be clearly identified through the
central database.

One unresolved issue that will affect the usefulness of the external
review of supervision documentation in the outpatient setting involves
selection of the sample of medical records. The two options under
consideration are relying on the main outpatient sample used for VA*s
other external peer reviews or developing a sample specifically for review
of the documentation of supervision. The main outpatient sample in any
given

year includes only patients who have received primary health care from VA
in the past and excludes most new patients who began obtaining health care
through VA within the preceding year 29 *a group that has greatly 28 The
Association of American Medical College*s Joint Committee of the Group on
Resident Affairs and Organization of Resident Representatives has reported
that computerized medical records can enhance the safety of patient care
in teaching hospitals.

29 VA told us that it excludes new patients from its main outpatient
sample to facilitate comparison of its performance measures to those from
the National Committee for Quality Assurance*s Health Plan Employer Data
and Information Set, which collects data from private- sector patients who
have been enrolled in a health plan for two consecutive years.

Page 20 GAO- 03- 625 VA*s Oversight of Resident Supervision expanded in
recent years. 30 Without a sample of records from new patients, it will
not be possible to assess adherence to VA*s requirement

for supervisory involvement during a veteran*s first outpatient visit. An
OAA official told us that developing an additional sample of outpatient
records for review of documentation of supervision, distinct from the main
outpatient sample used for other purposes, would add to the expense of the
review. As of May 2003, VA had not made a decision about which sample to
use.

VA is making efforts to obtain consistent access to information provided
by accrediting bodies and residents about the quality of resident
supervision in VA medical centers. VA has taken steps to gain direct
access to the letters accrediting bodies send to sponsoring institutions
to describe concerns about GME programs. VA headquarters also developed a
survey to obtain feedback from residents, but cannot send it to a random
sample of residents because VA does not have a complete list of its
residents. VA is improving its ability to obtain that information.
According to their annual reports for the 2000/ 2001 academic year, most
VA medical centers that provide GME have some procedure for obtaining
feedback from residents.

VA does not currently have direct access to accreditation letters that
contain reviews of the GME programs sponsored by VA medical centers*
affiliates. These letters document concerns about residents* education or
clinical experience that the GME program must address to retain
accreditation. Timely access to the information in these letters can allow
medical centers to take corrective actions. Until early 2000, ACGME sent
copies of its accreditation letters to OAA, 31 and OAA made VA support for
residency slots contingent on VA medical centers* taking action to correct
identified problems. In 2000, however, ACGME adopted new policies to
safeguard confidential accreditation information. As a result, ACGME

30 From fiscal year 1996 to fiscal year 2002, the number of patients who
received health care from VA increased from about 2.9 million to 4.7
million. 31 During the time when OAA received copies of ACGME*s
accreditation letters, OAA did not have direct access to accreditation
letters from the American Osteopathic Association, which accredits a small
number of the GME programs in which VA medical centers participate. VA Is
Acting to Obtain

Information about Supervision from Accrediting Bodies and Residents

VA Is Taking Steps to Gain Access to Accreditation Reviews of Its
Affiliates* GME Programs

Page 21 GAO- 03- 625 VA*s Oversight of Resident Supervision stopped
sending the letters to VA, instead sending these letters only to the
institution that sponsors the GME program.

Without direct access to ACGME accreditation letters, VA medical centers
are dependent on sponsoring institutions to inform them of concerns about
the GME programs in which VA participates, and we learned of one instance
in which a sponsoring institution did not do so when ACGME notified it of
problems. Officials from a medical center told us that the sponsoring
institution of a thoracic surgery program did not tell them that ACGME had
previously identified multiple problems with the program

until ACGME decided, in September 2002, to withdraw the program*s
accreditation. ACGME did not cite any problems with the VA rotation.
Nonetheless, unanticipated withdrawal of a program*s accreditation can
affect a medical center*s educational and patient care missions. In this

case, the VA medical center will lose one full- time advanced surgical
resident in July 2003 and had to hire a physician*s assistant to provide
some of the services that had been provided by the resident.

Most medical centers indicated in their 2000/ 2001 annual reports that
their GME sponsors had shared information from accreditation letters, and
these annual reports provided network and headquarters officials with
information about accrediting bodies* concerns and medical centers*
corrective actions. Fifty- six medical centers stated that accrediting
bodies had identified concerns about VA rotations in 145 of the more than
1,900 GME programs in which VA is involved. Concerns about 17 of these
programs related to resident supervision. 32 For example, according to one
medical center*s annual report, ACGME concluded that residents required
more direct supervision during certain oncology rotations. Medical centers
reported that they had taken corrective action in response in all but one
instance. In this case, the accrediting body expressed concern that the VA
medical center had provided inadequate supervision and teaching in its

physical medicine and rehabilitation rotation, but the medical center did
not describe a corrective action in its annual report.

We found that when OAA had direct access to ACGME accreditation letters*
through early 2000* it took action to ensure that VA medical centers knew
of and responded to ACGME concerns about VA rotations.

32 The annual reports indicated that most concerns noted by GME
accrediting bodies did not involve resident supervision, but instead
involved other problems, such as insufficient ancillary staff or
inadequate rooms where residents can rest while they are on- call in the
medical center.

Page 22 GAO- 03- 625 VA*s Oversight of Resident Supervision Our review of
OAA*s correspondence about accreditation issues covering a period from
late 1998 through early 2000 indicated that ACGME mentioned concerns that
were specific to VA rotations in its letters about

17 GME programs. In 6 of these cases, ACGME cited a concern about the
adequacy of resident supervision. For example, ACGME determined that
ophthalmology residents at one VA medical center had not been given clear
information about lines of supervisory responsibility. On receipt of these
letters, OAA contacted the participating VA medical center. Three of the
medical centers submitted documents to substantiate a resolution to

the problem within 2 months of hearing from OAA. In the other three cases,
OAA asked VA*s chief consultant for the relevant medical specialty (such
as the Chief Consultant for Ophthalmology) to assess the situation. In
each case, the consultant reported to OAA that a resolution had been

achieved. For example, the consultant reported that the ophthalmology
program cited for unclear lines of supervision was preparing a written
document to clarify supervisory responsibilities.

OAA has taken steps to arrange for renewed direct access to ACGME
accreditation letters. As part of that effort, VA issued a revised policy
on confidential documents in July 2002 to make sure that accreditation

reviews would be treated confidentially. In February 2003, VA signed a
memorandum of understanding with ACGME that lays the foundation for OAA to
receive copies of accreditation letters. According to this memorandum, VA
must now obtain revised affiliation agreements between VA medical centers
and GME sponsors that authorize ACGME to provide OAA with its
accreditation letters. VA is taking steps to ensure that these revised
agreements will be in place by July 2004. OAA has come to a similar
agreement with the American Osteopathic Association.

As a further step to obtain information about, and monitor responses to,
GME issues* including accreditation concerns* OAA reissued a policy
requiring VA medical centers to establish an affiliation partnership
council and submit minutes of council meetings to OAA. 33 The council is
to include

representatives of the medical center and its academic affiliate or
affiliates and is to advise VA managers as they work to meet educational
accreditation requirements and correct deficiencies or resolve problems.

33 By reissuing this policy, OAA reasserted its requirement for submission
of minutes, which it had not consistently enforced in recent years.

Page 23 GAO- 03- 625 VA*s Oversight of Resident Supervision A mechanism
OAA uses to obtain standardized information about residents* views on the
quality of their supervision and other aspects of

their training is its Learners* Perceptions Survey, which was first
distributed in March 2001. 34 The survey asks residents to indicate their
satisfaction with the supervision they received from VA faculty by rating
supervising physicians* teaching ability, accessibility/ availability, and

approachability/ openness, as well as overall satisfaction with VA
clinical faculty. Residents are also asked to evaluate their satisfaction
with the degree of supervision and degree of autonomy they experienced.

In 2001 and 2002, VA headquarters could not send the survey to a random,
representative sample of residents from each of its medical centers
involved in GME because it did not have a complete list of its trainees.
OAA was able to obtain feedback from many residents who did receive the
survey 35 and gave those results to medical centers and networks. OAA is
taking steps to capture each trainee*s name and address in its automated
and centrally accessible information system and expects to implement this
procedure in July 2003. Once VA has a full registry of its trainees, OAA
plans to send the survey to a representative sample of residents in
different medical specialties that will include residents from all VA
medical centers involved in GME.

Medical centers* annual reports can provide network and headquarters
officials with additional information about concerns expressed by
residents and steps taken to address those concerns. According to the
annual reports for the 2000/ 2001 academic year, most VA medical centers
used VA*s nationwide Learners* Perceptions Survey or another mechanism,
such as residents* confidential evaluations obtained by sponsoring
institutions, to obtain feedback about supervision. About half of the 109
medical centers whose annual reports indicate that they had a process for
obtaining residents* feedback said that residents had concerns about their
VA rotations. None of these concerns, however, involved the

adequacy of supervision. 34 VA*s Learners* Perceptions Survey is designed
to obtain information about the perceptions of all trainees who work
within the VA system, including residents, student nurses, and psychology
interns. Data from this survey are used to assess VA*s systemwide
performance measure involving trainees* ratings of their VA educational
experience. In addition to GME, VA provides training in more than 40
associated health disciplines.

35 During 2001, surveys were sent to 3, 338 residents and returned by
1,775. During 2002, surveys were sent to 6,084 residents and returned by
2,622. VA Is Improving Its Ability

to Obtain Feedback from a Representative Group of Residents

Page 24 GAO- 03- 625 VA*s Oversight of Resident Supervision VA
headquarters, network, and medical center officials use information from
VA*s programs for monitoring the quality and outcomes of patient

care to identify and correct problems with resident supervision. VA*s
monitoring programs include its new Patient Safety Program and NSQIP.
Reviews of paid tort claims by VA*s Chief Patient Care Officer provide
another mechanism for identifying problems with resident supervision. OAA
monitors medical centers* use of these programs through the annual reports
on residency training. In their annual reports for the 2000/ 2001 academic
year, most medical centers indicated that they monitor patient care
information to determine whether resident supervision affected the

quality or outcomes of patient care. The system for reporting adverse
events and close calls established by VA*s Patient Safety Program has the
potential to capture information about instances in which inadequate
resident supervision contributed to heightened risk of adverse health care
outcomes. Based on analysis of the 17,000 reports of adverse events and
close calls filed with VA*s National Center for Patient Safety as of April
2002, its director estimated that resident supervision was mentioned* in
any context* in less than 0.1 percent of the incidents reported by VA
medical centers and that inadequate supervision was a causal factor in
very few of those cases. 36 Analyses of postoperative outcomes recorded in
the NSQIP database,

including mortality and morbidity, provide VA with a way to study the
effects of residents* involvement in surgical procedures. NSQIP personnel
analyze nationwide data from major surgeries, provide site- specific
reports to medical centers and networks, and conduct site visits at
medical

centers. 37 A NSQIP official told us that these data are routinely
examined for signs that supervision of residents might be inadequate. For
example, NSQIP analysts review the data to ensure that residents are not
performing surgeries that are more advanced than would be appropriate for
their level of training. In addition to reviewing NSQIP reports,
headquarters officials who oversee VA*s surgical services monitor the

36 We did not independently verify this estimate. 37 Each VA medical
center that performs major surgeries receives an annual report that
reports its mortality and morbidity outcomes, adjusted for risk factors,
in comparison to VA*s other medical centers, along with suggestions for
improvement. Networks also receive these reports. In addition, a team of
experts visits medical centers with mortality rates that are consistently
higher than expected to identify problems and recommend improvements. VA
Uses Its Programs

for Monitoring Patient Care to Identify and Correct Problems with Resident
Supervision

Page 25 GAO- 03- 625 VA*s Oversight of Resident Supervision frequency with
which supervising physicians are in the operating or procedural suite when
residents perform surgeries. 38 Medical center and network officials have
used NSQIP reports to help

monitor resident supervision. For example, a team of experts selected by
NSQIP visited one medical center at its request in February 2002 to help
it evaluate the efficiency of its operating rooms. During its visit, the
team noted inadequate supervision of surgeries performed by urology
residents. 39 The medical center corrected this problem by arranging for
urologists to spend more time at the medical center and ensuring that they
understood VA*s requirements for supervision. In another instance, a
network GME manager observed that NSQIP data indicated that orthopedic
surgery outcomes at a particular medical center were less favorable than
expected. After a site visit, network officials concluded that the medical
center could not support complex surgeries and determined that continued
training of orthopedic residents at that medical center would require a
decrease in the complexity of cases and greater involvement by supervising
physicians. When the sponsoring institution decided that the medical
center would not meet its training needs under those conditions, VA
officials chose to transfer patients with complex surgical needs to VA*s
tertiary hospitals in the network and shift its two VA- funded residency
slots in orthopedic surgery to a different VA medical center.

Researchers using the NSQIP database have studied ways in which
participation in GME affects postoperative outcomes. To determine whether
residency training places surgical patients at risk for worse outcomes,
researchers using the NSQIP database 40 compared risk- adjusted mortality
rates in VA*s teaching and nonteaching hospitals and found that they did
not differ, although the patients who underwent surgeries at

teaching hospitals had a higher prevalence of risk factors, underwent more
complex operations, and had longer operation times. Morbidity rates were

38 The computer software used in VA medical centers for recording
information about surgical procedures allows the generation of hospital
reports that indicate the level of supervision provided for surgical
procedures. Quarterly reports submitted to the Surgical Service at VA
headquarters also include this information. 39 There was no evidence that
any adverse patient safety events resulted from inadequate supervision of
urology residents. 40 Shukri F. Khuri and others, *Comparison of Surgical
Outcomes Between Teaching and

Nonteaching Hospitals in the Department of Veterans Affairs,* Annals of
Surgery, vol. 234, no. 3 (2001).

Page 26 GAO- 03- 625 VA*s Oversight of Resident Supervision higher in
teaching than nonteaching hospitals for some surgical specialties that
were studied. 41 On the basis of their analyses, the authors suggested
that differences in morbidity rates could reflect incomplete adjustment
for

risks, such as severity of illness, or the more complex systems of
managing and coordinating care that characterize teaching hospitals, and
not necessarily the involvement of residents. Another study begun in
September 2001 is designed to use the NSQIP database to clarify the
relationship between residents* working conditions and surgical outcomes,
with data from 90 VA hospitals and 3 nonfederal hospitals in which
surgical residents are trained. Tort claims also provide information that
VA uses to identify problems

with resident supervision that affected patient care. Review of paid tort
claims by VA*s Chief Patient Care Services Officer resulted in
clarification of VA*s written requirements for resident supervision when
patients are admitted to inpatient units. In the specific case that led to
this change, a supervising physician did not come to the hospital during a
weekend to see a patient who had been admitted by a resident; the patient
died on Monday. At that time, the resident supervision policy of the VA
hospital in which the incident occurred did not specifically require
supervising physicians to come in on weekends. As a result of this case,
in October 2001 an explicit reference to weekends and holidays was added
to the handbook*s requirement that each new inpatient be seen by the
supervising physician within 24 hours of admission.

OAA monitors incidents in which resident supervision contributed to
adverse events or patient risks through the annual reports it requires
from medical centers. In their 2000/ 2001 annual reports on residency
training, all but 11 of 114 medical centers indicated that they monitored
patient safety events associated with residents. 42 They used a variety of
processes to collect this information, including root cause analyses and
tort claim reviews, as well as additional processes such as mortality and
morbidity

conferences and reviews triggered by unexpected events, such as 41 NSQIP
defines morbidity as the occurrence of any one or more of 20 specific
postoperative adverse events such as deep wound infection, pneumonia, or
stroke within 30 days of the operation. Morbidity rates were higher in
teaching than nonteaching hospitals for general surgery, orthopedics,
urology, and vascular surgery, but did not differ significantly for
otolaryngology, neurosurgery, or thoracic surgery. 42 Medical centers that
did not describe a process for monitoring patient safety events that
involve residents either left the section on patient safety events blank
or did not describe

systematic review processes that are specific to incidents involving
residents.

Page 27 GAO- 03- 625 VA*s Oversight of Resident Supervision readmission
within 10 days of discharge from the medical center. Annual reports
indicated that reviews of at least 18 actual or potential adverse patient
outcomes at a total of 14 medical centers identified resident

supervision as a possible contributing factor or led medical center
officials to strengthen supervision to minimize the chance of future
problems. For example, one medical center established a requirement for
greater involvement by supervising physicians before a resident initiates
chemotherapy orders. Medical centers described taking corrective actions
in response to these reviews.

VA cannot assure that the residents who provide care in its facilities
receive adequate supervision because its current procedures for monitoring
supervision are insufficient. To oversee the supervision of its residents,
VA needs various types of information, including information regarding
supervising physicians* adherence to VA*s requirements for resident
supervision, accrediting bodies* and residents* concerns about
supervision, and whether the quality or outcomes of patient care indicate
problems with supervision. Systematic monitoring of each of these types of
information would help ensure that problems with resident supervision

are detected and corrected by the various officials of VA medical centers
and affiliated institutions who have responsibilities for residents*
activities.

Although VA issued a handbook that established specific standards for
resident supervision, VA does not know what its medical centers*
supervision requirements are and does not ensure that its national
requirements are adopted at each medical center where residents train.
Moreover, VA does not know whether the supervision its residents receive
adheres to its national requirements. VA*s current plans for external peer
review of documentation have the potential to enhance its oversight
capability, but these plans have not been finalized. For example, as of
May 2003, VA had not decided whether external reviewers would examine
documentation of supervision for VA*s new outpatients, who make up a
significant and growing number of VA*s patients. Including these new
outpatients in the external review could help ensure adequate supervision
of residents during a patient*s first visit to VA.

To further improve its oversight of resident supervision, VA will need to
complete its initiatives to obtain timely access to evaluations by
accrediting bodies and residents. VA will also need to continue to take
advantage of its programs for monitoring the quality and outcomes of
patient care. VA officials have generally acted to improve supervision
Conclusions

Page 28 GAO- 03- 625 VA*s Oversight of Resident Supervision when faced
with evidence of problems, and better access to information will enhance
their ability to monitor and improve resident supervision.

By strengthening its oversight capabilities, VA could help promote both
the quality of the health care in its facilities and the education its
residents receive. As the largest provider of residency training sites in
the United States, VA*s actions to enhance the quality of resident
supervision and its oversight will have benefits beyond the VA health care
system.

We recommend that the Secretary of Veterans Affairs direct the Under
Secretary for Health to take steps to improve VA*s oversight of the
supervision of residents by

 ensuring that all VA medical centers that provide GME adopt and adhere
to the requirements for resident supervision established in VA*s handbook
and  ensuring that external peer review of documentation of resident

supervision includes examination of records from VA*s new outpatients. In
written comments on a draft of this report, VA agreed with our findings
and our recommendations. VA said our report described many steps it has
already taken that would help assure systematic implementation of its
national resident supervision policies and adequate headquarters oversight
of resident supervision. In concurring with our recommendation to ensure
that all VA medical centers that provide GME adopt and adhere to
requirements for resident supervision established in its handbook, VA
indicated its intention to monitor compliance with policy requirements and
highlight those requirements in a memorandum to network officials. In
concurring with our recommendation to ensure that external peer review of
documentation of resident supervision includes examination of records from
its new outpatients, VA indicated that it would develop a strategy to
identify new outpatients who were seen by a resident. It stated that it
expects to draw its first sample of records from outpatients,

including new outpatients, in the second quarter of fiscal year 2004. VA
also reported that it completed a revision of its centralized patient
information database. This revision was necessary to allow selection of an
appropriate sample of inpatient records for external peer review. VA*s
comments are in appendix II. Recommendations for

Executive Action Agency Comments

Page 29 GAO- 03- 625 VA*s Oversight of Resident Supervision We are sending
copies of this report to the Secretary of Veterans Affairs, appropriate
congressional committees, and other interested parties. We

will also make copies available to others who are interested upon request.
In addition, the report will be available at no charge on the GAO Web site
at http:// www. gao. gov.

If you or your staff have any questions, please contact me at (202) 512-
7101. An additional contact and the names of other staff members who made
contributions to this report are listed in appendix III.

Cynthia A. Bascetta Director, Health Care* Veterans*

Health and Benefits Issues

Appendix I: Scope and Methodology Page 30 GAO- 03- 625 VA*s Oversight of
Resident Supervision To do our work, we examined oversight of resident
supervision at each of the Department of Veterans Affairs (VA) Veterans
Health Administration*s

three organization levels* headquarters, networks, and medical centers.
Our work covered VA*s oversight of resident supervision and did not
include an evaluation of the quality of care provided by residents or the
quality of the supervision provided to residents. To assess oversight by
VA*s headquarters officials, we reviewed documents and interviewed
officials from VA*s Office of Academic Affiliations (OAA), Office of
Patient Care Services, National Center for Patient Safety, Office of
Quality and Performance, and Office of Information. We analyzed VA*s plans
to have

external peer reviewers examine documentation of supervision and compared
the instructions the reviewers are to be given with VA*s requirements for
supervision.

To assess oversight of resident supervision by network officials, we
analyzed each network*s annual report to OAA on resident supervision
covering the 2000/ 2001 academic year. 1 These were the most recent annual
reports available at the time. We did not assess the accuracy of
information provided in these reports. We also interviewed network GME
managers (known as network academic affiliations officers) from a sample
of 11 of VA*s 21 regional networks of health care facilities and analyzed
documents they provided (see table 4). We used a stratified random
sampling strategy to ensure variation in the number of VA- funded
residency slots among the selected networks. 2 Network 19 was included in
our sample prior to randomization because it is the only network that did

not summarize the information in its medical centers* reports. Another
network was excluded from our sample because it had been formed by the
merger of two former networks in January 2002. Our results from these 11
networks cannot be generalized to other networks.

1 These annual reports included separate reports from two networks that
were merged in 2002. 2 Numbers of VA- funded residency slots were based on
allocations for the 2001/ 2002 academic year. Appendix I: Scope and
Methodology

Appendix I: Scope and Methodology Page 31 GAO- 03- 625 VA*s Oversight of
Resident Supervision Table 4: VA Networks Included in Our Sample Network
Number of VA- funded residency slots a

1 (Boston) 501.43 3 (Bronx) 603.00 6 (Durham) 348.30 10 (Cincinnati)
255.90 11 (Ann Arbor) 314.00 15 (Kansas City) 339.00 16 (Jackson) 671.95
18 (Phoenix) 305.70 19 (Denver) 230.00 21 (San Francisco) 383.02 22 (Long
Beach) 729.50

Source: VA. a The number of VA- funded residency slots allocated to
networks during the 2001/ 2002 academic year ranged from 195.00 to 729.50.

To assess oversight of resident supervision by medical center officials,
we reviewed and analyzed 2000/ 2001 academic year annual reports to OAA on
resident supervision. OAA provided us with 114 annual reports from the
approximately 130 VA medical centers that were involved in GME during the
2000/ 2001 academic year after it removed identifying information, such as
the names of medical centers, affiliates, and specific individuals. These
were all the medical center annual reports for the 2000/ 2001 academic
year that OAA had received as of June 18, 2002. We did not assess the
accuracy of information in the annual reports. We also interviewed GME
managers

at 11 VA medical centers (see table 5) and analyzed their 2000/ 2001
academic year annual reports on resident supervision (without redaction)
and other documents. We used a stratified random sampling strategy to
ensure that the medical centers we selected varied in the number of
VAfunded residency slots they were allocated for the 2001/ 2002 academic
year. 3 We also ensured that our sample included one medical center from
each of the networks we had sampled and that the medical centers differed
in the number of medical specialties in which their residents train. We
did not review a systematically selected sample of medical centers*

3 We excluded medical centers that received an allocation of 10 or fewer
VA- funded residency slots or with fewer than three separate GME programs
during the 2001/ 2002 academic year from our sampling set, resulting in a
possible set of 97 medical centers.

Appendix I: Scope and Methodology Page 32 GAO- 03- 625 VA*s Oversight of
Resident Supervision resident supervision policies. Our results from these
11 medical centers cannot be generalized to other medical centers.

Table 5: VA Medical Centers Included in Our Sample VA medical center
Network

Number of VA- funded residency slots a Number of

medical specialties b

White River Junction, Vt. 1 (Boston) 39.70 14 New York, N. Y. 3 (Bronx)
135.00 25 Hampton, Va. 6 (Durham) 45.00 7 Cleveland, Ohio 10 (Cincinnati)
112.40 23 Detroit, Mich. 11 (Ann Arbor) 79.00 26 St. Louis, Mo. 15 (Kansas
City) 120.00 24 Biloxi, Miss. 16 (Jackson) 10.40 6 Tucson, Ariz. 18
(Phoenix) 93.01 21 Salt Lake City, Utah 19 (Denver) 110.50 25 Fresno,
Calif. 21 (San Francisco) 42.00 4 Long Beach, Calif. 22 (Long Beach)
158.50 28

Source: VA. a The number of VA- funded residency slots allocated to
medical centers involved in GME for the

2001/ 2002 academic year ranged from 0.60 to 218.00. b The number of
distinct medical specialties in which VA medical centers had residency
slots during

the 2001/ 2002 academic year ranged from 1 to 32. We also reviewed
documentary and testimonial evidence from four medical centers that
participate in internal medicine or general surgery GME programs that had
received adverse accreditation decisions as of May 2002. 4 One of these*
the Fresno VA Medical Center* was part of our sample of medical centers.
Of the others, we visited the medical centers in West Haven, Connecticut
and Gainesville, Florida and interviewed officials of the medical center
in Albuquerque, New Mexico. We also spoke to officials of the institutions
that sponsor these three GME programs.

To obtain additional information about GME and VA*s residency training, we
analyzed accreditation requirements of the Accreditation Council for
Graduate Medical Education, American Osteopathic Association, and Joint
Commission on Accreditation of Healthcare Organizations and interviewed
officials of those bodies. We also interviewed representatives

4 Two of these programs are no longer under an adverse accreditation
status. Reevaluation of the other two programs was not complete as of May
2003.

Appendix I: Scope and Methodology Page 33 GAO- 03- 625 VA*s Oversight of
Resident Supervision of professional associations that are involved in
GME, including the American Board of Medical Specialties, American College
of Surgeons,

American Hospital Association, American Medical Association, American
Medical Student Association, Association of American Medical Colleges and
its Council of Deans, Association of Professors of Medicine, Committee of
Interns and Residents, and Council on Graduate Medical Education, and we
reviewed relevant documents issued by these groups. We interviewed
representatives of physicians who teach internal medicine, ophthalmology,
psychiatry, general surgery, orthopedic surgery, and urology* specialties
for which a large number of VA medical centers provide residency slots. We
also interviewed representatives of veterans* service organizations. We
reviewed published literature regarding the quality of care provided by
residents.

We conducted our work from September 2001 through June 2003 in accordance
with generally accepted government auditing standards.

Appendix II: Comments from the Department of Veterans Affairs Page 34 GAO-
03- 625 VA*s Oversight of Resident Supervision Appendix II: Comments from
the Department of Veterans Affairs

Appendix II: Comments from the Department of Veterans Affairs Page 35 GAO-
03- 625 VA*s Oversight of Resident Supervision

Appendix II: Comments from the Department of Veterans Affairs Page 36 GAO-
03- 625 VA*s Oversight of Resident Supervision

Appendix III: GAO Contact and Staff Acknowledgments

Page 37 GAO- 03- 625 VA*s Oversight of Resident Supervision Helene F.
Toiv, (202) 512- 7162 In addition to the person named above, key
contributors to this report

were Kristen J. Anderson, William D. Hadley, Martha Fisher, Krister
Friday, and Donald Morrison. Appendix III: GAO Contact and Staff

Acknowledgments GAO Contact Staff Acknowledgments

(290096)

The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO*s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail

this list to you every afternoon, go to www. gao. gov and select
*Subscribe to daily E- mail alert for newly released products* under the
GAO Reports heading.

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to: U. S. General Accounting Office 441 G Street NW, Room LM
Washington, D. C. 20548 To order by Phone: Voice: (202) 512- 6000

TDD: (202) 512- 2537 Fax: (202) 512- 6061

Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470 Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512-
4800

U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548 GAO*s Mission Obtaining Copies of

GAO Reports and Testimony

Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
Programs Public Affairs
*** End of document. ***