Human Factors: FAA's Guidance and Oversight of Pilot Crew Resource
Management Training Can Be Improved (Letter Report, 11/24/97,
GAO/RCED-98-7).
Pursuant to a congressional request, GAO examined the role of airline
pilots' performance in accidents and the Federal Aviation
Administration's (FAA) efforts to address any inadequate performance,
focusing on the: (1) types and frequency of accidents in which an
airline pilot's performance was cited as a contributing factor,
including those in which failure to use crew resource management (CRM)
principles was identified; and (2) adequacy of FAA's guidance for and
oversight of the airlines' implementation of pilots' training for CRM.
GAO noted that: (1) of the 169 accidents that involved the major
airlines and that were investigated and reported on in detail by the
National Transportation Safety Board (NTSB) from 1983 through 1995,
about 30 percent were caused in part by the pilots' performance; (2) in
at least one-third of these accidents, GAO determined that the pilots
did not correctly use CRM principles; (3) for example, according to
NTSB, just before the 1994 crash in Charlotte, North Carolina, which
killed 37 people, the aircraft had encountered a sudden change in wind
direction and the captain gave an incorrect order to the first officer,
who did not question the order, as CRM principles would require; (4)
during the same period, of the nearly 4,000 incidents, GAO found that
about one-fifth were caused in part by the pilots' performance; (5)
FAA's guidance for and oversight of training in CRM does not ensure the
adequacy of this training under part 121 of the federal aviation
regulations, while they do under the new Advanced Qualification Program
(AQP); (6) FAA's guidance for the implementation of AQP specifies a
process for curriculum development that the airlines must follow in
order to integrate CRM training with technical flying skills; (7) FAA
inspectors overseeing this training assess the curriculum to see if
FAA's process has been followed, enabling them to determine whether the
pilots' training under this curriculum is adequate; (8) although FAA
requires airlines to teach CRM in their traditional part 121 training,
the guidance it provides on how to develop the curriculum for this
training is ambiguous and does not provide standards that inspectors can
use to evaluate airlines' CRM training; (9) because AQP training
generally differs from traditional part 121 training in how it develops
a curriculum for training CRM, the guidance for this training in AQP may
not be applicable to CRM training under part 121; (10) FAA needs to
develop guidance for teaching CRM under traditional part 121 training;
(11) and although 8 of the 10 major airlines plan to train all their
pilots under AQP, the need for guidance on CRM training under part 121
remains--both for those airlines that have opted not to enter AQP as
well as for those that participate in the program but will nonetheless
continue to have some of their pilots trained under part 121 for up to 8
years as they make the transition to AQP.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-98-7
TITLE: Human Factors: FAA's Guidance and Oversight of Pilot Crew
Resource Management Training Can Be Improved
DATE: 11/24/97
SUBJECT: Airline industry
Commercial aviation
Safety regulation
Human resources training
Aircraft pilots
Transportation safety
Aircraft accidents
Personnel management
Flight crews
IDENTIFIER: FAA Advanced Qualification Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
November 1997
HUMAN FACTORS - FAA'S GUIDANCE AND
OVERSIGHT OF PILOT CREW RESOURCE
MANAGEMENT TRAINING CAN BE
IMPROVED
GAO/RCED-98-7
Human Factors
(341513)
Abbreviations
=============================================================== ABBREV
AQP - Advanced Qualification Program
ASRS - Aviation Safety Reporting System
CRM - crew resource management
FAA - Federal Aviation Administration
GAO - General Accounting Office
NASA - National Aeronautics and Space Administration
NTSB - National Transportation Safety Board
Letter
=============================================================== LETTER
B-275381
November 24, 1997
The Honorable John McCain
Chairman, Committee on Commerce,
Science, and Transportation
United States Senate
The Honorable Slade Gorton
Chairman, Subcommittee on Aviation
Committee on Commerce, Science, and
Transportation
United States Senate
Previous studies of aviation safety have found that pilot performance
is a major contributor to airline accidents and incidents (events
that affect or could affect a flight's safety). Therefore, training
to improve pilots' performance has been a primary effort to improve
airline safety. As part of this effort, some airlines have provided
training in crew resource management (CRM) since the early 1980s, and
the Federal Aviation Administration (FAA) will require all airlines
to have implemented this training for pilots by March 1998. CRM is
an approach to improving pilot performance that focuses on better
coordination--among members of the cockpit crew as well as among the
cockpit crew and flight attendants, dispatchers, and air traffic
controllers--to handle certain routine and emergency situations.
Airlines can meet the CRM training requirement in one of two ways:
(1) by following FAA's traditional requirements for training pilots
and crew--specified in part 121 of the federal aviation regulations\1
--or (2) by instituting the Advanced Qualification Program (AQP),\2
which combines CRM training with technical training for pilots. Part
121 training requirements have been in place without significant
modification since the 1970s, and until 1990, all airlines had to
meet these requirements. Since 1990, FAA has offered airlines AQP
training as an alternative to traditional part 121 training, and
eight major airlines have chosen to train their pilots under AQP
requirements.
This report responds to your request that we examine the role of
airline pilots' performance in accidents and FAA's efforts to address
any inadequate performance. Specifically, we agreed to address the
following: (1) What are the types and frequency of accidents in
which an airline pilot's performance was cited as a contributing
factor, including those in which failure to use CRM principles was
identified, and (2) how adequate is FAA's guidance for and oversight
of the airlines' implementation of pilots' training for CRM? We
limited our review to the accidents and incidents experienced and
training implemented by the 10 major U.S. airlines--those generating
$1 billion or more in revenues annually.\3
--------------------
\1 14 C.F.R. part 121, subparts N and O.
\2 Special Federal Aviation Regulation No. 58--Advanced
Qualification Program.
\3 These airlines are Alaska, American, America West, Continental,
Delta, Northwest, Southwest, Trans World, United, and US Airways.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
Of the 169 accidents that involved the major airlines and that were
investigated and reported on in detail by the National Transportation
Safety Board from 1983 through 1995, about 30 percent were caused in
part by the pilots' performance, according to our analysis. In at
least one-third of these accidents (about 15), we determined that the
pilots did not correctly use the principles of crew resource
management. For example, according to the National Transportation
Safety Board, just before the 1994 crash in Charlotte, North
Carolina, which killed 37 people, the aircraft had encountered a
sudden change in wind direction and the captain gave an incorrect
order to the first officer, who did not question the order, as crew
resource management principles would require. Furthermore, during
the same period, of the nearly 4,000 incidents, we found that about
one-fifth were caused in part by the pilots' performance.
FAA's guidance for and oversight of training in crew resource
management does not ensure the adequacy of this training under part
121, while they do under the new Advanced Qualification Program.
FAA's guidance for the implementation of the Advanced Qualification
Program specifies a process for curriculum development that the
airlines must follow in order to integrate training in crew resource
management with technical flying skills.\4 FAA inspectors overseeing
this training assess the curriculum to see if FAA's process has been
followed; this assessment also enables them to determine whether the
pilots' training under this curriculum is adequate. In contrast,
although FAA requires airlines to teach crew resource management in
their traditional part 121 training, the guidance it provides on how
to develop the curriculum for this training is ambiguous and does not
provide standards that inspectors can use to evaluate airlines'
training in crew resource management. Because the Advanced
Qualification Program training generally differs from traditional
part 121 training in how it develops a curriculum for training in
crew resource management, the guidance for this training in the
Advanced Qualification Program may not be applicable to training for
crew resource management under part 121. Therefore, FAA needs to
develop guidance for teaching crew resource management under
traditional part 121 training. Furthermore, although 8 of the 10
major airlines plan to train all their pilots under AQP, the need for
guidance on crew resource management training under part 121
remains--both for those airlines that have opted not to enter the
Advanced Qualification Program as well as for those that participate
in the program but will nonetheless continue to have some of their
pilots trained under part 121 for up to 8 years as they make the
transition to the Advanced Qualification Program.
--------------------
\4 Initially, crew resource management was known as "cockpit resource
management" and referred only to individuals on the flight deck--that
is, to pilots and flight engineers.
BACKGROUND
------------------------------------------------------------ Letter :2
Airline travel is one of the safest modes of public transportation in
the United States. The current level of airline safety has been
achieved, in part, because the airline industry and government
regulatory agencies have implemented rigorous pilot training and
evaluation programs. The major airlines have training programs for
pilots that focus on, among other things, maintaining flying skills,
qualifying to fly new types of aircraft, and acquiring skills in
dealing with emergencies.
FAA's original regulations for the airlines' general training
programs--referred to in this report as part 121--spell out the
number of hours of training required in particular areas, such as the
time spent practicing emergency procedures. Effective for 1996, FAA
instituted a requirement for CRM training under part 121 that states
the following:
"After March 19, 1998, no certificate holder [airline] may use a
person as a flight crewmember, and after March 19, 1999, no
certificate holder may use a person as a flight attendant or aircraft
dispatcher unless that person has completed approved crew resource
management or dispatcher resource management initial training, as
applicable, with that certificate holder or with another certificate
holder."\5
FAA believes that this training should improve flight crews'
performance.\6
As an alternative to training under these regulations, airlines may
apply to participate in AQP.\7 Eight of the 10 major airlines have
applied to, and been approved for participation in, AQP. Unlike
traditional part 121 training, AQP specifies the criteria for the
required level of performance in certain types of maneuvers, rather
than hours of training, and it integrates CRM training with technical
flying skills. The airlines are expected to fully implement AQP over
a period of time, up to 8 years. Full implementation means that the
airlines have trained their pilots for each type of aircraft they
fly. Training, however, occurs only after the airline has gone
through three other stages: (1) getting approval to participate in
the program, (2) developing a training curriculum, and (3) training
instructors. Continuing crew training, the last stage, is to occur
annually.
Responsibility for AQP and traditional part 121 training rests with
different FAA branches. The AQP Branch within the Office of Flight
Standards Services oversees AQP, and the Branch expects to transfer
many of its oversight responsibilities to inspectors in the field as
each airline fully implements its AQP. The administration of
traditional part 121 training is divided between the Air Carrier
Training Branch, which sets training requirements, and the flight
standards inspectors in the field, who are responsible for overseeing
the training. FAA's inspectors periodically review and approve
airlines' curricula and training materials and observe training.
CRM is a "human factors" approach for improving aviation safety by
preventing or managing pilots' errors. Human factors refers to a
multidisciplinary effort to develop information about human
capabilities and limitations and to apply this information to
equipment, systems, facilities, procedures, jobs, environments,
training, staffing, and personnel management for safe and effective
human performance. Under this approach, pilots are trained to
recognize potential mistakes in judgment or actions and to compensate
for them to prevent accidents and incidents.\8
For example, in training for initial departure, CRM training has the
captain practice briefing the crew about the actions to be taken if
the takeoff must be aborted because of an emergency. CRM also
teaches the crew to question orders when they believe they have
information that indicates these orders are inappropriate.
Similarly, CRM training teaches the crew to anticipate problems and
make decisions that take these anticipated problems into account.
--------------------
\5 14 C.F.R., section 121.404.
\6 This requirement applies to all airlines operating under part 121
and those airlines certified under part 135 that conduct training
under part 121. Airlines now operating under part 121 use aircraft
configured for 10 or more passengers. New rules adopted by FAA in
1995 require certain commuter operators conducting scheduled
operations under part 135 to conduct those operations under part 121
beginning in March 1997. Included were those airlines conducting
scheduled operations carrying passengers with aircraft configured for
10 to 30 seats.
\7 FAA issued the final Special Federal Aviation Regulation (SFAR) 58
for AQP on October 2, 1990 and the termination date for the
regulation has been extended to October 2, 2000. The SFAR 58 is
found in part 121. The AQP advisory circular was issued on August 9,
1991.
\8 The National Transportation Safety Board (NTSB), the official
source of information on airline accidents, defines accidents as
occurrences in which individuals are killed or suffer serious injury,
or the aircraft is substantially damaged. By NTSB's definition,
accidents can range from fatal crashes in which all on board are
killed to events in which only one person suffers a broken bone and
the aircraft is not damaged, to still others in which there is
substantial aircraft damage but no fatalities or serious injuries.
NTSB generally distinguishes between accidents and incidents. NTSB
defines incidents as occurrences other than accidents associated with
the operation of an aircraft that affect or could affect the safety
of operations. (49 C.F.R. section 830.2)
AIRLINE PILOTS' PERFORMANCE WAS
A CONTRIBUTING FACTOR IN MANY
ACCIDENTS AND INCIDENTS
------------------------------------------------------------ Letter :3
About 30 percent of the 169 accidents and 18 percent of the 3,901
incidents that occurred from 1983 through 1995 were caused at least
in part by pilots' performance, according to our analysis of the
National Transportation Safety Board's (NTSB) and FAA's data.
Furthermore, the accident data indicate that nearly one-third of the
accidents occurred because the pilots either did not follow, or did
not correctly follow, CRM principles. The most frequently occurring
accidents and incidents included collisions on the ground with
objects and other airplanes, flights through turbulent weather that
resulted in injuries, and deviations from flight paths that had the
potential to cause an in-flight collision.
DATA SHOW PILOTS'
PERFORMANCE CONTRIBUTED TO
ACCIDENTS AND INCIDENTS
---------------------------------------------------------- Letter :3.1
On the ground, pilot performance was associated most frequently with
airplanes colliding with vehicles, buildings, other equipment, or
animals. This was the case for both accidents (32 percent) and
incidents (34 percent). Figure 1 shows the types of accidents and
incidents on the ground reported from 1983 through 1995.
Figure 1: On-the-Ground
Accidents and Incidents
Associated With Pilots'
Performance
(See figure in printed
edition.)
\a NTSB cited 62 events associated with pilots' performance in 169
accidents.
\b FAA cited 446 events associated with pilots' performance in 3,901
incident reports.
Sources: FAA and NTSB.
In the air, pilot performance was most frequently associated with
injuries to passengers and flight attendants during turbulent
weather--41 percent of accidents and 12 percent of incidents. Figure
2 shows the types of accidents and incidents in the air that were
reported.
Figure 2: In-the-Air Accidents
and Incidents Associated With
Pilots' Performance
(See figure in printed
edition.)
Note: Amounts may not add to 100 percent due to rounding.
\a NTSB cited 46 events associated with pilots' performance in 169
accidents.
\b FAA cited 209 events associated with pilots' performance in 3,901
incidents.
Sources: FAA and NTSB.
In addition to the accidents and incidents discussed above, FAA
maintains data separately for those occasions on which pilots failed
to comply with the air traffic controller's instructions--such as not
staying on the directed flight path and/or entering a runway without
clearance.\9 Of the 1,471 unauthorized maneuvers from 1987 through
1995,\10 80 percent occurred in the air, and most of these (73
percent) occurred when pilots did not maintain their assigned
altitude levels. The unauthorized pilot maneuvers on the ground were
most often (69 percent) associated with pilots' moving airplanes onto
runways without authorization from the air traffic control tower.
These types of incidents have the potential to cause accidents. For
example, the December 1990 crash at the Detroit Metropolitan Airport
occurred when an airplane taxied onto a runway being used for takeoff
by another airplane and collided with that airplane. Twelve people
died. The first plane had not gotten permission from the control
tower to enter this runway, as it should have. Figure 3 shows the
most frequently reported unauthorized pilot maneuvers in the air and
on the ground.
Figure 3: Most Frequently
Reported Unauthorized Pilot
Maneuvers in the Air and on the
Ground
(See figure in printed
edition.)
\a FAA reported 1,110 unauthorized pilot maneuvers in the air.
\b FAA reported 258 unauthorized pilot maneuvers on the ground.
Source: FAA's pilot deviation database.
--------------------
\9 These data are found in FAA's pilot deviation database.
\10 These data were available from FAA's National Aviation Safety
Data Analysis Center only for these years.
DEFICIENCIES IN CRM
CONTRIBUTED TO ACCIDENTS
---------------------------------------------------------- Letter :3.2
In our analysis of accidents, we found deficiencies in the airline
pilots' use of CRM in nearly one-third of all accidents involving
pilots' performance. Moreover, we found CRM deficiencies in half of
the serious accidents in which there was at least one fatality.
About 46 percent of these CRM deficiencies involved a lack of
coordination among members of the cockpit crew, as well as the
captain's failure to assign tasks to other crew members and to
effectively supervise the crew. Generally, these CRM deficiencies
illustrated the importance of effective communication.
For example, in the Charlotte, North Carolina, crash in July 1994,
communication among crew members did not occur, according to NTSB's
accident investigation report. NTSB believes that the captain, who
was not flying the aircraft at the time and could not see the ground
because of poor visibility, became disoriented and commanded the
first officer, "down, push it down," even though they were
encountering windshear, which is a sudden change in wind direction.
The first officer did not question the order, as he should have,
according to NTSB, because the windshear was creating an unstable
situation; the plane could not recover from the sudden downward shift
in direction caused by following the captain's order. The plane
crashed nose down into the ground, and 37 people died.
Similarly, in a June 1984 accident in Detroit, Michigan, a lack of
communication between the crew and air traffic controllers during a
landing in a severe thunderstorm contributed to the accident,
according to the NTSB report. The crew did not request clarification
about the weather conditions or change its course of action to take
these conditions into account. The winds associated with the storm
forced the plane down precipitously, causing an emergency landing
without the landing gear's being fully extended. The plane skidded
off the runway, causing serious damage to the aircraft and an
emergency evacuation of the passengers. NTSB reported that the lack
of CRM practices was a probable cause of the accident.
The National Aeronautics and Space Administration reported similar
results in its analysis of pilot reports submitted to its voluntary
reporting system.\11
Nearly half of the reports cited deficiencies in the pilots' use of
CRM principles; about 53 percent of the CRM deficiencies concerned
coordination among members, assignment of tasks, and crew
supervision.
--------------------
\11 An Analysis of part 121 Crew Resource Management Incidents,
National Aeronautics and Space Administration, Quick Response No.
296, Aviation Safety Reporting System (Feb. 6, 1997).
FAA'S GUIDANCE AND OVERSIGHT DO
NOT ENSURE EFFECTIVENESS OF CRM
TRAINING
------------------------------------------------------------ Letter :4
For AQP training, FAA has specified the process airlines need to
follow to develop and implement a curriculum that integrates CRM
concepts with technical flying skills, but FAA's guidance for CRM
training under part 121 does not have the same degree of specificity.
As a result, inspectors overseeing training under part 121 do not
have standards they can use to evaluate airlines' CRM training
curriculum and the delivery of that training. Generally, inspectors
could not use the guidance provided under AQP to evaluate part 121
training for the CRM curriculum because the curricula developed under
the two programs differ significantly. As a result, airlines
continue to need specific guidance for CRM under part 121--both those
airlines that have opted not to enter AQP as well those that will
continue to train at least some of their crews under part 121 until
they have fully implemented AQP, which could take up to 8 years.
FAA'S GUIDANCE FOR CRM
TRAINING IS DETAILED UNDER
AQP BUT NOT UNDER PART 121
---------------------------------------------------------- Letter :4.1
Once an airline elects to participate in AQP, it must follow SFAR 58
(the AQP regulation) for developing a formal curriculum--including
assessing the skills pilots need to safely operate the aircraft they
fly, developing curriculum objectives for teaching those skills,
having measurable criteria for evaluating whether the pilots have
achieved those objectives, and developing materials to teach those
objectives. FAA must approve this curriculum. Furthermore, AQP
requires all airlines to train their pilots in simulators so that
they gain experience with a number of emergency situations. Finally,
airlines must submit data to FAA demonstrating that their crews have
mastered the skills they need to fly for those airlines.\12
In developing its AQP curriculum, an airline is required to integrate
CRM training into every aspect of its crews' training. As a result,
the pilots trained under AQP are assessed on CRM principles as well
as on technical flying skills. For example, when a pilot changes the
aircraft's altitude--a technical flying skill--CRM principles dictate
that this pilot inform the other pilot by verbally announcing the new
altitude while continually pointing to the altitude indicator until
the other pilot also points to the altitude indicator and repeats the
new altitude. This procedure is used to ensure that neither pilot
will fail to maintain the appropriate altitude.
In contrast, FAA's requirements for CRM training under part 121 do
not require airlines to develop a curriculum for CRM training with
measurable criteria or to integrate that curriculum with other
aspects of part 121 training. For the CRM curriculum under part 121,
FAA provides suggested training topics but does not clearly lay out
how the airlines are to introduce these topics into their training
programs, according to airline officials and FAA inspectors. For
example, FAA recommends that airlines train crews in "workload
management and situational awareness." For this training, FAA
suggests such topics as "preparation/planning/vigilance" and
"workload distribution/distraction avoidance." However, for those
airlines that choose to integrate these topics with technical flying
skills, FAA does not explain how the airlines are to do so.\13
The lack of specificity in FAA's guidance for the development of a
CRM curriculum under part 121 contrasts with the detailed guidance
FAA provides for the development of a curriculum on technical flying
skills. For example, FAA's guidance on how pilots are to respond to
windshear under part 121 directs them in a number of technical flying
skills, such as how to handle the rudder, but it is silent on how to
employ CRM principles in this situation. In contrast, under AQP,
FAA's guidance instructs the airlines to specify not only the
technical skills but also the CRM principles that must be applied in
a windshear situation.
Because FAA's guidance on CRM training under part 121 is less
specific, airlines vary in how they deliver their CRM training.
While all the airlines provide classroom training in CRM principles
under part 121 training, they may not integrate this training with
technical flying skills. For example, airlines may (1) train pilots
in technical flying skills in flight simulators without integrating
CRM principles or (2) integrate CRM principles with technical flying
skills in flight simulators. Generally, we found that CRM training
had been integrated with technical flight training to a higher degree
at those airlines that were in later phases of AQP implementation.
--------------------
\12 Training records for pilots are maintained under part 121.
Performance data for crews submitted to FAA under AQP permit the
agency to conduct its own analyses of crews' mastery.
\13 See FAA's Advisory Circular 120-51B, Crew Resource Management
Training.
FAA'S OVERSIGHT OF CRM
TRAINING IS ADEQUATE FOR AQP
BUT NOT FOR PART 121
---------------------------------------------------------- Letter :4.2
In developing AQP, FAA incorporated procedures for evaluating CRM
training and developed a process for ensuring that FAA inspectors
would have the criteria they need to conduct the evaluations for
pilots' training on different types of aircraft. Specifically, AQP
provides a systematic way of identifying the tasks and subtasks
involved in a particular phase of flight. Therefore, an inspector
observing the training program can determine whether CRM principles
are being invoked in a given flight situation. For example, when a
crew is preparing for landing, AQP specifies that the first officer,
if unsure of the planned course of action in the event of a missed
approach, is to ask the captain to clarify the plan so that both have
a full understanding of the actions they will take. Similarly, if a
flight has to be diverted from one airport to another, the captain is
to direct the first officer to (1) get out the maps for the alternate
airport, (2) notify the flight attendants, and (3) make the
announcement to the passengers. This delegation of tasks allows the
captain time to handle radio contact with the airline's dispatchers
and air traffic controllers, obtain weather updates at the alternate
airport, and fly the plane.
In the early stages of AQP implementation, the AQP Branch is
evaluating airlines' training. FAA will transfer this responsibility
to inspectors in the field as airlines fully implement AQP. Field
inspectors will be trained in evaluating the CRM training as an
integral part of their evaluation of AQP training. The inspectors at
those airlines that had progressed beyond the initial phases of AQP
noted that they had received AQP training at the airlines for which
they were responsible. Moreover, all of the inspectors we spoke with
maintained that while certain facets of AQP were fixed, some parts
were still evolving. As a result of the program's flexibility and
evolution, the inspectors pointed out that it was not possible to
structure a training program for them that could cover every aspect
of AQP at every airline. Despite this fluidity, these inspectors
said that the AQP Branch Office made sure that the program's
standards were maintained across airlines.
While the evaluation of the delivery of CRM training is incorporated
into the oversight process for AQP training, it is not under
traditional part 121 training. Moreover, FAA has not provided its
inspectors with any specific guidance or training for evaluating
airlines' CRM training under part 121. Although FAA inspectors may
obtain some CRM training from a 3-hour computerized interactive
course, this lack of guidance for evaluating CRM training under part
121 is troublesome to the inspectors we spoke with because of what
they view as an inherent conflict between performance expectations
for individuals under part 121 and crew performance expectations
articulated in CRM principles.\14
Under part 121, pilots are to master technical flying skills and
perform these skills without reliance on any other crew member. In
contrast, CRM principles and training teach pilots how to use to
maximum effect the abilities and experience of other crew members, as
well as their own technical flying skills.
Without formal FAA instructions, inspectors have developed their own
approaches to this evaluation. For example, one inspector said that
he based his approval on his belief that the airline for which he was
responsible "had a good safety record" and "would probably establish
a good program." Another inspector said that in approving any
training program, he sought guidance first from any applicable
federal aviation regulation; the Inspector's Handbook, applicable
advisory circulars; and, finally, any other FAA publication, such as
the Introduction to CRM Training. However, this inspector added that
these sources did not provide the criteria he needed to evaluate CRM
training. As a result, he looked for behaviors such as crew members'
"working together" to resolve problems, "catching errors," or
"dealing with the consequences resulting from uncaught errors."
These ad hoc approaches to evaluating the delivery of CRM training
are not satisfactory to FAA officials at headquarters or to officials
for at least one airline. FAA officials told us that the agency
needed additional CRM training for its inspectors conducting reviews
under part 121. In addition, officials at one airline told us that
the lack of specific guidance and training for FAA inspectors
responsible for evaluating CRM training under part 121 has hampered
FAA's ability to review CRM programs. Furthermore, the problems FAA
inspectors face in evaluating CRM training under part 121 will
continue indefinitely in the absence of clearer guidance from FAA for
those airlines that have decided not to enter AQP and for those
airlines in the program that have not fully implemented it.
Because AQP is implemented by the type of aircraft the crew flies,
even the airlines that have been accepted for AQP will continue to
provide some CRM training under part 121. For the eight airlines
implementing AQP, we estimate that only about one-third of their
crews have begun to receive AQP training. Therefore, most crews are
still receiving traditional training under part 121, and some will
continue do so for up to 8 years. As of September 1997, the
airlines' estimated dates for completing the transition to AQP
training ranged between 2000 and 2005. (See table 1.)
Table 1
Estimate of Completed AQP Implementation
by Airline
Pilots being
trained
under AQP as Estimated
of September year of
Airline Year begun 1997 completion
---------------------------- ------------ ------------ ------------
Alaska 1994 0 2002
American 1994 0 2005
Continental 1997 0 2002
Delta 1992 60% 2001
Northwest 1994 23% 2001
Trans World 1995 0 2000
United 1991 50% 2001
US Airways\a 1994 0 2000
----------------------------------------------------------------------
\a Estimate is for one aircraft type. US Airways' aircraft fleet
composition is under review, and no determination has been made to
place any other aircraft types into AQP until the aircraft fleet plan
is resolved.
--------------------
\14 In a previous review of FAA's training for its inspector
workforce--Aviation Safety: Targeting and Training of FAA's Safety
Inspector Workforce (GAO/T-RCED-96-26, Apr. 30, 1996)--we found that
some inspectors were unaware that needed training was available
through computer-based courses.
CONCLUSIONS
------------------------------------------------------------ Letter :5
For the flying public, safety is the paramount issue, and FAA and the
airlines have worked to provide rigorous training programs for
pilots. Crew resource management, which focuses on making the best
use of all available experience and skills in the cockpit, is
increasingly seen as an important component of safe flights. FAA
recognized the importance of crew resource management by requiring
all airlines to include training in these principles and by
incorporating crew resource management into its Advanced
Qualification Program.
Pilots' performance is not the only factor in airline accidents, but
it is an important one. We identified pilots' performance as the
cause of about one-third of all the accidents and nearly one-fifth of
the incidents for the 10 major airlines from 1983 through 1995.
Training for safer performance by pilots that teaches crew resource
management can occur under either the Advanced Qualification Program
or part 121. However, while FAA's guidance for the implementation of
the Advanced Qualification Program specifies a process for curriculum
development that integrates this training with training in technical
flying skills, FAA's guidance for curriculum development under part
121 is ambiguous and does not provide standards that inspectors can
use to evaluate and approve airlines' training in crew resource
management. As a result, FAA cannot be assured that airlines are
developing a curriculum for teaching crew resource management that
will effectively teach pilots how to best use all the skills and
experience available to them in the cockpit. Furthermore, without
specificity in the development of training for crew resource
management under part 121 and without any guidance on how to evaluate
this training under part 121, FAA inspectors are relying on their own
experience in observing pilots or even on the belief that the airline
"would probably establish a good program." These problems are
especially troublesome because pilots who have not completed
FAA-approved crew resource management training by March 1998 may not
fly for airlines.
RECOMMENDATIONS
------------------------------------------------------------ Letter :6
To help ensure that airlines appropriately train pilots in CRM
principles under part 121 and that FAA inspectors are able to
uniformly evaluate this CRM training, we recommend that the Secretary
of Transportation direct the Administrator of FAA to develop a
process that airlines must follow for creating a CRM curriculum, with
measurable criteria, under part 121 as it has for the Advanced
Qualification Program.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :7
We provided a draft of this report to FAA for review and comment. We
met with the Deputy Associate Administrator for Regulation and
Certification, the Deputy Director of Flight Standards Services, the
Managers for the Air Carrier Training Branch and the Advanced
Qualification Program, and other officials. FAA commended our review
of CRM training at the nation's airlines. FAA accepted the report's
recommendation in part. FAA agreed that it should ensure that pilots
are appropriately trained and noted that CRM training can provide
desirable consequences in aviation safety. It further agreed that
uniform evaluation of CRM training using measurable criteria is a
commendable objective. However, FAA stated that science has not yet
developed valid, reliable criteria for measuring CRM performance.
FAA also agreed that more can be done to develop a process that
airlines and inspectors can follow to create a CRM curriculum. FAA
indicated that better guidance would be provided in a number of ways,
such as updating Advisory Circular 120-51, Crew Resource Management
Training, and supplemental guidance for inspectors included in the
inspectors' handbook and holding regional meetings with CRM
specialists from Flight Standards Services and other organizations.
We concur with FAA that CRM training for pilots could improve
aviation safety. However, we believe that before the contribution of
CRM training to aviation safety can be measured, it is necessary to
determine the extent to which the delivery of CRM training for pilots
has occurred. We further concur with FAA that more should be done to
develop processes for airlines and inspectors to follow in creating a
CRM curriculum. We believe that until FAA establishes a process for
CRM curriculum development that includes an assessment of the extent
to which pilots have mastered that curriculum, it will not be
possible to measure CRM's performance in contributing to aviation
safety.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :8
To determine the extent to which inadequate performance by pilots was
a problem for the 10 major U.S. airlines, we examined the types and
frequency of safety-threatening events--incidents and accidents--from
1983 through 1995.
To determine the adequacy of FAA' s guidance for and oversight of
pilots' training, we reviewed FAA's role in the airlines'
implementation of CRM. We focused primarily on CRM training because
FAA has described the failure to apply CRM principles as a more
important contributing factor in accidents than technical flying
skills. We also compared FAA's rules and regulations and other
guidance for CRM training with that provided for other training
programs, as well as interviewed FAA and airline officials. A
detailed discussion of our methodology is presented in appendix I.
Related GAO products are listed at the end of this report.
Our work was performed from October 1996 through October 1997 in
accordance with generally accepted government auditing standards.
---------------------------------------------------------- Letter :8.1
As arranged with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 30 days after the date of this letter. At that time, we will
provide copies of the report to the Secretary of Transportation,
appropriate congressional committees, and other interested parties.
We will also make copies available to others upon request.
Please contact me at (202) 512-3650 if you or your staff have any
questions about this report. Major contributors to this report are
listed in appendix II.
Gerald L. Dillingham
Associate Director, Transportation
Issues
METHODOLOGY
=========================================================== Appendix I
To identify the types and frequencies of accidents and
incidents--safety-threatening events--related to pilot performance,
we reviewed accident and incident data, including pilot deviations,
contained in the National Transportation Board's (NTSB) and the
Federal Aviation Administration's (FAA) electronic databases. We
obtained these data from FAA's National Aviation Safety Data Analysis
Center. We limited our review to the reported events in accident and
incident data sources involving the 10 major U.S. passenger airlines
from 1983 through 1995. We did not independently verify these data.
To facilitate the comparison of accidents with incidents in our
analysis of the types and frequencies of safety-threatening events,
we made two adjustments to the data. First, because of differences
in the way information is recorded in these databases, we matched the
similar categories contained in both databases and used these
categories in our analysis. For example, both NTSB's and FAA's
databases contain the category "on ground collision with object,"
which means an airplane struck an object, such as a vehicle or
structure, while moving on the ground. Second, because the
occurrences of events in accidents closely conform to those in
incidents, we used the events that occurred in each of the 169
accidents as our unit of analysis. In our analysis of crew resource
management (CRM) deficiencies, we used the accident as the unit of
analysis because NTSB's findings of CRM deficiencies were by accident
and not by the individual events that occurred within accidents.
To characterize the prevalence of pilot performance as a factor in
safety-threatening events over time and between airlines, we examined
FAA's incident and pilot deviation databases. We used these two
databases because they are the only such sources with adequate
numbers of observations to make such comparisons.
To determine the extent to which the inadequate use of CRM by pilots
contributed to accidents and incidents, we performed a content
analysis of the textual information found in the factual reports,
briefs, and final reports of the 169 accidents investigated by NTSB
from 1983 through 1995. We then classified CRM deficiencies
according to the classification framework presented at a National
Aeronautics and Space Administration (NASA)/Ames workshop in 1980.\15
This framework groups CRM issues into five broad clusters:
-- (1) Resource management--the application of specialized
cognitive skills to effectively and efficiently utilize
available resources, such as the ability to plan, organize, and
communicate.
-- (2) Organization processes--crew members' actions and behaviors
in the context of their assigned duties and expected
responsibilities.
-- (3) Personal factors--the knowledge, skills, abilities, and
limitations that individual crew members bring with them to the
cockpit.
-- (4) Material resources internal to the aircraft--the cockpit
crew's appropriate, effective, and efficient use of
instructional items, such as checklists, and navigational charts
and equipment, such as on-board weather radar, navigational
controls, and engine fire extinguisher.
-- (5) Resources external to the aircraft--those people (air
traffic controllers), entities (airports), and circumstances
(emerging poor weather) that may affect pilots' plans,
decisions, and actions.
Table I.1 shows the classification framework used to categorize CRM
issues.
Table I.1
Classification Framework for
Categorizing CRM Issues
Clusters
--------------------------------------------------------------------------
Material
resources Resources
Resource Organization Personal internal to external to
management processes factors the aircraft the aircraft
------------- -------------- ------------ -------------- ------------ --------------
CRM issues �Social Role Knowledge Textual Human
Monitoring Proficiency Equipment Facility
�Communication Workload Experience Environment
�Leadership Motivation
�Management Stress
�Planning reaction
�Problem- Fatigue
solving
�Decision-
making
-----------------------------------------------------------------------------------------
To verify the results of our content analysis, we requested a similar
analysis by NASA's Aviation Safety Reporting System (ASRS) staff of
voluntarily submitted pilot reports contained in the ASRS database.
According to the aviation experts we consulted, ASRS incident reports
provide the best source of information on deficiencies in CRM.
Furthermore, because ASRS staff are most familiar with the data and
have expertise in analyzing this free-form data, we concluded that it
was more appropriate for them to perform this analysis.
To evaluate the adequacy of FAA's oversight of airline pilot
training, we obtained FAA's training policies, requirements,
guidance, and handbooks relevant to CRM training. We discussed
training programs, including CRM, and training procedures with
appropriate FAA officials, including officials in the Office of
System Safety, the Office of Regulation and Certification's Flight
Standards Services, the Advanced Qualification Program Branch, the
Office of Accident Investigation, and the Human Factors Division. In
addition, we discussed airline training evaluation and approval
processes and obtained training documents from FAA inspectors
responsible for monitoring airline training. Finally, we contacted
safety directors and trainers at the major airlines and obtained
documents on their policies, procedures, research, and training
curricula.
We requested comments from recognized experts in the field of human
factors in academia and the aviation industry, pilots, and government
officials from FAA, NTSB, and NASA. We incorporated their comments
where appropriate and made adjustments to our methodology as
warranted.
--------------------
\15 Murphy, M.R. (1980). "Analysis of Eighty-four Commercial
Aviation Incidents: Implications for a Resource Management Approach
to Crew Training." 1980 Proceedings Annual Reliability and
Maintainability Symposium. Ames Research Center, NASA, Moffet Field,
California.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C.
John H. Anderson, Jr.
Emilie G. Heller
Luann M. Moy
Richard R. Scott
E. Jerry Seigler
Carol Herrnstadt Shulman
Robert E. White
OFFICE OF GENERAL COUNSEL
Mindi G. Weisenbloom
ATLANTA REGIONAL OFFICE
Christopher A. Keisling
RELATED GAO PRODUCTS
Aviation Safety: New Airlines Illustrate Long-Standing Problems in
FAA's Inspection Program (GAO/RCED-97-2, Oct. 17, 1996).
Human Factors: Status of Efforts to Integrate Research on Human
Factors Into FAA's Activities (GAO/RCED-96-151, June 27, 1996).
Military Aircraft Safety: Significant Improvements Since 1975
(GAO/NSIAD-96-69BR, Feb. 1, 1996).
Aviation Safety: Data Problems Threaten FAA Strides on Safety
Analysis System (GAO/AIMD-95-27, Feb. 8, 1995).
Aviation Safety: Unresolved Issues Involving U.S.-Registered
Aircraft (GAO/RCED-93-135, June 18, 1993).
Aviation Safety: Changes Needed in FAA's Service Difficulty
Reporting Program (GAO/RCED-91-24, Mar. 21, 1991).
*** End of document. ***