Aviation Safety: System Safety Approach Needs Further Integration
into FAA's Oversight of Airlines (28-SEP-05, GAO-05-726).
The Federal Aviation Administration (FAA) uses the Air
Transportation Oversight System (ATOS), which was developed
around the principles of system safety, to oversee seven "legacy
airlines" and nine other airlines. In this report, we refer to
airlines that are not in ATOS as non-legacy airlines. Two other
processes are used to oversee 99 non-legacy passenger airlines,
which represent a fast-growing segment of the commercial aviation
passenger industry and carried about 200 million passengers in
2004. The National Work Program Guidelines (NPG) establishes a
set of inspection activities for non-legacy airlines. The
Surveillance and Evaluation Program (SEP) uses principles of
system safety to identify additional risk-based inspections for
those airlines. GAO's objective was to assess the processes used
by FAA to ensure the safety of non-legacy passenger airlines. GAO
reviewed the strengths of FAA's inspection oversight for
non-legacy passenger airlines and the issues that hinder its
effectiveness.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-726
ACCNO: A38565
TITLE: Aviation Safety: System Safety Approach Needs Further
Integration into FAA's Oversight of Airlines
DATE: 09/28/2005
SUBJECT: Airlines
Inspection
Program evaluation
Safety regulation
Safety standards
Transportation safety
FAA Air Transportation Oversight System
FAA Program Tracking and Reporting
System
FAA Surveillance and Evaluation Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-726
United States Government Accountability Office
GAO Report to Congressional Requesters
September 2005
AVIATION SAFETY
System Safety Approach Needs Further Integration into FAA's Oversight of
Airlines
a
GAO-05-726
Highlights of GAO-05-726, a report to congressional requesters
The Federal Aviation Administration (FAA) uses the Air Transportation
Oversight System (ATOS), which was developed around the principles of
system safety, to oversee seven "legacy" airlines" and nine other
airlines. In this report, we refer to airlines that are not in ATOS as
non-legacy airlines. Two other processes are used to oversee 99 non-legacy
passenger airlines, which represent a fast-growing segment of the
commercial aviation passenger industry and carried about 200 million
passengers in 2004. The National Work Program Guidelines (NPG) establishes
a set of inspection activities for non-legacy airlines. The Surveillance
and Evaluation Program (SEP) uses principles of system safety to identify
additional risk-based inspections for those airlines.
GAO's objective was to assess the processes used by FAA to ensure the
safety of non-legacy passenger airlines. GAO reviewed the strengths of
FAA's inspection oversight for non-legacy passenger airlines and the
issues that hinder its effectiveness.
GAO recommends that FAA develop an evaluative process for SEP and improve
communications and training for inspectors in system safety and risk
management. FAA agreed with the recommendations on training and
communications and will consider developing an evaluative process.
www.gao.gov/cgi-bin/getrpt?GAO-05-726.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Gerald L. Dillingham at
(202) 512-2834 or [email protected].
September 2005
AVIATION SAFETY
System Safety Approach Needs Further Integration into FAA's Oversight of
Airlines
A key strength of FAA's inspection oversight of non-legacy airlines is the
introduction of system safety concepts to some inspections, which FAA
accomplished by adding SEP to its traditional inspection process, NPG.
Although NPG has risk-based elements, it lacks the structured approach to
risk identification found in SEP. Under SEP, data are used to help
determine trends or problems. The SEP process uses a team of inspectors to
identify inspection activities, which we have previously reported is
generally more effective than the use of individuals due to their
collective ability to identify risks. Under SEP, inspectors also ascertain
risks internal to FAA, such as staffing shortages. FAA's oversight of
non-legacy airlines further incorporates processes to ensure that
inspectors follow up on airline actions taken in response to inspection
findings. These efforts address several past GAO concerns, including that
NPG did not allow FAA to identify risks and allocate inspection resources
accordingly.
The full potential of FAA's inspection program for non-legacy airlines,
however, is not being realized due to incomplete implementation of its
system safety approach and other challenges. The inspection workload is
still heavily oriented to nonrisk-based activities, with 77 percent of
inspection activities being identified through the NPG and the remaining
relatively small percentage identified through SEP. The emphasis on NPG,
including FAA's guidance that inspectors must complete NPG-required
inspection activities, acts as a disincentive to identifying further
inspection activities through SEP. Inspectors face workload challenges as
staff lost through attrition may not be replaced due to a hiring freeze.
FAA estimates that over 1,100 inspectors of non-legacy airlines will leave
the agency in fiscal years 2005 to 2010. In addition, some FAA inspectors
indicated that a lack of technical training on airline systems and
equipment posed potential risks to the agency's oversight process.
Finally, FAA lacks a process to communicate information to inspectors on
how certain internal risks identified through SEP are being resolved.
Moreover, FAA has not established a process to evaluate the effectiveness
of SEP.
Aircraft Safety Inspection
Source: FAA.
Contents
Letter 1
Results in Brief 3
Background 5
FAA's Oversight Process Uses System Safety, Which Is Designed to
Identify and Control Risks and Improve Resource Utilization 12
Most Inspection Activities Are Not Prioritized Based on a
Structured
Risk Assessment Process 18
Conclusions 34
Recommendations for Executive Action 35
Agency Comments and Our Evaluation 35
Appendixes
Appendix I:
Appendix II:
Appendix III:
Objectives, Scope, and Methodology 37
FAA's Surveillance and Evaluation Process 44
SEP Incorporates System Safety into FAA's Inspection Oversight of
Non-legacy Airlines 44
SEP Incorporates Some Elements of ATOS, While NPG Relies on a Set Number
of Inspections 46
GAO Contact and Staff Acknowledgments 48
Tables Table 1:
Table 2:
Table 3:
Table 4:
Table 5:
Table 6: Table 7: Table 8:
Table 9:
Number of Times Risks Internal to FAA Were Identified by
Inspectors through SEP, Fiscal Years 2002-2004 15
Severity of Internal Risks Identified by FAA Inspectors,
Fiscal Years 2002-2004 16
SEP-and NPG-Initiated Required Inspections for the Top
25 Non-legacy Airlines, Fiscal Years 2002-2004 19
Allocation of FAA Inspection Staff under SEP and ATOS
for Four Airlines 23
Number of Enplanements for the Top 25 Non-legacy
Airlines, 2004 38
FAA Offices and Airlines Interviewed by GAO 40
Organizations Interviewed by GAO 41
Safety Systems and Examples of Risk Indicators in
SEAT 44
Various Elements of ATOS, NPG, and SEP 46
Figures Figure 1: Percentage of Enplaned Passengers on Non-legacy Airlines
and Legacy Airlines, 1995-2004 6
Contents
Figure 2: Number of FAA Aviation Safety Inspectors for Commercial
Airlines, Fiscal Years 2002-2004, and Estimates for Fiscal Years 2005-2007
8
Figure 3: Non-legacy Airline Inspectors' Views on the Extent Technical
Training Received in the Last 2 Years Has Helped Them Do Their Jobs 28
Figure 4: Views of Inspectors of Non-legacy Airlines on the Extent
Inspectors Have Received Technical Training in a Timely Manner during
Their FAA Careers 30
Figure 5: View of Inspectors of Non-legacy Airlines on the Extent to Which
They Received Training in Automated Systems Used in the Inspection Process
in a Timely Manner 31
Abbreviations
ATOS Air Transportation Oversight System
DOT Department of Transportation
FAA Federal Aviation Administration
NPG National Work Program Guidelines
PTRS Program Tracking and Reporting Subsystem
SEAT Surveillance and Evaluation Assessment Tool
SEP Surveillance and Evaluation Program
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States Government Accountability Office Washington, D.C. 20548
September 28, 2005
The Honorable James L. Oberstar
Ranking Democratic Member
Committee on Transportation and Infrastructure
House of Representatives
The Honorable Jerry F. Costello
Ranking Democratic Member
Subcommittee on Aviation
Committee on Transportation and Infrastructure
House of Representatives
The Honorable Peter A. DeFazio
House of Representatives
The U.S. commercial aviation industry, with less than one fatal accident
per
5 million flights from 2002 through 2004, has an extraordinary safety
record. However, when passenger airlines have accidents or serious
incidents, regardless of their rarity, the consequences can be tragic. In
order to maintain a high level of aviation safety, it is critical to have
well
established, efficient, and effective systems in place to provide an early
warning of hazards that can lead to accidents. The Federal Aviation
Administration (FAA) has established a number of systems and processes
to inspect and oversee various aspects of passenger airline safety, such
as
aircraft maintenance and flight operations. About 585 of FAA's
approximately 3,200 inspectors are dedicated to overseeing the largest
commercial passenger airlines, including the seven "legacy airlines," and
nine other airlines through FAA's Air Transportation Oversight System
(ATOS), which was developed around the principles of system safety.1
System safety involves the continual evaluation of all of an airline's
operations for the purpose of identifying and mitigating risks.
Approximately 1,100 inspectors2 oversee other entities and individuals,
including 99 smaller commercial passenger airlines-which we refer to as
1The legacy airlines are Alaska, American, Continental, Delta, Northwest,
United, and US Airways. As of July 2005, the following non-legacy
passenger airlines and cargo airlines were also part of ATOS : America
West, American Eagle, Champion, ExpressJet, SkyWest, Southwest, Trans
States, FedEx, and United Parcel Service. In this report, we refer to all
passenger airlines that are not in the ATOS program as non-legacy
airlines.
2The remaining approximately 1,500 inspectors oversee general aviation.
non-legacy passenger airlines-about 5,200 aircraft repair stations, and
approximately 625,000 pilots. Non-legacy passenger airlines, a
fast-growing segment of the commercial aviation passenger industry,
carried about 200 million passengers in 2004.
FAA's inspection process for airlines not covered by ATOS has two
components. The National Work Program Guidelines (NPG) is the baseline
component of the oversight program for these airlines. In 2002, FAA added
another component, the Surveillance and Evaluation Program (SEP), to the
inspection process to incorporate principles of ATOS into its oversight of
non-legacy passenger airlines. The two components are used together to
establish the number of annual inspections for non-legacy airlines.
Inspections can encompass many different activities, such as visually
spotchecking an airplane at a gate, monitoring procedures on a scheduled
flight, or observing maintenance being performed on an aircraft. Each
year, FAA headquarters establishes baseline inspections for each airline
through NPG, while through SEP, teams of FAA inspectors analyze the
results of an airline's prior inspections at periodic meetings and, based
on their assessment of specific risks, establish other inspections that
may be needed.
In response to your request, we assessed FAA's processes for ensuring the
safety of non-legacy passenger airlines. Specifically, we addressed the
following questions: (1) What are the strengths of FAA's inspection
approach for non-legacy passenger airlines? and (2) What issues hinder the
effectiveness of FAA's inspection approach?
To address these questions, we obtained and analyzed information from a
variety of sources. We examined FAA documents about SEP, NPG, and ATOS. We
also reviewed prior reports prepared by us and others on SEP and NPG to
determine significant issues involving those programs and how they were
resolved. In addition, we surveyed a statistical sample of FAA safety
inspectors to obtain their views about the training they receive. We had
no practical way to assess information on the amount of training necessary
for inspector proficiency or the timeliness of the training provided. We
conducted semistructured interviews and analyzed relevant documents from
FAA headquarters officials, and from field managers and inspectors in 7
regional and 13 field offices,3 which were selected because
3For this report, we use "field office" to refer to FAA field, regional,
and certificate management offices, unless otherwise noted.
they oversee the top 25 non-legacy airlines4 ranked by the number of
enplanements in 2004. At these locations, we collected information on the
inspection process and inspector staffing levels, workload, and training.
In addition, we conducted semistructured interviews with safety officials
at 16 of those top 25 non-legacy airlines, and with officials at seven
industry organizations that represent airlines, inspectors, pilots,
mechanics, and maintenance facilities. We analyzed data on required NPG
and SEP inspections for fiscal years 2002 through 2004 from FAA's
nationwide inspection database-the Program Tracking and Reporting
Subsystem (PTRS).5 We assessed the reliability of the database and found
the data sufficiently reliable for the types of analyses that we conducted
for this report. We also tested for the presence of several management
controls, including the processes for verifying inspection results,
establishing a process to evaluate FAA's inspection oversight process,
managing the PTRS database, and communicating among managers and
inspectors relating to NPG and SEP. In addition, we reviewed literature on
system safety and compared FAA's system safety framework with that
presented in the literature. We conducted our work from August 2004
through September 2005 in accordance with generally accepted government
auditing standards. Additional information on our methodology is found in
appendix I.
Results in Brief A key strength of FAA's inspection oversight of
non-legacy airlines is the introduction of system safety concepts to some
inspections, which FAA accomplished by adding SEP to its traditional
inspection process, NPG. SEP presents a shift in concept from FAA's
customary method that relied on conducting a set number of inspections of
an airline's operations to an approach that allows for the efficient use
of inspection staff and resources by prioritizing workload based on areas
of highest risk. To facilitate the implementation of a system safety
approach, FAA has made an effort to train its staff in system safety. In
addition, FAA utilizes teams of inspectors in SEP, which we have
previously reported is generally more effective than
4Four of the 25 non-legacy airlines in our review are now in the ATOS
program--American Eagle, ExpressJet, SkyWest, and Trans States.
5PTRS also includes information on "planned" inspection activities and
other surveillance activities, which are not inspections, such as
conducting telephone conversations with airline officials and reading
documents related to an airline. However, the database does not
distinguish between planned inspections and other activities. As a result,
we excluded these activities from our analyses.
the use of individuals because of the team's collective ability to
identify risks. SEP also allows inspectors to identify risks internal to
FAA, such as staffing shortages or training deficiencies, and FAA has
established a chain of command to address these risks. FAA's oversight of
non-legacy airlines further incorporates processes to ensure that
inspectors follow up on airline actions taken in response to inspection
findings. These efforts address several past concerns noted by us-that FAA
needed to better identify risks and allocate inspection resources
accordingly, and verify and monitor inspection findings to ensure that
priorities were achieved.
The full potential of FAA's inspection program for non-legacy airlines,
however, is not being realized due to incomplete implementation of the
agency's system safety approach and other challenges. The inspection
workload is still heavily oriented to the NPG's nonrisk-based activities.
For fiscal years 2002 through 2004, 77 percent of inspection activities
required for the top 25 non-legacy airlines were identified through NPG,
so that only the remaining relatively small percentage of inspection
activities were identified based on risk through SEP. We found that the
large percentage of NPG-identified activities occurred because FAA's
guidance places greater emphasis on the NPG-identified activities, even
though the guidance establishes a process-referred to as
retargeting-whereby inspectors can replace NPG-identified activities with
SEP-identified activities that they deem constitute a greater safety risk.
For example, the guidance emphasizes the importance of the NPG-identified
activities by requiring inspectors to complete all of the NPG-identified
activities by the end of each fiscal year but permitting SEP-identified
activities to be rescheduled to the following fiscal year. In addition,
very few activities are being retargeted, partly because field offices
have interpreted FAA's emphasis on NPG activities as discouraging
retargeting. Inspector workload also presents a challenge to FAA's
oversight, as the number of inspector staff available to oversee
non-legacy airlines has declined due to attrition and workload shifts
related to transferring staff to ATOS, and most losses were not replaced
due to a hiring freeze. In 11 of the 13 FAA field offices that we
contacted, officials indicated a shortage of different types of inspectors
needed to oversee the non-legacy airlines, which has sometimes resulted in
inspections being delayed or eliminated. We also identified limitations
concerning technical and SEP-specific training for inspectors of
non-legacy airlines. For example, inspectors identified a lack of
technical training on airline systems and equipment as an internal risk to
the agency. In addition, FAA does not provide SEP-specific procedural
training to certain types of inspectors, which some inspectors told us was
needed to maximize their
usefulness.6 Further, there is inadequate communication from headquarters
to inspectors on the resolution of internal risks identified under SEP,
according to some inspectors. Headquarters officials acknowledged that
there is no formal feedback process to inform the inspectors about issues
they raised concerning internal risks. FAA has not established a way to
evaluate its inspection oversight process for non-legacy airlines.
Moreover, the agency's ability to evaluate this process is hindered by the
lack of important inspection-related information-such as whether the risks
identified through SEP have been mitigated-in its nationwide inspection
database.
To improve the effectiveness of the agency's oversight of non-legacy
airlines, we recommend that the Secretary of the Department of
Transportation (DOT) direct the FAA Administrator to develop a continuous
evaluative process for FAA's activities under SEP and link SEP to the
performance-related goals and measures developed by the agency, track
performance toward these goals, and determine appropriate program changes.
We also recommend that the agency improve communication and training to
ensure inspectors understand FAA's policies and procedures in areas such
as system safety and risk management. DOT generally agreed with our
recommendations to improve communication and training. DOT said that it
would consider our recommendation to develop a continuous evaluative
process for SEP and link SEP to agency goals, but that its plan to put the
remaining non-legacy airlines in the ATOS program by the end of fiscal
year 2007 may make this recommendation unnecessary. In the past, FAA's
efforts to move airlines to ATOS have experienced delays, therefore, we
retained this recommendation. The department also provided clarifying
comments and technical corrections, which we incorporated as appropriate.
Background Non-legacy airlines represent a fast-growing segment of the
passenger airline industry. From 2002 through 2004, the annual
enplanements for these airlines grew from 122 million to about 200 million
passengers, or from about 20 percent to 28 percent of all passenger air
travel. During this same period, the percentage of people flying on legacy
airlines declined from 80 percent to about 72 percent of all passengers
flown (see fig. 1).
6Geographic inspectors, who are based around the country and not assigned
to any particular airline, do not normally receive training in the SEP
process.
Figure 1: Percentage of Enplaned Passengers on Non-legacy Airlines and
Legacy Airlines, 1995-2004
Percent 100 80
60
40
20 0
FAA's safety oversight of non-legacy airlines is carried out by inspectors
located at 109 field offices throughout the world that are part of 9
regional offices. For each airline, FAA puts together a team led by
principal inspectors who maintain primary responsibility for managing the
airline's certificate requirements7 and focus on one of three disciplines:
avionics,8 maintenance, or operations. Additional team members include
those based
7Before commencing operations, an airline must obtain an operating
certificate from FAA. FAA issues the certificate after determining that an
airline's manuals, aircraft, facilities, and personnel meet federal safety
standards. FAA subsequently monitors the airline's operations, primarily
through safety inspections to ensure that it continues to meet the terms
of its certificate.
8Avionics focuses on the electronic components of an aircraft.
Non-legacy
Source: GAO analysis of U.S. Department of Transportation data.
FAA's Inspector Workforce Carries Out Oversight
at the FAA office that holds the airline's operating certificate-typically
an aircraft dispatch inspector, a cabin safety inspector, and assistants.
FAA locates its principal inspectors close to their respective airlines'
primary operational base. For example, the principal inspectors for
Independence Air (formerly Atlantic Coast Airlines) are located at FAA's
field office at Dulles International Airport, where the airline has its
headquarters. In addition, FAA has geographic inspectors based at the 109
field offices to conduct additional inspections. Rather than being
designated to particular airlines, geographic inspectors may conduct
inspections of aircraft of any non-legacy airlines that land in their
area.
FAA's safety inspector workforce for all commercial airlines has remained
steady, averaging about 1,780 inspectors over fiscal years 2002 through
2004. FAA cannot determine how many of these inspectors were assigned to
non-legacy passenger airlines during those 3 years, because the agency did
not collect that information, according to an FAA headquarters' official.
FAA expects the number of inspectors assigned to non-legacy airlines to
remain steady for fiscal years 2005 and 2006 and decline slightly in
fiscal year 2007, as shown in figure 2. Inspectors record information
about these inspections in PTRS-a nationwide computerized database that
maintains such information as inspector findings and airline activities in
response to the findings.
Figure 2: Number of FAA Aviation Safety Inspectors for Commercial
Airlines, Fiscal Years 2002-2004, and Estimates for Fiscal Years 2005-2007
Aviation safety inspectors 2,000
NPG and SEP are the main inspection processes that FAA uses to oversee the
safety of non-legacy airlines. Since 1985, FAA has used NPG, which
includes both required and planned inspections, as its primary means of
ensuring that airlines comply with safety regulations. In NPG, an FAA
committee of program managers identifies an annual minimum set of required
inspections that are to be undertaken to ensure that airlines are in
compliance with their operating certificates. In addition, inspectors
determine annual sets of planned inspections based on their knowledge and
experience with the particular airlines they oversee. Typically,
inspections would include ramp inspections, in which inspectors examine an
aircraft while it is parked at the airport, and maintenance inspections.
However, we found problems with NPG throughout the 1990s, including: (1)
FAA's routine inspections were ineffective in identifying serious safety
1,810 1,784 1,833
1,500
1,000 500 0
2002 2003 2004 2005 2006 2007 Fiscal year
ATOS inspectors
Total aviation safety inspectors
Non-legacy airline inspectors
Source: GAO analysis of U.S. Department of Transportation data.
Note: FAA does not have information to separate out the number of ATOS
inspectors and inspectors of non-legacy airlines prior to fiscal year
2005.
FAA Uses NPG and SEP to Oversee Non-legacy Airlines
problems, (2) critical airline inspections had not been conducted, (3)
FAA's follow-up actions often did not ensure that problems were corrected
once identified, and (4) FAA did not have a methodology for estimating
airline safety risks so that it could target limited inspection resources
to high-risk inspections. We also found that FAA's inspection database was
of limited use in providing early warning of potential risks or targeting
inspection resources. 9
In response to these findings, and in the aftermath of the 1996 ValuJet
crash, an FAA task force reviewed the agency's safety inspection process
and recommended in part that the agency initiate a project to make
surveillance of airlines more systematic and targeted to deal with
identified risks. This recommendation resulted in the agency's development
and implementation of ATOS in 1998 at the nation's 10 largest commercial
passenger airlines, with the goal of eventually including all commercial
passenger and cargo airlines.10 ATOS emphasizes a system safety approach
that extends beyond periodically checking airlines for compliance with
regulations to the use of technical and managerial skills to identify,
analyze, and control hazards and risks.11 The goal of ATOS is to identify
safety trends in order to spot and correct problems at their root cause
before an accident occurs. This program allows FAA inspectors to look at
an airline as a whole, to see how the many elements of its operations,
including aircraft, pilots, maintenance facilities, flight operations, and
cabin safety, interact to meet federal standards. Collectively, the
airlines under ATOS had a dedicated inspector staff of 585 inspectors as
of July 2005. The
9GAO, Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA
in Identifying and Responding to Risks, GAO/RCED-98-6 (Washington, D.C.:
Feb. 27, 1998); Aviation Safety: FAA Needs to More Aggressively Manage Its
Inspection Program, GAO/T-RCED-92-25 (Washington, D.C.: Feb. 6, 1992);
Aviation Safety: Problems Persist in FAA's Inspection Program,
GAO/RCED-92-14 (Washington, D.C.: Nov. 20, 1991); and Aviation Safety:
FAA's Safety Inspection Management System Lacks Adequate Oversight,
GAO/RCED-90-36 (Washington, D.C.: Nov. 13, 1989).
10Our review of ATOS's early implementation found problems, which FAA
addressed by improving guidance to inspectors and increasing data
usefulness. Since then, DOT's Inspector General has found additional
problems with FAA's implementation of ATOS. GAO,
Aviation Safety: FAA's New Inspection System Offers Promise, but Problems
Need to be Addressed, GAO/RCED-99-183 (Washington, D.C.: June 28, 1999);
and DOT Office of Inspector General, Safety Oversight of an Air Carrier
Industry in Transition, AV-2005-062 (Washington, D.C.: June 3, 2005).
11System safety is a process by which inspectors perform continual
evaluations of an airline's operations for the purpose of identifying
safety hazards and assessing the severity, frequency, and probability of
the hazards and monitoring their resolution.
number assigned to each airline depends on the size of the airline's
operations. ATOS uses special checklists and databases that are intended
to cover all areas of airline operations. Part of FAA's oversight is
expected to include an in-depth look at an airline's policies and
procedures and whether the airline is following them. There are 16
airlines in the ATOS program, including two cargo carriers as of September
2005. Due to resource constraints, FAA determined it would not be able to
immediately place the remaining passenger airlines in the ATOS program.
FAA developed SEP as a bridge to introduce safety risk concepts used in
ATOS into the oversight process for non-legacy airlines in order to
facilitate the ultimate transition of these airlines to ATOS. As of
September 2005, FAA estimates that it will move the remaining non-legacy
airlines to ATOS by the end of fiscal year 2007.
SEP Implemented as a Transition to ATOS
Since the introduction of ATOS, FAA has been moving toward integrating
system safety into its oversight activities. However, the agency has been
delayed in placing a significant number of non-legacy airlines in the ATOS
program, resulting in those airlines continuing to be overseen through
NPG, a process that is not system safety oriented. FAA agrees that NPG is
not a system safety process, but explained that the planned NPG
inspections, which inspectors identify based on their expertise, provide a
risk-based element to the process. To address the delay in moving airlines
to ATOS, FAA, in 2002, added SEP to NPG as a way to introduce airlines to
a system safety oversight process until those airlines were transitioned
to ATOS.12 Although originally envisioned by FAA as a transitional program
for airlines that were awaiting placement in ATOS, SEP has become a more
permanent oversight process as budget and staff constraints have prevented
the agency from moving all airlines to ATOS.
SEP is complementary to NPG and permits the risk-based, data-driven
alteration of required NPG inspections. SEP provides a formal structure to
risk identification that is absent from the NPG process. Under SEP, the
principal FAA inspectors for each airline meet periodically during the
year to discuss the results of their inspections and identify risks using
the same
12SEP was also developed to increase surveillance of new entrant airlines
that are less than 5 years old.
risk assessment principles and checklists used in ATOS.13 An outcome of
the meetings is the identification of inspections to augment the NPG
baseline of required inspections, which FAA views as the minimum number of
inspections that need to occur to ensure that certain areas of all
airlines are reviewed.14 During the SEP process, FAA headquarters allows
principal inspectors to replace NPG-required inspections with inspections
targeting higher-risk areas identified through the SEP meetings. This
process is known as retargeting. At the end of the meeting, entries are
created in the PTRS database that indicate all inspections to be performed
for that airline. SEP-initiated inspections that are designated as
priorities are required to be completed.15 See appendix II for additional
information on SEP, ATOS, and NPG.
FAA Also Uses Industry Partnership Programs to Provide Safety Oversight of
Airlines
FAA also oversees the safety of both legacy and non-legacy passenger
airlines through participation in industry partnership programs. Two of
these programs-the Aviation Safety Action Program and Voluntary Disclosure
Reporting Program-encourage certain airline employees, such as pilots and
mechanics, or airlines to voluntarily report safety information that might
be critical to identifying potential precursors to accidents without fear
that FAA or their companies will use reports accepted under the programs
to take legal enforcement or disciplinary actions against them. The
Aviation Safety Action Program provides for the voluntary selfreporting of
safety incidents under procedures set out in memorandums of understanding
between FAA, airlines, and participating employee groups such as aircraft
mechanics. As of June 2005, 20 of the top 25 non-legacy airlines (in terms
of the number of enplanements) were participating in the Aviation Safety
Action Program. The Voluntary Disclosure Reporting
13The checklists are required under ATOS and are suggested but not
mandatory for use under SEP.
14Required NPG inspections consist of activities to help validate that
critical air carrier subsystems do not have potential latent failures.
According to FAA, periodic validation of these critical subsystems is
important due to the inherent risks associated with their failures.
15Both NPG and SEP consist of required and planned activities. According
to FAA, the required NPG activities account for 12 percent of the total
inspections performed in a given year and the planned NPG activities
account for about 45 percent. The remaining 43 percent of NPG activities
are created by unplanned requirements, according to the agency. SEP
inspections are also designated as required or planned. Any planned SEP
inspections that are not completed by the end of the fiscal year become
required inspections for the following year.
Program allows airlines to voluntarily report safety incidents to FAA.
However, our prior work found that FAA has not analyzed violation data
derived from these two programs to monitor national trends in airline
operations, so that it can target resources to address operational
risks.16 A third program, the Internal Evaluation Program, requires
airlines to continuously monitor and evaluate their practices and
procedures.
FAA's Oversight Process Uses System Safety, Which Is Designed to Identify
and Control Risks and Improve Resource Utilization
The incorporation of system safety into FAA's oversight presents a shift
in concept from the agency's traditional oversight method, which relied
upon periodic inspections, to an approach that allows for the
prioritization of inspections based on areas of highest risks. The
development of SEP, a key element in FAA's system safety oversight
process, allows for the efficient use of inspection staff and resources by
incorporating this risk-based approach. FAA's incorporation of SEP into
its oversight addresses a past concern by us that the agency was not
prioritizing inspectors' workload. Our review of literature by government
and private organizations supported FAA's concept of system safety because
it makes apparent the risks that are the basis for changes. In addition,
FAA's oversight process includes a requirement that inspectors verify that
corrective actions have occurred, providing a level of assurance that the
safety problem has been mitigated.
FAA Has Provided System Safety Training for Inspectors
To facilitate the implementation of a system safety approach, FAA has made
an effort to train its staff in system safety, which is a strength of its
oversight approach. FAA has recognized that inspectors need training in
the system safety concept in order to effectively incorporate this
approach into airline oversight. Specifically, FAA believed that its
inspectors have gaps in their needed competencies and skills that could
affect the agency's system safety approach to inspections. Among the
largest training gaps for inspectors, according to FAA, were training in
risk analyses and systems thinking. As a result of these findings, FAA has
undertaken training activities to improve inspector competencies in system
safety.17 According
16GAO, Aviation Safety: Better Management Controls Are Needed to Improve
FAA's Safety Enforcement and Compliance Efforts, GAO-04-646 (Washington,
D.C.: July 6, 2004).
17GAO, Aviation Safety: FAA Management Practices for Technical Training
Generally Effective; Further Actions Could Enhance Results, GAO-05-728
(Washington, D.C.: Sept. 7, 2005).
to FAA, between fiscal years 2002 and 2004, it provided basic system
safety training to almost 3,675 staff, including non-legacy and ATOS
inspectors.
SEP Incorporates System Safety by Focusing on the Identification and
Prioritization of Inspections Based on Risks
By incorporating risk assessment into the inspection process, SEP
addresses a long-standing problem. Since the mid-1980s, our reports have
shown that FAA did not have a methodology for assessing airline safety
risks so that it could target limited inspection resources to high-risk
conditions.18 SEP's system safety concept of reducing risk through the
identification, analysis, and control of hazards is also consistent with
the views presented in literature of other government and private
organizations, such as the Department of Defense, Canada's civil aviation
authority (Transport Canada), and the Flight Safety Foundation. For
example, DOD looks at system safety as a means of reducing risk through
early identification, analysis, elimination, and control of hazards. For
the airlines it oversees, Transport Canada requests that each airline
incorporate system safety into its operations by integrating safety into
its policies, management and employee practices, and operating procedures.
FAA incorporates such concepts in its system safety oversight approach.
SEP's system safety process has a number of strengths. At periodic
meetings, the principal inspectors of an airline identify potential risks
that they believe should be addressed. These inspectors consider a variety
of information that they have obtained through personal observation and
from reports prepared by geographic inspectors and located in the PTRS
database. A strength of this approach, consistent with findings in our
past reports,19 is that teams of inspectors are generally more effective
than individual inspectors in their ability to collectively identify
concerns. Another strength is that FAA has developed risk assessment
worksheets aligned with key airline systems that guide inspectors through
identifying and prioritizing risks. The worksheets guide inspectors to
organize the results of their previous inspections and surveillance into a
number of areas, such as flight operations, personnel training, and cabin
safety, in order to identify specific risks in each area and target the
office's resources to mitigating those risks. (See app. II for more
details on the risk assessment process.)
18GAO/RCED-98-6; GAO/T-RCED-92-25.
19GAO/RCED-99-183 and GAO, Aviation Safety: FAA Oversight of Repair
Stations Needs Improvement, GAO/RCED-98-21 (Washington, D.C.: Oct. 24,
1997).
SEP Also Allows for the Identification of Internal Risks That Might Hinder
Oversight
During their periodic meetings, principal inspectors also have the
opportunity to identify risks in FAA's internal operations that could
adversely affect the inspectors' ability to conduct safety oversight of
nonlegacy airlines, which is another strength of FAA's oversight approach.
FAA inspectors and managers agreed that identifying internal risks is
helpful, as they considered it an efficient and effective way of
identifying needed program improvements. Inspectors can identify
deficiencies in a number of categories, including inspector staffing,
training, availability of geographic inspectors, and resources such as
travel funds. In addition, the inspectors can quantify the degree of the
risk that each of these categories poses. Being able to identify and make
officials aware of situations that can hinder an agency's ability to
perform its mission is a key management control.
This process has resulted in FAA headquarters receiving significant
information on areas that inspectors believe are deficient. For fiscal
years 2002 through 2004, inspectors identified about 560 risks that, in
their opinion, could have an impact on how FAA managed the inspection
process of airlines, as shown in table 1. Inadequate training
(aircraftspecific training and other training) was the most frequently
identified concern of the inspectors, accounting for about 40 percent of
the total risks identified during the 3-year period-27 percent of the
risks were related to inadequate or nonexistent aircraft-specific training
and 15 percent were related to the lack of other training. We discuss this
issue in detail later in this report. The total number of risks identified
by FAA inspectors declined over the 3 years. Inspectors from two FAA field
offices opined that the decline may be due to better staff understanding
of risk identification and to the risks being addressed expeditiously so
that inspectors did not need to restate them the following year.
Table 1: Number of Times Risks Internal to FAA Were Identified by Inspectors
through SEP, Fiscal Years 2002-2004 Number of risks identified
Risk 2002 2003 2004 Total
Inadequate staffing on the certificate
management
teama 67 53 55 175 (31%)
Inadequate or nonexistent aircraft-specific 58 57 33 148 (27%)
training
Lack of other training 35 21 29 85 (15%)
Inadequate geographic staffing 30 9 5 44 (8%)
Inadequate resources (such as travel funds) for
the
certificate management team 24 19 10 53 (9%)
Other 28 11 13 52 (9%)
Total 242 170 145 557 (99%)b
Source: GAO analysis of FAA data.
aThe certificate management team may include the principal avionics,
maintenance, and operations inspectors; aircraft dispatch inspector; and
cabin safety inspector.
bPercentages do not add to 100 due to rounding.
The process not only allows inspectors to identify the risk, but also to
describe the severity of the risk in order to assist FAA headquarters in
prioritizing its consideration of internal risks. Inspectors rate each
risk they identify in terms of severity (negligible to catastrophic) and
likelihood of occurrence (improbable to frequent). Risks are then
categorized using a scale of high, medium, or low depending on the
severity and likelihood of occurrences. For example, risks rated high are
those that have high levels of likelihood and severity. As shown in table
2, over a 3-year period, 125 out of 557 risks were rated as high severity,
which could help FAA determine which issues most need targeting. Among the
risks rated as high, training and inadequate staffing were identified most
frequently, accounting for 34 percent and 31 percent, respectively. As of
May 2005, about 87 percent of identified risks had been closed, including
about 90 percent of the risks identified as high. Risks are closed either
because action has been taken to mitigate the risk or FAA has determined
that the level of risk is acceptable and no action is warranted.
Table 2: Severity of Internal Risks Identified by FAA Inspectors, Fiscal
Years 20022004
Number of risks
Severity of risk 2002 2003 2004 Total
High 37 58 30 125 (22%)
Medium 183 88 107 378 (68%)
Low 17 22 8 47 (8%)
Unknown 5 2 0 7 (1%)
Total 242 170 145 557 (99%)a
Source: GAO analysis of FAA data.
aPercentages do not add to 100 due to rounding.
An additional strength of the internal risk identification process is that
FAA has established a chain of command for responding to these internal
risks at the appropriate level. For example, local training issues are
likely to be handled by field offices, geographic support issues would
normally be handled by regional offices, and issues that cannot be
resolved by these field and regional offices would be referred to
headquarters.
FAA's Oversight Process Includes Monitoring to Follow Up on Airline
Actions Taken in Response to Findings
Another strength of FAA's oversight of non-legacy airlines is that
inspectors monitor the actions that those airlines have taken in response
to inspection findings through subsequent inspections and participation in
safety partnership programs with the airlines. However, the inspectors'
monitoring process could be improved by better data, as we discuss later
in this report. SEP requires inspectors to monitor the actions taken by
nonlegacy airlines that will address the risks identified during an
inspection and to verify that the actions taken have resulted or will
result in resolution of the problems. This requirement addresses a concern
of ours that the NPG inspection program did not often ensure that the
agency follow-up actions corrected problems once they were identified.20
In addition, FAA's requirement for monitoring and verifying corrective
action conforms to a management control standard on the need for federal
agencies to have monitoring and verification policies or procedures for
ensuring that the findings of audits and other reviews are promptly
resolved. FAA inspectors monitor and confirm that corrective action has
been taken by either
20GAO/RCED-92-14.
conducting a special inspection of the area, following up during a
subsequent inspection, or reviewing airline documentation that addresses
the deficiency. According to inspectors and field office managers, the
verification method used is based in many instances on the severity of the
deficiency, with special inspections conducted for areas that could affect
air safety.
According to some FAA inspectors and airline officials, the regular
meetings held to discuss concerns about airlines have opened up
communications both within FAA and between FAA and the airlines. Lack of
communication among the airlines and FAA was, according to the director of
FAA's Flight Standards Service, historically a significant cause of
airline safety problems. FAA inspectors and managers noted that they
maintain an open line of communication with airline management by meeting
regularly with key airline officials (such as directors of operations and
directors of maintenance) to discuss the results of the SEP risk analyses
and the airline's proposed resolutions.
FAA's Oversight Process Is Hindered by Certain Program Weaknesses and
Human Capital Management Challenges
FAA's oversight of non-legacy airlines is hindered by the incomplete
implementation of its system safety approach and other challenges. FAA's
incorporation of system safety into its oversight process is incomplete,
as the agency continues to emphasize a nonsafety system-based process,
NPG, to identify most of its inspection activities. Human capital
management challenges also impede FAA's oversight. These challenges
include a hiring freeze that began in January 2005; attrition; in certain
cases, the reallocation of inspectors' duties as airlines are moved to the
ATOS program; insufficient training for inspectors on risk management, a
key element of SEP; and insufficient communication between FAA
headquarters and field offices. In addition, FAA lacks a process to
continuously evaluate SEP. Finally, the agency's PTRS database lacks some
important inspection-related information, such as whether risks have been
mitigated, that would aid in targeting further oversight activities as
well as a nationwide analysis of inspection results.
Most Inspection Activities Are Not Prioritized Based on a Structured Risk
Assessment Process
We found that NPG remains the primary basis for FAA's inspection of
nonlegacy airlines. For fiscal years 2002 through 2004, about 77 percent
of the required inspection activities for the top 25 non-legacy airlines
were initiated through the NPG annual planning session, compared with 23
percent that were SEP-initiated. As a result, most of the required
inspection activities are not prioritized based on risk. (See table 3.) In
addition, the total number of required inspections for non-legacy airlines
declined during the 3-year period, while the number of passengers on those
airlines grew from 122 million to 200 million annual enplanements. The
decline in inspections during this period may be due, in part, to the
movement of three airlines from SEP to ATOS. FAA headquarters officials
did not know why this decline occurred, but told us that it may be due to
agency efforts to eliminate unnecessary activities and shift from a
"quality control" oversight approach, in which individual products or
outputs are inspected to determine if they meet specifications, to a
"quality assurance" approach, in which airlines' processes are inspected.
While NPG also includes planned inspections, which have a risk-based
element, these inspections lack the structured approach to risk
identification that SEP has.21 FAA officials acknowledged that the NPG
planned inspections would benefit from the use of risk management tools.
21In addition to required NPG and SEP inspections, FAA inspectors conduct
planned inspections and unplanned activities, such as enforcement
investigations. While we recognize that unplanned activities account for a
portion of an inspector's workload, we did not include them in our
analysis because they are generated on an "as discovered" basis and,
therefore, lack comparability with other inspections that are generated
through the NPG or SEP processes. For our analysis, we looked only at
required inspections because they have higher priority than the planned
inspections and because PTRS does not distinguish between planned
inspections and other planned activities. According to FAA, the required
NPG inspections account for about 12 percent of the total inspections
performed in a given year, planned NPG activities account for about 45
percent, and unplanned activities account for the remaining 43 percent of
NPG activities. According to data provided by FAA, for the planned
inspections in fiscal years 2002-2004, 6 percent were SEP-initiated and 94
percent were NPG-initiated, indicating that an even larger percentage of
these inspections compared with required inspections are not prioritized
based on a structured risk identification process.
Table 3: SEP-and NPG-Initiated Required Inspections for the Top 25
Non-legacy Airlines, Fiscal Years 2002-2004
2002 2003 2004 Total
SEP-initiated 1,261 1,567 927 3,755 (23%)
NPG-initiated 5,470 3,623 3,338 12,431 (77%)
Total 6,731 5,190 4,265 16,186 (100%)
Source: GAO analysis of information from FAA's PTRS database.
According to FAA officials, there are no minimum numbers or percentages of
required SEP inspections. Therefore, a large percentage of NPG-initiated
activities is not unexpected or negative in their view. FAA officials said
NPG-initiated activities are an important part of the inspection process
because they provide information to the agency on an annual basis for
certain types of inspections and enable periodic assessments of key
programs in an airline's operations. According to FAA, NPG is organized to
systematically validate on an annual basis the performance of all of an
airline's safety-critical programs. FAA considers this type of recurring
assessment to be an important system safety principle and believes that it
provides an opportunity to discover and correct latent failures before
they cause safety problems. In addition, according to FAA officials, NPG
identifies a minimum number of inspections that they believe need to occur
to ensure that certain areas of an airline are reviewed, unlike SEP, which
does not provide the built-in assurance that these areas will be inspected
eventually.
We agree with FAA officials that the agency should not establish minimum
numbers or expected percentages of SEP inspections. However, we found that
the large percentage of required NPG activities compared with SEP
activities occurs, in part, because FAA's guidance places a greater
emphasis on completing required NPG activities. This situation, along with
other factors we discuss below, may deter inspectors from identifying
additional inspections through SEP. FAA's guidance includes the following:
o Field offices must complete 100 percent of the required NPG-identified
activities by the end of each fiscal year22 (unless they are
retargeted).23
o Field offices are not required to complete SEP-initiated activities by
the end of the fiscal year and may reschedule them to the following fiscal
year.
In addition, the budget given to field offices for inspections is based on
required NPG-initiated activities, and funding to cover SEP-initiated
activities must come out of that budget. Inspectors have indicated concern
regarding resources for inspections; for example, a lack of resources,
including a lack of travel funds to conduct inspection activities, made up
about 10 percent of the internal risks identified by inspectors through
SEP during fiscal years 2002 through 2004. According to headquarters
officials, requests for funding high-risk SEP-identified inspections are
always approved; however, some noncritical activities may be delayed due
to budget constraints. For example, they said that some certification
work- such as certifying that new aircraft or aircraft parts meet FAA
standards- is being delayed until an inspector travels to the airline to
perform another activity and can do the certification at little or no
extra cost.
Under FAA's guidance, if inspectors identify a risk under SEP that they
wish to mitigate through an additional inspection activity, they must
either add the SEP-initiated activity to the list of required
NPG-initiated activities or retarget a required NPG activity and replace
it with the SEP-initiated activity. Adding SEP-initiated activities to the
required NPG-identified activities may increase the overall workload of
inspectors. This situation may affect the number of additional activities
that inspectors identify during the SEP meetings. In fact, we found that
in some cases, no SEPinitiated activities were included with NPG-initiated
activities as part of an airline's overall required inspections. For
example, for one airline overseen by FAA's Central Region, PTRS data show
that, in fiscal year 2002, inspectors conducted about 400 required
NPG-initiated activities and no SEP-initiated activities. According to FAA
headquarters officials, many inspectors view SEP-initiated activities as
additional work. This view was also held by some inspectors we spoke with.
22FAA has no similar requirement for planned NPG activities, the
completion of which depends upon available resources.
23Inspectors can substitute or retarget required NPG-initiated activities
with activities identified during the periodic SEP meetings that they deem
constitute a greater safety risk.
Moreover, our analysis of PTRS data indicates that retargeting occurs
infrequently. In fiscal year 2002, about 5 percent of all NPG-required
inspections were retargeted by inspectors and 3 percent were retargeted in
fiscal years 2003 and 2004. This low rate occurs, in part, because field
offices have interpreted FAA's emphasis on NPG inspections as discouraging
retargeting. For example, three of the principal inspection teams for the
16 airlines told us they do not retarget any NPG activities. Further, in 5
of the 13 field offices we visited, we were told by both managers and
inspectors that they believe NPG activities take priority over all but the
most risky situations identified through SEP. Similarly, an official with
the Professional Airways Systems Specialists, a union that represents many
FAA safety inspectors, told us that its members believe that FAA's
management does not give as much attention and priority to SEP as it does
to ATOS.
FAA's emphasis on required NPG-initiated inspections is also shown through
its data-tracking efforts. According to FAA officials, for fiscal years
2002 through 2004, all required NPG activities had been completed by the
end of the year. FAA did not know how many completed SEP-initiated
activities were mitigated or required further inspection activities that
were then carried over to the next fiscal year because it does not collect
such information nationwide in PTRS. Moreover, field office managers also
rely on PTRS, which lacks information on the number of activities that are
carried over. For example, one field office supervisor told us that one
inspector had carried over 19 activities from fiscal year 2004 to fiscal
year 2005. However, the field office supervisor said it was difficult to
determine how many additional activities were carried over because PTRS
does not track this information.
Shifts in Inspectors' Workload Present a Challenge to FAA's Inspection
Oversight Process
The number of FAA inspectors available to oversee non-legacy airlines will
be affected by recent and anticipated trends in attrition and a hiring
freeze. First, for fiscal year 2004, the number of inspectors (those
assigned to nonlegacy airlines, ATOS, and general aviation) who left FAA
was greater than the number of inspectors hired, resulting in a net loss
of 67 inspectors. Second, the number of inspectors of commercial airlines
(both ATOS and non-legacy) is expected to continue to decline from 1,810
at the end of fiscal year 2004 to 1,704 at the end of fiscal year 2005.
Third, from fiscal years 2006 to 2007, the agency anticipates the number
of inspectors of nonlegacy airlines will decline from 1,119 to 1,088.
Fourth, for fiscal years 2005 through 2010, FAA estimated that over 1,100
inspectors of non-legacy
airlines and general aviation will leave the agency, with an average loss
due to attrition of about 195 inspectors per year.
FAA will have to hire inspectors to offset the anticipated losses.
However, the agency put a hiring freeze in place in January 2005 for
budgetary reasons, which prevents the agency from hiring inspectors to
replace those lost due to attrition unless they are deemed critical.
Furthermore, in order to absorb a budget reduction in fiscal year 2005,24
FAA expects to eliminate about 200 inspector positions in the Office of
Aviation Safety. These reductions will be accomplished through attrition.
In fiscal year 2006, the office expects the staffing level to increase by
about 80, which would still be below the fiscal year 2004 staffing
level.25 FAA is filling safety-critical positions that become vacant
through internal appointments. According to FAA officials, principal
inspectors and managers are considered safety critical. As other safety
inspectors, including geographic inspectors, leave the agency or are
appointed to safety-critical positions, they are not being replaced at
this time, according to those officials. Moreover, their workload is being
divided among the remaining inspectors, thereby increasing the inspectors'
workload, which has resulted in other, less critical, work being delayed
or deferred. In 11 of the 13 FAA field offices we contacted, officials
indicated a shortage of different types of inspectors needed to oversee
the non-legacy airlines. For example, the Washington, D.C., field office
did not have a principal avionics inspector. Officials with FAA field and
regional offices said they have developed ways to deal with the inadequate
staffing of inspectors in their offices. For example, the Southern
Regional Office told us that it uses geographic inspectors more frequently
than in the past to help conduct inspections rather than having the
inspectors assigned to the non-legacy airline travel to conduct them.
However, two field offices--Chicago and Minneapolis--indicated the need
for more geographic inspectors. Another way that FAA offices are dealing
with inspector shortages is by having an inspector perform inspections for
several offices. For example, the Atlanta field office has been without a
cabin safety inspector for a year and uses the inspector assigned to the
Southern Regional Office. According to an FAA inspector, some inspections
are delayed due to lack of staff availability. In addition, according to
our
24The Office of Aviation Safety was expected to absorb about a $17.4
million reduction in its fiscal year 2005 budget. The reductions were
required by Congress and FAA.
25In September 2005, FAA headquarters officials told us they expected
Congress to appropriate an additional $8 million in fiscal year 2006,
which would allow them to hire more inspectors than the previously planned
80.
analysis of FAA data (discussed earlier in this report), for fiscal years
2002 through 2004, principal inspectors identified lack of staff for the
airlines' certificate management teams as among the top two risks for each
year, and overall, more than one-third of the risks identified were
related to lack of inspectors.
The move of additional airlines into ATOS, which requires more inspectors
per airline than SEP, also affects the workload of inspectors for
non-legacy airlines. In addition, unlike SEP inspectors, ATOS inspectors
are dedicated to an airline and generally cannot be used to conduct
inspections of other entities. When four airlines were recently
transitioned into ATOS, the total size of the four inspection teams
increased 30 percent, from 73 to 95 inspectors, as shown in table 4. As a
result, the number of inspectors available to oversee non-legacy airlines
was reduced not only by those 73 who had covered the four airlines while
they were under the NPG and SEP processes (who would have also had
responsibilities for other entities) but by an additional 22 inspectors as
well.
Table 4: Allocation of FAA Inspection Staff under SEP and ATOS for Four
Airlines
Number of FAA Number of FAA Change in
inspectors under inspectors under number of
Airline SEP ATOS inspectors
Champion 4 6 +2
American Eagle 31 35 +4
ExpressJet 18 24 +6
SkyWest 20 30 +10
Total 73 95 +22
Source: FAA.
FAA is aware that transitioning additional inspectors to airlines in ATOS
is putting a burden on the workload of those inspectors who continue to
inspect non-legacy airlines, and the agency has, in one case, revised its
procedures for dedicating staff to individual ATOS airlines. When Champion
Airlines became part of ATOS in January 2005, FAA changed ATOS policy to
allow the Northwest Airlines inspection team to share its data analyst and
manager with the Champion inspection team. FAA officials believe that
sharing staff resources will address the issue of staff shortages.
All of the inspectors at the offices we contacted had other duties in
addition to overseeing non-legacy airlines. The inspectors performed
activities such as certifying and approving new aircraft types; overseeing
repair stations and aviation schools; reviewing FAA directives and
aircraft updates; responding to complaints; investigating accidents,
incidents, and complaints; overseeing designees;26 and attending training.
Although primarily responsible for a particular airline, some principal
inspectors conducted inspections of other airlines and entities. For
example, a principal inspector in the Great Lakes Region was responsible
for overseeing three repair stations in addition to a non-legacy airline.
In addition, a principal operations inspector in the Southern Region was
also responsible for overseeing designees such as pilot examiners. Our
analysis of the inspectors' workload contained in the PTRS database showed
that for fiscal years 2002 through 2004, about 75 percent of FAA
inspectors had responsibility for more than 3 entities, and about half had
responsibility for more than 15. While the size and complexity of the
various entities will dictate the inspector workload, having
responsibility for a large number of entities can increase an inspector's
workload because of the need to be familiar with the operating procedures
of each entity as well as to spend time physically inspecting each entity.
Inspectors Are Concerned That Training Limitations Present a Challenge to
Inspection Oversight Process
While FAA provided system safety training to its inspector workforce, as
mentioned earlier in this report, the lack of sufficient training in other
areas was identified by principal inspectors during their risk analysis
meetings and by inspectors we spoke with.27 Our analysis of FAA's database
of internal risks identified by principal inspectors of non-legacy
airlines indicated that 42 percent (233 out of 558) of the internal risks
identified for fiscal years 2002 through 2004 related to training.
Although FAA policy requires that principal inspectors assigned to
non-legacy airlines complete technical training on the airline's specific
policies, procedures, and equipment before they can conduct airline
inspection activities, principal inspectors identified risks associated
with the lack of such training. For example, our analysis of the internal
risk database found that principal
26Designees are private persons and organizations that handle the vast
majority of FAA's safety certification activities, such as testing pilots
and mechanics for FAA-issued certificates.
27We have reported that FAA has followed or is taking many effective
management practices in planning, developing, and delivering technical
training, yet inspectors express widespread dissatisfaction with this
training. See GAO-05-728.
inspectors identified a lack of technical training for themselves or other
inspectors on their team on specific aircraft, engine types, navigation
equipment, and avionics that pertained to the airlines they were
overseeing.28 Others cited a lack of training in cargo loading, hazardous
material identification, or weight and balance. Still others indicated the
need for training in FAA's systems or processes. For example, several
indicated that they had not received the basic courses for principal
inspectors (such as "foundation for principal inspectors"). As another
example, several principal inspectors reported that certain inspectors on
their team lacked training in using the computer system that analyzes
information collected in PTRS, noting that without this training they were
unable to directly access the system. Even though inspectors expressed
these concerns, approximately 87 percent of identified internal risks have
been closed by FAA, according to our analysis of FAA data. However, we
could not determine how many of those risks were actually mitigated, as
FAA closes risks either when action has been taken to mitigate the risk or
because the agency has determined that the level of risk is acceptable and
no action is warranted.
During our field office visits, inspectors also spoke about the lack of
technical training in certain areas. For example, inspectors told us they
had not received training on some newer aircraft they were responsible for
inspecting and that they were not always able to attend free training
provided by the manufacturers, which they felt would help them obtain
needed technical knowledge. We found instances where management
acknowledged that such training was needed. In the past, concerns had been
raised that FAA's acceptance of free training from a regulated entity
might have the appearance of a conflict of interest. In September 2005, we
reported that FAA accepts such training under limited circumstances.29 FAA
has established safeguards to help preclude the appearance of a
28We did not verify these reports that inspectors made in the internal
risk database. However, FAA procedures call for all risks to be reviewed
by managers, who can also enter comments indicating concurrence or
disagreement with the risk, among other things. The examples we cite in
this report do not contain information indicating that management
questioned the accuracy of the inspectors' statements.
29GAO-05-728.
conflict of interest when FAA accepts training from the entities it
regulates at no cost to the agency or in exchange for an in-kind
service.30
In addition, the principal inspectors we spoke with indicated that some
geographic inspectors who assist them in providing oversight of non-legacy
airlines lack the technical and SEP-specific procedural training to
maximize their usefulness. Several field office managers and principal
inspectors told us that geographic inspectors needed to be knowledgeable
about the operations of airlines they inspect in order to provide
effective assistance. Several principal inspectors indicated that
misperceptions about an airline's operating requirements are not uncommon
because geographic inspectors often lack training on the aircraft they
inspect.31 As a result, the inspectors indicated that geographic
inspectors had at times incorrectly coded inspection information in PTRS
and erroneously identified risks, which resulted in principal inspectors
having to spend time determining that a problem did not actually exist. In
addition, several airline officials told us that geographic inspectors had
incorrectly identified problems and appeared to not understand airlines'
operational procedures. For example, we were told that a geographic
inspector improperly cited an airline for not complying with a deicing
program that the airline had previously informed FAA inspectors it was
changing. FAA headquarters officials told us that it is not feasible to
train all geographic inspectors on all airlines they may encounter.
Further, they noted it would be good for geographic inspectors to contact
principal inspectors when they are unsure of whether an action or
condition they observe is a problem. However, FAA headquarters has not
provided that instruction to the geographic inspectors. Industry
representatives also told us this was a problem that could be alleviated
by inspectors having access to airlines' operating manuals online.
Another issue identified as a problem by principal inspectors we spoke
with is that not all geographic inspectors are provided training in SEP,
including how to use the uniform coding scheme that was developed for SEP
to facilitate the identification and analysis of risks. Geographic
inspectors may therefore note the results of their inspections through
30The safeguards include (1) executing an agreement with the aviation
industry training provider outlining the conditions under which the
training will be accepted, (2) conducting a legal review of the agreement
to ensure that there are no conflict of interest issues, and (3) obtaining
the approval of the FAA Administrator for the acceptance of the training.
31We were not provided sufficient information to verify these statements.
narrative in the comments section of PTRS rather than through SEP codes,
so that the principal inspectors must read through all the comments in
order to identify geographic inspectors' concerns. As PTRS may contain
thousands of entries on one airline each year, reading every narrative
comment can be time consuming and difficult to interpret. The lack of
coding of geographic inspectors' findings could therefore result in
principal inspectors missing problems or risks identified by geographic
inspectors. We do not know how often this problem occurs, as FAA has not
assessed the reliability of this data in PTRS. However, several FAA staff
brought this issue to our attention. One field office manager and SEP
coordinator told us that PTRS entries by geographic inspectors are not
helpful because they do not provide meaningful comments for FAA's risk
assessments, and several principal inspectors told us that training for
geographic inspectors on SEP coding would help them more easily identify
risks during their periodic meetings.
These training issues may arise in part because FAA has not systematically
identified technical training needs for principal or other inspectors. Our
previous report found that FAA has not systematically identified overall
training needs of its inspectors to ensure that the curriculum addresses
the unique training needs of each type of inspector.32 Instead, course
development focuses on individual courses. FAA said it recognizes that it
manages courses as individual components of an overall curriculum that is
only loosely defined and that it needs to develop courses and address
training needs as part of an overall curriculum. It has established a
curriculum transformation plan that it estimates it will fully implement
in 2008.
In responding to our survey, inspectors had mixed views on the usefulness
of technical training.33 For about three-quarters34 of the responses,
roughly
32GAO-05-728.
33We defined "technical training" as training that develops knowledge of
the production, maintenance and operation of aircraft (including currency
training), aircraft parts, and systems. For the purposes of our survey,
technical training did not include automation training. Our definition
differs somewhat from FAA's use of the term. FAA defines "technical
training" to include aviation technologies as well as topics such as
inspector job skills, risk analysis, data analysis, and training in
software packages, such as spreadsheets.
34The actual estimate is 77 percent. All percentage estimates from the
survey have a margin of error of plus or minus 7.1 percentage points or
less. Survey estimates presented are statistically significant when the 95
percent confidence intervals do not overlap.
equal percentages of inspectors of non-legacy airlines responded that the
technical training they received in the last 2 years35 helped them do
their current jobs to some extent, to a moderate extent, or to a great
extent. On the other hand, 6 percent indicated that the technical training
had not helped them do their jobs and another 26 percent indicated it had
only helped to some extent. (See fig. 3.)
Figure 3: Non-legacy Airline Inspectors' Views on the Extent Technical
Training Received in the Last 2 Years Has Helped Them Do Their Jobs
Percent
27 26
Very great Great Moderate Some No extent
Source: GAO survey of FAA inspectors.
Note: Approximately 10 percent of inspectors responded that they had no
basis to judge or did not know.
Further, the timeliness of certain training was identified as a concern by
both principal inspectors we spoke with and inspectors responding to our
survey. Specifically, some principal inspectors we spoke with indicated a
lack of timeliness for SEP training. Since FAA's introduction of the SEP
35FAA officials told us that training occurs over the span of a career and
cautioned that asking inspectors' views about 2 years' experience would
present a distorted view. We recognize FAA's point and the fact that FAA
requires candidates for safety inspector positions to have extensive
technical qualifications and experience. However, it is not reasonable to
expect inspectors' to recall their views on training received over a large
time span, as doing so could lead to unreliable results.
training for non-legacy inspectors in 2001, a total of 700 inspectors have
received the training--just over 300 inspectors between 2001 and 2002 and
almost 400 inspectors between 2003 and 2004. This means that less than 40
percent of the approximately 1,800 inspectors of both ATOS36 and nonlegacy
airlines (as of fiscal year 2004) could have received SEP training. FAA
requires all principal inspectors and other inspectors who are part of
certificate management teams to receive SEP training. Geographic
inspectors are not part of those teams and may not receive that training.
While all of the principal inspectors who were on staff when SEP was
implemented in 2002 received initial SEP training, according to FAA, since
that time, newly assigned inspectors may not have received the training
before beginning their responsibilities. Four principal inspectors who
were assigned relatively recently to non-legacy airlines and were not
provided the initial SEP training told us they had received subsequent
training given to all inspectors to explain revisions to the SEP program.
They said that the lack of initial SEP training hindered their ability to
understand SEP terms and processes and participate fully in the periodic
meetings to analyze information from inspections to identify risks. Those
inspectors also believed that the subsequent SEP training was not as
helpful without the initial SEP training.
The views of the inspectors we spoke with are consistent with our survey
results, which indicated that many inspectors of non-legacy airlines are
not highly satisfied with the timeliness of technical training they say
they need to do their jobs. Our survey found that only about 16 percent of
non-legacy inspectors believe to a great or very great extent that they
have received technical training in a timely manner to do their jobs.
Moreover, about the same percentage (15 percent) of the inspectors
reported that the timing of their training had not been useful to their
jobs at all. (See fig. 4.)
36FAA does not have information to separate out the number of ATOS
inspectors and inspectors of non-legacy airlines prior to fiscal year
2005.
Figure 4: Views of Inspectors of Non-legacy Airlines on the Extent
Inspectors Have Received Technical Training in a Timely Manner during
Their FAA Careers
Percent
38
Very great Great Moderate Some No extent
Source: GAO survey of FAA inspectors.
Note: Approximately 1 percent of inspectors responded that they did not
know.
About one-third of the inspectors37 in our survey indicated that the
training they received in the automated systems used in the inspection
process- including PTRS-was provided in a timely manner to a moderate
extent. Another 30 percent thought the training was only somewhat timely.
(See fig. 5.) Timely training on automated systems is important in order
to have accurate data. Inaccurate PTRS data hinder the identification of
risks by principal inspectors, as we discussed previously in this report.
37The actual estimate is 33 percent.
Figure 5: View of Inspectors of Non-legacy Airlines on the Extent to Which
They Received Training in Automated Systems Used in the Inspection Process
in a Timely Manner
Percent 40
33
Some inspectors told us that when they identified internal risks to FAA,
they were able to see the results of actions taken by their field and
regional offices to address the issues, but that they did not know what
actions, if any, were taken by FAA headquarters. This situation indicates
the lack of a key management control that calls for information to be
recorded and communicated to individuals within an entity who need the
information in a form and within a time frame that enable them to carry
out their responsibilities. However, unless the inspectors have access to
a secure intra-agency Web site and know how to locate the issue they
submitted, they do not know how FAA headquarters has responded to the
risks because this information is not directly provided to them by
headquarters. Headquarters officials acknowledged that there is no formal
feedback loop by which to inform the inspectors about the issues that they
raised. In prior work, we found that a lack of communication with
inspector staff on the resolution of enforcement actions that resulted
from inspections is frustrating to inspectors and might be a disincentive
to reporting violations
30
20
10 0 Very great Great Moderate Some No extent
Source: GAO survey of FAA inspectors.
Note: Approximately 1 percent of inspectors responded that they did not know.
FAA Lacks Effective Communication in How Internal Risks Are Resolved
they find during inspections.38 This suggests that a similar lack of
communication could reduce inspectors' interest in identifying internal
risks to FAA. We found that the overall number of times internal risks
were identified by inspectors declined for fiscal years 2002 through 2004.
PTRS Data Are Missing Elements That Would Make the Data More Useful
The PTRS database has limitations that reduce the usefulness of the data
in helping inspectors ensure that they are effectively conducting
inspections. PTRS does not facilitate FAA's ability to monitor whether
risks identified through SEP have been mitigated. While the SEP process
generates required inspection activities based on risks that have been
identified through analysis of prior inspections, inspectors record these
inspection activities as closed in the PTRS database once they have
completed the inspection, whether or not the risks identified in SEP have
been mitigated, according to some principal inspectors we spoke with.
While inspectors are expected to use the comments section in PTRS to
record additional information, such as whether identified risks have been
mitigated, they do so infrequently. Our analysis of about 1.8 million PTRS
records showed that fewer than 400 records included any form of the words
"mitigate," "fix," or "resolve." Moreover, such analysis is time consuming
to conduct because a uniform coding scheme is not employed for entering
the information. As a result, managers are not easily able to use PTRS
data to monitor whether risks identified through SEP have been mitigated
and lack data that would be useful in evaluating SEP. It is left to
inspectors to determine subsequently if the risks have been mitigated,
usually through follow-up onsite inspections.
In addition, while a specific risk identified through SEP might result in
multiple inspections, there is no clear linkage between these inspections
when they are recorded in PTRS, according to a regional SEP manager. For
example, if a risk was identified through SEP-generated inspection
activities that included looking at manuals, checking aircraft records,
and performing a ramp check, inspectors would have to review all comment
records to identify the resulting activities in PTRS that are related to
that particular risk. This lack of linkage in PTRS may reduce inspectors'
overall understanding of an airline's situation. Inspectors who do not
know why activities have been generated may not target or fit their work
to assess these identified risks, while a greater understanding of how
inspection
38GAO/RCED-98-6.
activities are connected to identified risks could help ensure that
inspections are conducted in a way that maximizes the benefits of a
riskbased approach.
FAA Has Not Evaluated Its Inspection Oversight Process or Linked It to
Agency Goals
FAA has not evaluated its inspection oversight process for non-legacy
airlines to determine how the process contributes to the agency's mission
and overall safety goals. In addition, FAA has not explicitly linked SEP
to the overall safety goals. According to management control standards for
federal agencies and our prior reports on results-oriented management,39
federal agencies should establish measurable performance goals for their
programs and operations. Agencies should also have an evaluation process
for their programs so that agency officials, Members of Congress, and
others will be able to determine whether goals are being achieved.
While FAA has not established a specific goal for SEP, it has an overall
goal to achieve the lowest possible accident rate and constantly improve
safety. To address this goal, FAA's strategy includes expanding
cost-effective safety oversight and surveillance by targeting its
inspection resources better. However, FAA's strategy does not explicitly
show how SEP contributes to the safety goal. In addition, FAA has not yet
evaluated SEP to determine if it is achieving the agency's goal. As a
consequence of not having an evaluative process for SEP, FAA does not have
the information it needs to determine what changes should be made to
improve its system safety process. Also, FAA does not have a process to
examine the nationwide implications of or trends in the risks that
inspectors have identified through their risk assessments. Consequently,
FAA does not have the information it needs to proactively determine on a
continuous basis risk trends at the national level. While FAA has an
evaluation office, this office is not doing this type of analysis; rather,
the office conducts analyses of the types of inspections generated under
SEP by airline and FAA region, according to the manager of that office.
One FAA field office we contacted had taken the initiative to analyze
risks identified during the SEP meetings for trends. In FAA's Eastern
Region, a manager routinely analyzes trends in types of findings within an
airline and across airlines and provides this information to the
inspectors. This is an informal process, and it is not
39GAO, Managing for Results: Strengthening Regulatory Agencies'
Performance Management Practices, GAO/GGD-00-10 (Washington, D.C.: Oct.
28, 1999); and Executive Guide: Effectively Implementing the Government
Performance and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996).
certain that it will continue since the manager has been reassigned to
another office. Moreover, such analyses at the local or regional level do
not fulfill the management control to have a process in place to determine
if program goals are being achieved.
In addition, we found that while FAA has created specific national goals
for NPG that call for completing all required inspections each year, in
some cases these goals impede the agency's ability to effectively
implement its system safety approach. For example, in order to meet the
annual goal, the regional offices have established interim goals to
complete 25 percent of the required inspections each quarter. Inspectors
in one field office told us that this situation created a disincentive for
identifying additional activities under SEP due to concerns about
completing the quarterly requirements. Those inspectors also noted that
these goals encouraged them to prioritize their work based on what they
can most easily accomplish, rather than on what represents the most
significant risks-the antithesis of a system safety approach.
Conclusions Our review of FAA's oversight of non-legacy airlines suggests
that the full benefits of a system safety approach can be realized only if
the approach is more fully implemented and utilized. FAA's articulation of
its system safety approach, its application of system safety principles to
its oversight process through SEP, and its widely distributed training on
system safety are positive steps toward improving oversight by using the
advantages of system safety-particularly the ability to identify and
prioritize inspections based on risk. Fully developing SEP is important
since FAA has been unable to move significant numbers of airlines to ATOS
and has only recently established the goal to do so by the end of fiscal
year 2007. A process such as SEP is needed to identify risks in and among
non-legacy airlines and system-wide problems. However, the usefulness of
FAA's system safety approach is reduced by limitations in the
implementation of SEP-such as FAA headquarters' predominant focus on NPG,
which has led to only a small percentage of inspection activities being
SEP-initiated; the lack of training of geographic inspectors on SEP codes
that could make their inspections more useful to identifying risks during
the SEP process; and the lack of linkage to national goals and evaluations
of SEP. Until SEP is more fully implemented, it is clear that FAA's
approach to overseeing non-legacy airlines is not largely risk based.
Unless SEP is more thoroughly integrated into its oversight, FAA may not
be fully maximizing the efficiency of its inspections in identifying and
mitigating risks in order to ensure the safety of non-legacy airlines.
With FAA operating under a hiring
freeze and the number of inspectors available for non-legacy airlines
possibly further reduced by attrition and the move of inspection staff to
the ATOS program, the need to maximize the effectiveness of inspection
activities in ensuring the safety of non-legacy airlines is even more
critical.
Recommendations for Executive Action
To improve the effectiveness of the agency's oversight of non-legacy
airlines, we recommend that the Secretary of Transportation direct the FAA
Administrator to implement the following four recommendations:
o To improve its safety oversight of airlines, FAA should (1) develop a
continuous evaluative process for its activities under SEP and link SEP to
the performance-related goals and measures developed by the agency, track
performance towards agency goals, and determine appropriate program
changes. The evaluation should include an analysis of inspection findings
to identify trends and risks at the national level.
o In order to ensure that all regional and field offices have a complete
and timely understanding of FAA's policies relating to the inspection
process, FAA needs to (2) improve communication in areas such as whether
and how internal risks identified by inspectors have been resolved and (3)
improve training in areas such as risk management, coding items in the
PTRS database, and how and under what circumstances SEP-identified
activities can replace NPG-identified activities through retargeting.
o To better utilize geographic inspectors' support, FAA needs to (4)
improve the geographic inspectors' understanding of the system safety
approach and operations of the airlines they inspect. FAA should consider
actions such as additional training, additional oversight in particular
areas, having airlines' operating manuals available online for review by
inspectors, and improving communication between geographic inspectors and
principal inspectors on issues related to identifying safety violations.
Agency Comments and We provided a draft of this report to DOT for review
and comment. We obtained oral comments from FAA and DOT officials,
including FAA's
Our Evaluation Deputy Associate Administrator for Aviation Safety. In
particular, FAA officials made the point that planned NPG activities have
an element of risk identification. We agree and revised the report. The
FAA officials generally
agreed with our recommendations to improve communication, training, and
geographic inspectors' understanding of the system safety approach. The
officials said that FAA would consider our recommendation to develop a
continuous evaluative process for SEP and link it to the agency's goals,
but that its plan to eventually place the remaining non-legacy airlines in
the ATOS program might make this recommendation unnecessary. In the past,
FAA's efforts to move airlines to ATOS have experienced delays, therefore,
we retained this recommendation. FAA officials also provided clarifying
comments and technical corrections, which we incorporated as appropriate.
As agreed with your office, unless you announce the contents of this
report earlier, we plan no further distribution until 14 days from the
report date. At that time, we will send copies of this report to other
congressional committees, the Secretary of Transportation, and the
Administrator, FAA. We will also make copies available to others upon
request. In addition, the report will be available at no cost on GAO's Web
site at www.gao.gov.
If you or your staff have any questions about this report, please contact
me at (202) 512-2834 or [email protected]. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on the
last page of this report. GAO staff who made major contributions to this
report are listed in appendix III.
Gerald L. Dillingham, Ph.D. Director, Physical Infrastructure Issues
Appendix I
Objectives, Scope, and Methodology
The objective of this report is to assess the Federal Aviation
Administration's (FAA) processes for ensuring the safety of a fast-growing
portion of the commercial airline industry-the non-legacy passenger
airlines.1 Specifically, this report responds to the following questions:
(1) What are the strengths of FAA's inspection oversight of non-legacy
passenger airlines? (2) What issues hinder the effectiveness of FAA's
inspection approach?
The scope of our review included FAA's oversight activities from fiscal
year 2002-when FAA implemented a new process (the Surveillance and
Evaluation Program or SEP) to its oversight of non-legacy airlines-
through fiscal year 2004. To address the two questions, we obtained and
analyzed information about FAA's oversight of the top 25 non-legacy
airlines ranked according to the number of enplanements in 2004. Those 25
airlines are identified in table 5, and each was covered by SEP during the
entire 3-year period or a portion of the period.2 Together, these 25
airlines accounted for about 90 percent of all non-legacy passenger
enplanements in 2004.
1In this report, we refer to all passenger airlines overseen by FAA's Air
Transportation Oversight System (ATOS) as legacy airlines and those that
are not as non-legacy airlines. As of July 2005, the following passenger
airlines and cargo airlines were part of ATOS : Alaska, America West,
American, American Eagle, Champion, Continental, Delta, ExpressJet, FedEx,
Northwest, SkyWest, Southwest, Trans States, United, United Parcel
Service, and US Airways.
2Four of the 25 airlines-American Eagle, ExpressJet, SkyWest, and Trans
States-were transitioned into the ATOS program from 2003 through 2005.
Prior to then, they were covered under SEP.
Appendix I
Objectives, Scope, and Methodology
Table 5: Number of Enplanements for the Top 25 Non-legacy Airlines, 2004
Rank Airline Enplanements
American Eaglea 14,869,258
ExpressJeta 13,664,642
SkyWesta 13,417,720
Airtran 13,178,118
Comair 12,637,210
JetBlue 11,731,733
Atlantic Southeast 10,427,885
American Trans Air 10,340,914
Mesa 9,122,237
Atlantic Coast 7,046,971
Air Wisconsin 6,954,187
Frontier 6,437,921
Pinnacle 6,362,805
Horizon 5,930,448
Chautauqua 5,608,947
Mesaba 5,427,694
17 Hawaiian 5,234,766 18 Spirit 4,592,640 19 Aloha 4,187,019 20 Trans
Statesa 3,462,869 21 Executive 2,796,163 22 Midwest Express 2,376,304 23
PSA 2,030,870 24 Piedmont 1,948,292 25 Ryan International 1,626,437
Source: GAO analysis of U.S. Department of Transportation information.
aThese airlines were transitioned into the ATOS program from 2003 through
2005.
To determine FAA's legal oversight responsibility for commercial passenger
airlines, we obtained and analyzed regulations that govern FAA oversight
of these airlines. We also obtained and reviewed FAA handbooks,
procedures, and orders that describe the role and responsibilities of FAA
managers and
Appendix I
Objectives, Scope, and Methodology
inspectors3 in implementing the two inspection oversight processes for
non-legacy airlines-SEP and the National Work Program Guidelines (NPG). We
conducted literature searches and reviewed prior reports and articles on
the oversight processes, including those prepared by the Department of
Transportation's Inspector General and us. We reviewed documentation
provided by FAA, and contacted officials there and at GAO, to determine
whether findings identified in those reports have been addressed. We also
reviewed literature on system safety and compared it with FAA's system
safety framework, which is incorporated in SEP.
We interviewed FAA headquarters officials from the Office of Aviation
Safety and its Flight Standards Service to obtain descriptions of NPG and
SEP. To understand how these processes were implemented, we collected
information from 7 regional offices and 13 field and certificate
management offices. We selected these offices because they have oversight
responsibility for most of the top 25 non-legacy airlines ranked by the
number of enplanements in 2004. We also interviewed officials from 16 of
the 25 non-legacy airlines. The regional offices, field and certificate
management offices, and airlines we interviewed are shown in table 6. For
each FAA office, we conducted semistructured interviews with managers, SEP
coordinators, and inspectors to obtain information on how they implemented
NPG and SEP. We also collected information on inspector staffing levels,
workload, and training. In addition, we reviewed whether management
controls have been established for NPG and SEP,4 and determined if they
were linked to FAA's overall safety performance goals. These controls
included the (1) establishment of performance goals, (2) verification of
inspection results, (3) management of the inspection database (Program
Tracking and Reporting Subsystem or PTRS), and (4) communication among
managers and inspectors relating to NPG and SEP.
3In this report, we refer to FAA staff who perform safety audits,
inspections, and surveillance as inspectors.
4Management controls are the continuous processes that federal agencies
are required by law-i.e., the Federal Managers' Financial Integrity Act of
1982, the Government Performance and Results Act of 1993, the Chief
Financial Officers Act of 1990, and the Federal Financial Management
Improvement Act of 1996-and by the Office of Management and Budget to use
to provide reasonable assurance that their goals, objectives, and missions
are being met. We identified, with the assistance of GAO specialists in
that area, those control standards established by the Office of Management
and Budget and us that are appropriate to FAA's inspection process. See
GAO, Standards for Internal Control in the Federal Government,
GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999); and Internal Control
Management and Evaluation Tool, GAO-01-1008G (Washington, D.C.: August
2001).
Appendix I
Objectives, Scope, and Methodology
We also interviewed airline safety officials to obtain their views on
FAA's inspection and surveillance activities and their participation in
FAAindustry partnership programs.
Table 6: FAA Offices and Airlines Interviewed by GAO
Regional office FAA field office Airline
Central None None
Eastern Baltimore Flight Standards District Piedmont
Office (FSDO)
Garden City FSDO JetBlue
Washington FSDO Independence Aira
Great Lakes Detroit FSDO Spirit
Indianapolis FSDO American Trans Air Chautauqua
Minneapolis FSDO Mesaba
Chicago FSDO Air Wisconsin Midwest Express
Northwest Mountain None None
Southern Atlantic Southeast (Atlanta) Atlantic Southeast
Certificate Management Unit
Comair (Louisville) Certificate Comair
Management Office (CMO)
Orlando FSDO Executive
Airtran (Orlando) CMO Airtran
Memphis FSDO Pinnacle
Southwest None None
Western Pacific Honolulu FSDO Aloha Hawaiian
Source: GAO.
aFormerly Atlantic Coast Airlines.
We also conducted interviews with seven industry organizations to obtain
their views on FAA's oversight processes. (See table 7.) We selected these
organizations because they represented diverse segments of the aviation
industry. In addition, we selected organizations that met one or more of
the following criteria: membership inspected by FAA, familiarity with
FAA's safety oversight of airlines, familiarity with FAA's SEP or Air
Transportation Oversight System processes, and familiarity with inspector
training.
Appendix I
Objectives, Scope, and Methodology
Table 7: Organizations Interviewed by GAO
Type of organization Organization contacted
Trade associations and unions Aircraft Mechanics Fraternal Association
Professional Aviation Maintenance Association Professional Airways Systems
Specialists
Transportation safety organization National Transportation Safety Board
Organizations whose members are Regional Airline Association certificated
by FAA and subject to FAA Aeronautical Repair Station Association
oversight Airline Pilots Association International
Source: GAO.
We analyzed information about FAA's inspection activities for the top 25
non-legacy airlines using the agency's PTRS database for fiscal years 2002
through 2004. Specifically, we analyzed information on the number of
required NPG and SEP inspections, inspector workload (i.e., number and
type of entities inspected), and number of NPG activities that were
retargeted (i.e., NPG-identified activities that were replaced with
SEPidentified activities). This analysis excluded enroute activities,
which take place when an inspector monitors an aircraft as it travels from
one destination to another. These inspections were excluded because
inspectors often use these as secondary inspections as a means to travel
to a location where their primary inspection is such as a repair station.
In addition, we analyzed data that FAA maintains on the internal risks
identified by inspectors through regular SEP meetings. For fiscal years
2002 through 2004, we examined the type, severity, and closure status of
the identified risks. We also analyzed comment fields in that data to
determine if they indicated that the risks had been mitigated.
To assess the reliability of the data used in this report, we interviewed
knowledgeable agency officials about the data, performed electronic
testing of relevant data fields for obvious errors in accuracy and
completeness, and collected and reviewed documentation from data system
managers about the data and the systems that produced them. We determined
that the data were sufficiently reliable for the purposes of this report.
To gather information about inspectors' perspectives on the technical
training available to them, we conducted a Web-based survey of a
representative sample of FAA safety inspectors. The survey asked a
combination of questions concerning the amount and timeliness of training
Appendix I
Objectives, Scope, and Methodology
that allowed for open-ended and close-ended responses. We had no practical
way to assess information on the amount of training necessary for
inspector proficiency or the timeliness of the training provided. We drew
a stratified random probability sample of 496 inspectors from the
population of 2,989 FAA aviation safety inspectors.5 We stratified the
population into 12 groups on the basis of the type of work the inspector
performed. Each sample element was subsequently weighted in the analysis
to account statistically for all members of the population. For this
report, we used a subgroup sample of 205 non-legacy safety inspectors.
Because we followed a probability procedure based on random selections,
our sample is only one of a large number of samples that we might have
drawn. Since each sample could have provided different estimates, we
express our confidence in the precision of our overall sample's results as
a 95 percent confidence interval (e.g., plus or minus 7.1 percent). This
is the interval that would contain the actual population value for 95
percent of the samples we could have drawn. As a result, we are 95 percent
confident that each of the confidence intervals in this report will
include the true values in the study population. The percentage estimates
for the non-legacy subgroup has a margin of error of plus or minus 7.1
percent or less. Survey estimates presented as comparisons between groups
are statistically significant when the 95 percent confidence intervals do
not overlap.
The survey was conducted using self-administered electronic questionnaires
accessible on the Internet through a secure Web browser. We sent e-mail
notifications to 496 inspectors (205 of which were in our subgroup of
inspectors of non-legacy airlines) beginning on December 6, 2004. We then
sent each potential respondent a unique password and user name to ensure
that only members of the target population could participate in the
survey. To encourage respondents to complete the questionnaire, we sent a
subsequent e-mail message to further prompt each nonrespondent
approximately 2 weeks after the initial e-mail message. We sent
nonrespondents two more notices and closed the survey on February 4, 2005.
Of the 496 inspectors we surveyed, we received 392 usable responses (79
percent). Among our subgroup of 205 non-legacy inspectors, we received 161
usable responses (79 percent).
5Our population included only those inspectors that actively participate
in inspection activities as part of their regular job duties. It did not
include managers, supervisors, or inspectors detailed to headquarters or
regional offices.
Appendix I
Objectives, Scope, and Methodology
In addition to sampling errors, the practical difficulties in conducting
surveys of this type may introduce other types of errors, commonly
referred to as nonsampling errors. For example, questions may be
misinterpreted, or the respondents' answers may differ from those of the
inspectors who did not respond. We took steps to reduce these errors.
Finally, we pretested the content and format of the questionnaire with
safety inspectors at local FAA offices in Baltimore, Los Angeles, and
Seattle. During the pretests we asked the inspectors questions to
determine whether (1) the survey questions were clear, (2) the terms used
were precise, (3) the questionnaire placed an undue burden on the
respondents, and (4) the questions were unbiased. We made changes to the
content and format of the final questionnaire based on the pretest
results.
We conducted our work from August 2004 through September 2005 in
accordance with generally accepted government auditing standards.
Appendix II
FAA's Surveillance and Evaluation Process
SEP Incorporates System Safety into FAA's Inspection Oversight of
Nonlegacy Airlines
The Surveillance and Evaluation Program (SEP) is a process designed to
introduce a data driven risk analysis system for non-legacy airlines and
is guided by a Surveillance and Evaluation Assessment Tool (SEAT). SEP
models itself on a system safety approach that is incorporated in the
oversight of legacy airlines through the Air Transportation Oversight
System (ATOS). SEP allows teams of inspectors to identify suspected trends
through a data based system rather than relying totally on past
experiences. A certificate management team oversees each of the 99
airlines under SEP. Each team is led by three principal inspectors, one
for each major area of inspections (operations, maintenance, and
avionics). Additional team members include those based at the FAA office
that holds the airline's operating certificate-typically an aircraft
dispatch inspector, a cabin safety inspector, and assistants.
The principal inspectors are required to meet as a team at least twice a
year to assess the safety risks associated with the airlines they oversee
and develop a surveillance plan for the airline, including designating
inspection duties to other FAA offices at locations to which the airline
flies or conducts business. In some field offices, teams meet each
quarter. The teams use a planning tool-SEAT-to assess the systems in place
at an airline and to identify any potential internal and external risks.
Using SEAT, the team analyzes 10 systems, shown in table 8.
Table 8: Safety Systems and Examples of Risk Indicators in SEAT
System and purpose Selected examples of risk indicators in SEAT
Aircraft configuration and control: 1. Does the airline have the proper
maintenance and inspection programs in place? maintains the physical
condition of the 2. Do the airline and vendor follow their fueling
procedures, policies, and controls? aircraft and associated components 3.
Is the airline following its approved deicing procedures and policies?
Manuals: controls the information and 1. Is content consistent and
complete across manuals? instructions that define and govern an 2. Are
manuals up-to-date and available? airline's activities
Flight operations: governs aircraft 1. Are passengers boarding in a safe
environment?
movement 2. Are airlines ensuring that cargo is handled and carried safely
according to their policies?
3. Does the airline adhere to its approved weight and balance program?
Personnel training and qualifications: 1. Does the airline adhere to its
training programs for crew members (attendants, dispatchers, ensures that
an airline's personnel are station personnel, check airman and
instructors, and maintenance personnel)? trained and qualified 2. Are
current and appropriate certifications of personnel available upon
request, and does
the airline have the programs, policies, and procedures in place to ensure
certificates are valid?
Appendix II
FAA's Surveillance and Evaluation Process
(Continued From Previous Page)
System and purpose Selected examples of risk indicators in SEAT
Route structures: maintains an airline's facilities on approved routes
1. Does the airline adhere to its policies on weather-reporting
facilities?
2. Do the airline's maintenance and service facilities comply with its
policies, procedures, and controls?
Airman/crew member flight, rest, and
1. Does the airline adhere to its policies on flight crew (attendant or
dispatcher) flight/duty/rest
duty time: prescribes time limitations for airline employees
time?
2. Does the airline adhere to its procedures and controls for its
scheduling and reporting system?
Technical administration: addresses 1. Does the director of maintenance
(chief inspector, director of operations, and chief pilot)
other aspects of an airline's certification accomplish assigned duties and
responsibilities?
and operations, including key safety 2. Does the director of safety
effectively administer the safety program?
personnel and programs
Risk indicators: reflects the impact 1. Consider the impact of changes in
required management personnel (airline management,
external and internal events have on an turnover in personnel, reduction
in workforce, layoffs, buyout, etc.).
airline's system safety and stability 2. Consider the complaints that
affect surveillance planning and how that airline responded to
them.
3. Consider the age of the airline's fleet, its process to survey and
inspect aging aircraft, and the effectiveness of its aging aircraft
program.
Other: records the presence and 1. Does the airline understand FAA's
authority to conduct drug tests and demonstrations?
implementation of airline developed 2. Does the airline have a security
(drug and alcohol) program and is the airline in compliance
security and substance abuse with the program?
programs
Certificate management risks: reflects 1. Consider if there is an adequate
number and type of inspectors assigned to the certificate.
the impact of FAA's resources on 2. Consider if there is an adequate
amount of geographic surveillance.
providing oversight of an airline 3. Consider the resources (travel
budget, country clearance, etc.) of the certificate holding
district office.
Source: FAA.
To complete SEAT, the principal inspectors rely on their knowledge of the
airline and on the data available through FAA's Safety Performance
Analysis System database, which contains the Program Tracking and
Reporting Subsystem and 12 additional databases with safety and
performance information. SEAT includes a set of risk indicators for each
of the elements (often in the form of a question) to be discussed by the
team to indicate concerns about any real or potential problem that could
contribute to the failure of one of the airline's elements, subsystems, or
systems. Inspectors rate each risk they identify in terms of severity
(negligible to catastrophic) and likelihood of occurrence (improbable to
frequent), and SEAT calculates an overall risk rating. Risks rated high
have high levels of likelihood and severity.
Using this information, the team identifies a set of required inspections
that must be completed during that fiscal year and planned inspections.
The required inspections are added to the NPG-identified inspections, and
the combined list becomes the annual surveillance plan for the airline.
During
Appendix II
FAA's Surveillance and Evaluation Process
the SEAT process, principal inspectors are allowed to substitute
NPGrequired inspections with SEP-identified inspections targeting
higher-risk areas.
SEP Incorporates Table 9 describes FAA's three inspection processes for
overseeing legacy
and non-legacy airlines: ATOS, NPG, and SEP. Many of the elements ofSome
Elements of ATOS, such as the use of data to identify risks and the
development of ATOS, While NPG surveillance plans by inspectors, are
incorporated in the SEP process. The Relies on a Set Number NPG process,
in contrast, is not focused on the use of data and relies on an
established set of inspections that are not risk based.
of Inspections
Table 9: Various Elements of ATOS, NPG, and SEP
ATOS NPG SEP
Description of o Focuses on safety o Focuses on inspectors completing
o Focuses on inspectors conducting a
program vulnerabilities rather than a prescribed number of inspection risk
assessment of various areas regulatory compliance activities o
Inspectors review data to identify areas
o Analysts and inspectors review o Primarily based on checking airline
of safety risk and use Flight Standards airline data to identify areas of
compliance with regulations Safety Analysis Information Center and safety
risk o Relies on inspectors' expertise to the Safety Performance
Analysis
o Inspectors develop surveillance identify trends and risks System as
analytical tools plans for each airline, based on o Inspectors develop
surveillance plans data analysis and assessment for each airline, based on
data analysis of risks, and adjust the plans and assessment of risks, and
adjust periodically based on plans periodically based on inspection
inspection results results
o Inspectors can also verify that planned NPG activities meet the
surveillance needs for a particular year
Type of Legacy commercial Non-legacy Non-legacy commercial
commercial airlines commercial airlines airlines
passenger
airline
inspected
Continuous safety Periodic; regular Periodic; inspections
Frequency of oversight inspections are are established
inspections established during meetings held
annually by an FAA at least twice a
headquarters year using risk-based
committee criteria
Approximate number 585 1,100b 1,100 b
of aviation safety
inspectors
conducting
inspectionsa
Appendix II
FAA's Surveillance and Evaluation Process
(Continued From Previous Page)
ATOS NPG SEP
Number of 14c 99 99
commercial
passenger airlines
under the programa
Sources: GAO and FAA.
aAs of July 2005.
bThere are a total of about 1,100 inspectors for both NPG and SEP
inspections.
cFedEx and United Parcel Service, two cargo air carriers, are also in the
ATOS program.
Appendix III
GAO Contact and Staff Acknowledgments
GAO Contact Gerald L. Dillingham, (202) 512-2834
Staff In addition to the above, Teresa Spisak, Assistant Director; Carl
Barden; Bari Bendell; Curtis Groves; Bert Japikse; Catherine Kim; Minette
Acknowledgments Richardson; Phillis Riley; Larry Thomas; Alwynne Wilbur;
and Matt Zisman made key contributions to this report.
GAO's Mission The Government Accountability Office, the audit, evaluation
and investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To Testimony have GAO e-mail you a list of newly posted products
every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Congressional Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125
Relations Washington, D.C. 20548
Public Affairs Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***