Aviation Safety: FAA's New Inspection System Offers Promise, but Problems
Need to Be Addressed (Letter Report, 06/28/1999, GAO/RCED-99-183).
The aviation industry has forecast a potential 66-percent increase in
passenger travel from 1999 to 2008. The U.S. aviation accident rate,
which has remained relatively constant during the last two decades, must
be substantially lowered to avoid escalating numbers of aviation deaths
as air traffic increases. A key to reducing the aviation accident rate
is for the Federal Aviation Administration (FAA) to have an effective
process for inspecting the nation's airline operations. GAO and others
have raised concerns about the adequacy of FAA's inspection process to
meet that challenge. The report addresses the following questions: To
what extent does the Air Transportation Oversight System address past
concerns about FAA's aviation safety inspections? What factors, if any,
surfaced during the system's implementation that could impede its
success? What is FAA doing to address any factors that could impede the
system's success? GAO found that the system is largely responsive to
past concerns raised about key aspects of FAA's aviation safety
inspections and the usefulness of inspection data.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-99-183
TITLE: Aviation Safety: FAA's New Inspection System Offers
Promise, but Problems Need to Be Addressed
DATE: 06/28/1999
SUBJECT: Transportation safety
Management information systems
Airline regulation
Safety regulation
Airline industry
Commercial aviation
Safety standards
Data bases
Inspection
IDENTIFIER: FAA Air Transportation Oversight System
FAA Safety Performance Analysis System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
United States General Accounting Office GAO Report
to the Subcommittee on Aviation, Committee on Transportation and
Infrastructure, House of Representatives June 1999
AVIATION SAFETY FAA's New Inspection System Offers Promise, but
Problems Need to Be Addressed GAO/RCED-99-183 GAO United
States General Accounting Office Washington, D.C. 20548 Resources,
Community, and Economic Development Division B-281502 June 28,
1999 The Honorable John J. Duncan Chairman The Honorable William
O. Lipinski Ranking Democratic Member Subcommittee on Aviation
Committee on Transportation and Infrastructure House of
Representatives The aviation industry has forecast a potential 66-
percent increase in passenger travel from 1999 to 2008. The U.S.
aviation accident rate, which has remained relatively constant
over the past two decades,1 must be substantially lowered to avoid
escalating numbers of aviation deaths as air traffic increases. A
key to reducing the aviation accident rate is for the Federal
Aviation Administration (FAA) to have an effective process for
inspecting the nation's airline operations. In the past, we and
others have expressed concerns about the adequacy of FAA's
inspection process to meet that challenge. Concerns about the
inspection process focused on unstructured, nonsystematic
inspections that produced few reports of safety problems and on
the adequacy of inspectors' technical training. These concerns
also raised questions about the quality and consistency of the
resulting inspection data and their usefulness for conducting
analyses and targeting FAA's resources to the greatest safety
risks. FAA has responded to these concerns by redesigning the
safety inspection system that it uses to oversee the nation's
airlines. FAA began using the revised approach, called the Air
Transportation Oversight System (ATOS), for a limited number of
airlines during the system's initial implementation on October 1,
1998. Currently, the nation's 10 largest passenger airlines are
under ATOS.2 At your request, we reviewed FAA's implementation of
the new system. This report summarizes our work by addressing the
following questions: * To what extent does ATOS address past
concerns about FAA's aviation safety inspections? * What factors,
if any, surfaced during the implementation of ATOS that could
impede its success? 1The National Transportation Safety Board's
statistics show an accident rate of 5 fatal accidents for each 10
million flights on scheduled and nonscheduled service by U.S.
airlines operating under part 121 of the Federal Aviation
Regulations from 1982 through 1998. 2These airlines are Alaska,
America West, American, Continental, Delta, Northwest, Southwest,
Trans World, United, and US Airways. Page 1
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 *
What is FAA doing to address any factors that could impede the
success of ATOS? Results in Brief The Air Transportation
Oversight System is largely responsive to past concerns raised
about key aspects of FAA's aviation safety inspections and the
usefulness of inspection data. These concerns centered on FAA's
unstructured inspection process, the adequacy of technical
training for inspectors, the quality and consistency of inspection
data, and the usefulness of those data for identifying safety
problems and targeting the agency's resources to the greatest
risks. Addressing these concerns involved a fundamental redesign
of the way FAA inspects the nation's airlines. To improve
inspection quality, the new program emphasizes a system safety
approach that goes beyond spot-checking airlines for compliance
with Federal Aviation Regulations. Using safety principles
originally created for the nuclear industry, it calls for a
systematic review of airlines' policies and procedures to ensure
that they incorporate basic safety principles, such as clear lines
of responsibility and written documentation. It fosters more
consistent, structured inspections by standardizing inspection
tasks, linking inspectors' training more closely to their assigned
responsibilities, and using teams rather than individual
inspectors to perform many inspections. The program also calls for
a number of enhancements to improve the usefulness of inspection
data for analysis and targeting. They include a standardized
database for reporting inspection results and the addition of data
quality assurance managers and analysts. The goal of this redesign
is to target inspection resources to those areas that present the
greatest safety risks. ATOS offers promise for significantly
strengthening FAA's inspection process, but FAA must also address
the problems identified in this report to ensure that the new
system fulfills its promise. FAA's ability to conduct effective
inspections remains limited by a lack of clear guidance, staff
turnover, and continued difficulties with the adequacy of
inspectors' technical training and experience. The anticipated
enhancements to make inspection data more useful have not been
achieved because of problems with reporting requirements and the
incompatibility of the program's database with FAA's primary
inspection analysis system. In addition, FAA planned to hire an
analyst for each of its new inspection teams to analyze inspection
data for safety trends and to guide inspection planning, but has
not yet done so because of higher priorities, such as increasing
salaries for air traffic controllers. These problems resulted
largely from FAA's decision to implement the new inspection system
on an overly ambitious schedule. Page 2
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
Meeting FAA's target date for implementation meant that complex,
critical steps-such as developing guidance, training inspectors,
creating databases, and consulting with the affected airlines-had
to be compressed into a very short time. FAA has begun to address
some of the problems with the program. FAA management officials
acknowledged that ATOS faces significant challenges. They agreed
with our conclusion that the program should not be expanded beyond
the nation's 10 major airlines until the problems that emerged
during the program's initial implementation are resolved. However,
some of these problems have not yet been fully addressed.
Consequently, we recommend several specific actions to clarify the
program guidance and improve the usefulness of FAA's database for
targeting inspection resources to the areas of greatest potential
safety risk. Background Federal law establishes that the
safety of U.S. air passengers is a joint responsibility of the
airlines and FAA. The airlines are responsible for operating their
aircraft safely. FAA is responsible for, among other things,
examining an airline's operations when the airline seeks a
certificate to operate and for conducting periodic inspections to
ensure continued compliance with safety regulations. Within FAA,
the Office of Flight Standards Service develops the Federal
Aviation Regulations that airlines must follow and prepares
guidance on how FAA's safety inspectors should perform
inspections. FAA has nearly 3,300 safety inspectors located in 101
district offices throughout the United States. One of the
inspectors' primary functions is conducting what FAA calls
"routine surveillance"-a process of continuous periodic safety
inspections of airlines and aviation-related activities.3 These
inspections include having an inspector visually spot-check an
airplane at the gate, monitor procedures on a scheduled flight, or
observe maintenance being performed on an aircraft or its
component parts. The inspections cover four main areas: *
Operations inspections focus on such items as pilots'
certification and performance, flight crews' training, and in-
flight record keeping. * Maintenance inspections examine an
airline's overall maintenance program, including the training of
aviation mechanics, the development of 3Other primary functions
include certifying airlines' operations, investigating accidents
and incidents, and taking other steps to promote safety. Page 3
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
maintenance manuals, and procedures for repairing aircraft and
their components. * Avionics inspections focus on electronic
components of the aircraft. * Cabin safety inspections concentrate
on cabin procedures, passenger safety, and carry-on baggage. In
recent years, we and others have reported on problems with FAA's
inspections.4 Reacting to these reports and to a series of fatal
aviation accidents in the mid-1990s, FAA conducted a number of
studies of its own, including the report entitled FAA 90 Day
Safety Review, issued in September 1996.5 This review recommended
that inspections be made more systematic and that they be targeted
to deal with identified risks, such as airlines' financial
instability and significant contracting out of maintenance. ATOS,
developed by the Office of Flight Standards Service and adapted in
part from safety principles originally created for the nuclear
industry, resulted from these recommendations. The ATOS Concept Is
FAA incorporated features in ATOS to address past concerns about
Responsive to Many inspection quality and the usefulness
of inspection data for identifying potential safety threats and
for targeting resources to areas that pose the Past Concerns About
greatest risk. Before developing its new aviation safety
inspection system, FAA's Safety FAA analyzed past
concerns about its inspections. The resulting ATOS inspection
concept focuses on ensuring that an airline has operating
Inspections systems in place to control the
potential hazards and risks of flying and to prevent accidents.
FAA structured ATOS to evaluate both an airline's operating
systems and its adherence to those systems in day-to-day
operations. In addition, FAA incorporated specific features into
ATOS to make inspections more consistent, structured, and thorough
and to improve the collection and analysis of inspection data.
ATOS Focuses on System FAA emphasizes a system safety
approach in ATOS that replaces routine Safety and Accident
surveillance and goes beyond spot-checking airlines for compliance
with Prevention aviation regulations. System
safety involves the application of technical and managerial skills
to identify, analyze, assess, and control hazards and risks. It
covers every aspect of an airline's operations, from the design of
4For a list of relevant reports, see the bibliography and the
section citing related GAO products at the end of this report.
5Challenge 2000: Recommendations for Future Aviation Safety
Regulation, prepared for the Federal Aviation Administration by
Booz-Allen & Hamilton Inc. (Apr. 1996); and FAA 90 Day Safety
Review, Federal Aviation Administration (Sept. 16, 1996). Page 4
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 the
hardware to the culture and attitudes of the airline's personnel.
ATOS calls for a systematic review of an airline's policies and
procedures to ensure that they incorporate such basic safety
principles as clear lines of responsibility and written
documentation. FAA intended ATOS to ensure that an airline has and
follows policies and procedures that build in system safety. FAA
implemented ATOS on October 1, 1998, to cover the nation's 10
major passenger airlines. Under ATOS, FAA assigns a team of
inspectors to oversee each airline. Three principal inspectors
lead the team, one for each major area of inspections (operations,
maintenance, and avionics). Additional team members can be based
in one of two ways. Inspectors based at the FAA office that holds
the airline's operating certificate work full time on the ATOS
team. In contrast, field inspectors, who work in other locations
to which the airline flies, work part time on the ATOS team and
complete additional duties, such as accident investigations and
other inspections, for the local FAA office to which they are
assigned. About 540 inspectors are assigned to the 10 ATOS teams.
Each ATOS team also includes one cabin safety specialist, whose
inspections focus on such areas as flight attendants' training,
carry-on baggage, and emergency evacuation procedures. FAA
included two kinds of guidance in ATOS to help a team plan and
carry out inspections of the airline it oversees. First, automated
ATOS planning guidance is used to develop the comprehensive
surveillance plan for each airline. The planning guidance calls
for using existing safety data, risk indicators, and the
inspectors' knowledge of an airline's operations to determine the
priority and frequency of inspection activities. The resulting
comprehensive surveillance plan includes a series of inspection
tasks to determine whether the airline has systems in place to
ensure safety and a second series of inspections to verify that
the airline is actually using those systems. FAA also developed
ATOS guidance for conducting inspections that is intended to
describe the tasks to be performed for each type of inspection.
For a more detailed description of the ATOS guidance and the
development of the comprehensive surveillance plan, see appendix
I. FAA designed ATOS to be improved on an ongoing basis. FAA has
established an ATOS Program Office to formulate and implement
changes to ATOS and to support FAA inspection teams through a
hotline, help desk, and Web site. FAA has also established an
internal audit team of aviation safety inspectors to evaluate the
program, the System Process Audit Group. This internal audit team
is an independent FAA organization that reports directly to the
Director of Flight Standards. Page 5
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 ATOS
Addresses Many In the past, Flight Standards typically
allocated a large portion of its Past Concerns About inspection
resources to thousands of unstructured inspections that Inspection
Quality produced few reports of problems. We reported last
year, for example, that inspectors reported no problems for 96
percent of the inspections they conducted in fiscal years 1990
through 1996.6 Reviews of FAA's inspection program suggest that
FAA detects more problems through rigorous structured inspections
than through unstructured inspections. Our recent review of FAA's
oversight of the facilities airlines use to repair aircraft
confirmed that standardizing inspection tasks through the use of
checklists promotes more comprehensive inspections.7 Past concerns
also included problems with inspectors' training. Specifically,
inspectors have performed inspections for which they did not have
appropriate or current credentials, in part because of limited
funding for training. Providing adequate technical training for
FAA's inspector workforce has proven difficult because of the
rapid change in aviation technology. In addition, airlines can
meet regulatory requirements in a variety of ways, making it
difficult for FAA's inspectors who inspect many different airlines
to be familiar with the FAA-approved procedures of each airline.
Both principal inspectors and airline officials we interviewed
said that this lack of familiarity sometimes resulted in airlines'
being unfairly cited for noncompliance. Finally, our review of
aircraft repair facilities noted that individual inspectors
generally identify far fewer deficiencies than teams do. The
unstructured inspection activities and the underreporting of
violations by inspectors resulted in inaccurate, incomplete, and
inconsistent information that was not very useful for analyzing
safety risks or targeting the agency's resources to the problems
that pose the greatest risks. FAA included features in ATOS to
move toward more consistent, structured inspections by using a
system safety approach and by providing new, standardized
inspection tasks. FAA developed automated ATOS planning guidance
to ensure that inspectors use the same criteria to determine the
annual inspection activities for each of the major airlines. An
ATOS team uses the planning guidance to identify potential problem
areas at each airline that should be inspected more frequently.
Similarly, to standardize inspection activities across airlines,
the guidance for conducting inspections lists tasks for each
inspection. The inspection guidance is designed to ensure that
each inspector looks at an airline's systems and 6Aviation Safety:
Weaknesses in Inspection and Enforcement Limit FAA in Identifying
and Responding to Risks (GAO/RCED-98-6, Feb. 27, 1998). 7Aviation
Safety: FAA Oversight of Repair Stations Needs Improvement
(GAO/RCED-98-21, Oct. 24, 1997). Page 6
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
elements in the same way. It is also supposed to serve as a
checklist that inspectors can use to ensure that their inspections
are thorough. While the old inspection system listed multiple
tasks that could be performed, inspectors were not required to
complete any specific tasks or to document which tasks they
performed. Thus, the scope of the inspection work actually
completed could not be determined. In contrast, ATOS requires
inspectors to document whether the airline being inspected
complied with each item on the inspection checklist. The checklist
also serves as a template for reporting inspection results in the
ATOS database. FAA also incorporated team inspections in the ATOS
approach. We have reported in the past that teams have been more
effective than individuals in identifying areas where airlines
were not in compliance with FAA regulations. In many cases, the
deficiencies identified by teams are systemic and long-standing.
Under ATOS, teams will identify deficiencies and plan inspections.
Many inspections will be performed by teams rather than by
individual inspectors, as has been done in the past. Individual
inspectors will continue to perform some of the inspection work
identified in the plan. FAA also included several features in ATOS
to address past concerns about inspectors' training by more
effectively linking inspectors' technical training and
qualifications to their job responsibilities. First, FAA designed
ATOS to link inspection assignments to the technical background of
each inspector and to identify any additional technical training
needed to accomplish the work plan. Inspectors cannot adequately
inspect aircraft or systems unless they have had the appropriate
technical training. Second, inspectors assigned to an airline,
including field inspectors, must complete training on both ATOS
and the airline's specific policies and procedures before they can
conduct inspection activities. The training on ATOS provides an
overview of the system safety concept and how it differs from
FAA's past inspection approach. The training on the airline's
policies and procedures familiarizes inspectors with the approved
operating procedures of the airline they oversee. Page 7
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 ATOS
Addresses Many FAA needs complete, accurate inspection
data to target its limited Past Problems With the inspection
resources to the areas that pose the greatest potential safety
Usefulness of Inspection risks. We reported in 1989 and again
in 1991 on inaccurate and incomplete Data
data in FAA's inspection database and recommended in 1995 that FAA
develop a comprehensive and coordinated strategy for improving its
data.8 More recently, we reported last year that 70 percent of
Flight Standards' inspectors did not enter all of the violations
they found into their inspection tracking system in fiscal year
1996.9 Some inspectors said they did not report violations when
compliance could be achieved informally by bringing problems to
the attention of the airlines. Others said they handled less
serious violations informally because the paperwork involved in
reporting violations was too burdensome. FAA is implementing a
streamlined procedure for documenting and processing minor
administrative violations, which should better enable the agency
to target its limited inspection resources to the areas that pose
the greatest risks. The streamlined procedure will reduce
paperwork for some types of enforcement cases, but other efforts
will be needed to ensure the complete, accurate inspection data
needed for improved targeting of inspection resources. FAA
included several features in ATOS to address past concerns about
the usefulness of inspection data for analysis and targeting.
First, the standardization of inspections and the development of
guidance for planning and conducting inspections are steps
intended to improve the quality of FAA's data by making
inspections more systematic and thorough. When inspections are
more standardized across airlines, data quality is improved.
Second, FAA created a new position within the ATOS team overseeing
each airline: a data evaluation program manager, whose job will be
to review data for validity, accuracy, and completeness before
they are finalized in the ATOS database for analysis. ATOS also
added a new position for an analyst on each team. The analyst is
responsible for collecting and analyzing data to support
inspection planning and retargeting. Finally, FAA included
features in ATOS to improve the targeting of inspection resources.
FAA designed ATOS to allow the targeting of inspections based on
an airline's size, operations, past history, and known problem
areas. The automated planning guidance can be used to indicate the
risk factors applicable to the airline, such as whether an airline
is a 8Aviation Safety: FAA's Inspection Management System Lacks
Adequate Oversight (GAO/RCED-90-36, Nov. 13, 1989); Aviation
Safety: Problems Persist in FAA's Inspection Program (GAO/RCED-92-
14, Nov. 20, 1991); and Aviation Safety: Data Problems Threaten
FAA Strides on Safety Analysis System (GAO/AIMD-95-27, Feb. 8,
1995). 9GAO/RCED-98-6, Feb. 27, 1998. Page 8
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 new
entrant or contracts with other companies for its maintenance,
training, or ground handling services. Based on the risk factors,
the comprehensive surveillance plan targets particular areas. FAA
also recognized that needed inspection work must drive the
assigned inspection resources for each airline. Finally, ATOS
gives inspectors the flexibility to retarget resources at any
point during the year based on inspection results. This
flexibility allows FAA to focus on new problems as they surface,
rather than waiting until the next year's work. Because ATOS
currently focuses on major airlines, it does not address concerns
about the need to provide additional oversight of new entrant
airlines (that is, airlines in their first 5 years of operation).
A separate FAA initiative, the Certification Standardization and
Evaluation Team, has standardized and automated the process for
granting operating certificates to new airlines. ATOS system
safety concepts have been integrated into the certification
process for new airlines. The new certification concept includes a
national team to assist local district offices in reviewing the
applications of new airlines and monitoring these airlines for
their first 5 years of operation. As new airlines receive
certification, FAA plans to oversee them using the ATOS program.
FAA is not, however, currently providing any additional oversight
of new entrant airlines that were already in operation prior to
the new certification process. ATOS' Design and
Problems that emerged during design and implementation limit the
Implementation potential of the ATOS concept to
bring about needed improvements in FAA's aviation safety
inspections. Problems with the ATOS inspection guidance, Problems
Limit FAA's the links between inspectors' qualifications
and their work assignments, Efforts to Improve and
assembling effective teams affect the improvements envisioned for
inspection quality and the usefulness of inspection data for
analysis and Safety Inspections targeting. In addition,
FAA did not take advantage of the expertise of airline or industry
representatives in developing ATOS. Design and
Although ATOS calls for (1) more systematic, structured
inspections, Implementation Problems (2) closer links between
inspectors' training and their assigned work Limit Improvements to
responsibilities, and (3) greater use of team inspections to
improve Inspection Quality inspection quality, its
success in the first 6 months has been limited: * Inspection
guidance is not complete and is not sufficiently clear and
detailed to accomplish the systematic, structured inspections
promised by the ATOS concept. Page 9
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 *
ATOS has not resolved the long-standing problems with matching
inspectors' qualifications to their work assignments. * Team
inspections are hampered by problems with assigning inspectors,
including turnover, work locations that do not match inspection
needs, an unwillingness of inspectors to travel, and FAA's
inability to estimate the resources needed to complete ATOS
inspections. ATOS Inspection Guidance The new inspection
guidance is not adequate to ensure the Needs Significant
Improvement comprehensive, standardized inspections
envisioned by the ATOS concept. We found several problems with the
ATOS inspection guidance as implemented. Guidance does not cover
all applicable regulations. One basic purpose of inspections is to
ensure that an airline complies with Federal Aviation Regulations.
Although ATOS aims to go beyond ensuring compliance to see whether
safety is built into an airline's operations, compliance should
also be confirmed. Principal inspectors we interviewed expressed
several concerns about the link between the ATOS inspection
guidance and aviation regulations. One group of inspectors
analyzed the ATOS inspection guidance to determine whether it
covered all applicable Federal Aviation Regulations. They
identified 296 specific regulatory requirements that the ATOS
inspection guidance did not address out of approximately 2,300
applicable requirements. While we have not verified each of the
296 requirements said to be missing, our sampling of the
inspectors' results confirmed that ATOS overlooked some key
regulatory requirements. For example, the inspection guidance does
not cover regulations requiring airline employees to be trained to
handle hazardous materials. Because the ATOS guidance does not
include all applicable regulatory requirements, inspections may
not be thorough enough to ensure compliance. Some guidance is not
applicable to ATOS airlines. In addition to the regulatory
requirements not covered by ATOS, principal inspectors we
interviewed identified a number of regulations referenced in ATOS
that are not applicable to the major airlines currently under
ATOS. For example, FAA requires that major airlines maintain an
aircraft's weight and balance to ensure that it remains within
approved limits. However, the ATOS inspection guidance for the
weight and balance program is based on the FAA regulations
governing commuter airlines.10 Because ATOS currently covers only
the largest airlines, the inspection guidance should exclude
10Commuter airlines are those that conduct scheduled passenger-
carrying operations in aircraft that have 10 or fewer seats and
operate under part 135 of the Federal Aviation Regulations. Page
10 GAO/RCED-99-183 FAA's New
Aviation Inspection System B-281502 those regulations that are
applicable to commuter airlines or other types of operators, such
as general aviation aircraft. Principal inspectors we interviewed
also questioned the appropriateness of some parts of the ATOS
guidance that have no basis in regulatory requirements. For
example, the ATOS guidance calls for reviewing the rsums of some
airline officials, although regulations do not specify
qualifications and experience for their positions. Principal
inspectors also questioned the basis for some of the ATOS
determinations that rely on very subjective judgments, such as
whether the airline has a "safety focus." They noted that the ATOS
guidance does not distinguish inspection tasks and findings based
on regulations, which are legally enforceable, from those based on
such other sources as inspector handbook guidance, which is
advisory. Safety officials at most of the major airlines echoed
the inspectors' concerns. Guidance is not sufficiently clear and
detailed. The fact that ATOS provides guidance to inspectors on
how to plan and perform their inspections represents a major step
toward the standardization of inspection tasks. However, the
guidance it offers is not yet thorough or detailed enough to
achieve that goal. Principal inspectors we interviewed questioned
its usefulness, saying that it was not clear or detailed enough.
They reported that they found the language of the guidance for
planning inspections difficult to use because it does not detail
the tasks to be performed well enough. Staff at Sandia National
Laboratories, who were asked by FAA to comment on the ATOS program
because of their expertise in system safety in the nuclear
industry, had pointed to similar concerns before ATOS was
implemented. In a report on ATOS, the Sandia staff noted that the
inspection guidance was not based on analyses of specific ATOS
inspection tasks and the recording of the results.11 The Sandia
report noted that a proper task analysis describes the steps to
take and the standards for determining that the results are
correct and complete. The inspection guidance does not provide
this level of detail. FAA's guidance material was not thoroughly
tested. The lack of clarity and detail in the ATOS guidance, both
for (1) planning and (2) conducting inspections, reflects the fact
that FAA did not thoroughly test and validate it before
implementation. When FAA first tested the planning guidance in
April 1998, the inspectors involved said they had trouble using it
because the questions were too vague and broad. When FAA later
tested a revised 11"Analysis and Data Issues for the
Implementation of a Systems Safety Focus in Air Transportation
Oversight," Oct. 6, 1998. Page 11
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
automated version of the planning guidance, most test participants
were members of the ATOS work group. Work group members filled in
the answers without actually performing the relevant analysis or
inspection tasks. While FAA tested the planning guidance, it never
tested the guidance for conducting inspection tasks. Flight
standards did not allow sufficient time for ATOS' implementation.
While FAA spent several years developing the ATOS concept, the
target implementation date left only 7 months to take the agency's
new inspection program from concept to reality. The target date of
October 1, 1998, did not allow time for the adequate development,
testing, and validation of new inspection procedures and tracking
systems or for training and preparing the inspector workforce. The
ATOS work group established in February 1998 faced multiple
challenges in meeting the implementation deadline in the allotted
7 months. These challenges included (1) implementing a completely
reengineered process for conducting inspections, (2) developing
new automated systems for planning ATOS inspections and tracking
their results, and (3) preparing FAA for the cultural changes
involved in having inspectors perform their work differently. FAA
did not adequately train inspectors to use ATOS. The full impact
of the shortcuts taken to meet the implementation deadline became
apparent during the inspectors' training and, subsequently, as the
teams of inspectors used the new ATOS planning guidance to develop
a comprehensive surveillance plan for each major airline and to
initiate inspections. FAA trained more than 800 inspectors,
managers, and other FAA staff on the ATOS concept and planning
guidance from September through November 1998. Because of the
speed with which the ATOS guidance and tracking systems were
developed, FAA had little time to develop training for its
inspectors. For example, very few of the 88 ATOS lists of
inspection tasks were available in time for the training sessions,
and the database for reporting inspection results was not yet
completed. Participants in the training sessions repeatedly noted
that the implementation of ATOS had been rushed and that there
were too many unknowns to allow for this initiative to go
smoothly. In course evaluations and in our interviews, inspectors
said that they were not adequately trained on (1) how to perform
the new inspection tasks, (2) how to record the results of their
inspections in the ATOS database, and (3) how to use the data on
inspections to retarget resources. The ATOS internal audit team
concluded, "The ATOS training . . . Page 12
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
dedicated an insufficient amount of time to training on the
[inspection guidance]." Standardized inspections and reporting
will not occur without clear, understandable guidance, detailed
information on tasks to be completed, and training in both
conducting inspections and reporting findings. Links Between
Inspectors' The ATOS model for linking inspectors'
qualifications and training to Qualifications and Assignments
assignments is not yet a reality. We found that some inspectors
still lack Are Not Fully Established experience with FAA
or major airlines or lack the specific training needed to perform
their jobs effectively. Lack of experience with FAA or major
airlines. In selecting staff to fill positions on ATOS teams, FAA
did not effectively match the qualifications of field inspectors
to their new positions. The selection of field inspectors to fill
ATOS assignments was not based on any nationwide criteria or
guidance. Principal inspectors told us that they had no say in the
selection of the field inspectors assigned to their teams. As a
result, the qualifications of some field inspectors did not match
the teams' needs. Several principal inspectors and managers said
that field inspectors assigned to them included newly hired staff
who were unfamiliar with FAA or inspectors-including experienced
FAA staff-who lacked background with major airlines. For example,
one manager said that some of the field inspectors assigned to his
team had no experience in large aircraft and lacked appropriate
qualifications. FAA principal inspectors have told us that it
takes several years to develop familiarity with the agency's
regulations and procedures or with an airline's procedures.
Consequently, newly hired personnel may need several years of
experience before they can work independently and be fully
productive. Similarly, inspectors trained on small aircraft may
need extensive on-the-job training to understand the workings of a
major airline. Principal inspectors questioned whether some team
members had the appropriate skills to oversee a major airline.
Lack of specific training. Even inspectors who have experience
with major airlines may not have the specific technical
qualifications to perform the ATOS inspection tasks planned for
the airline to which they are assigned. Principal inspectors we
interviewed said that the technical qualifications of a number of
the inspectors assigned to them did not match those needed to
oversee the airline in question. For example, one operations
inspector said that three of his field inspectors did not have the
appropriate license required to fly the aircraft used by the
airline or to conduct flight checks to observe its pilots.
Similarly, a principal avionics Page 13
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
inspector said that neither he nor his assistant had the
appropriate technical training in the Airbus aircraft that the
airline flies. Furthermore, the rapid changes in the aviation
industry make it difficult to provide sufficient technical
training to keep experienced inspectors current. These
difficulties in keeping pace with technological advances reflect
FAA's persistent problems with providing technical training for
inspectors and matching inspectors' qualifications to their job
assignments. Assignment Issues Affect the Staff turnover,
travel requirements, and conflicting demands on inspectors'
Quality of Team Inspections time undermine FAA's ability to
assemble effective ATOS teams. Many of these assignment issues
cannot be resolved within the context of the ATOS program alone
because they involve broader decisions about inspectors'
responsibilities and Flight Standards' staffing. Staff turnover.
The turnover among field inspectors assigned to the airlines
covered by ATOS has made it difficult to plan inspection work or
to meet the ATOS training requirements. Principal inspectors said
that they had lost field inspectors originally assigned to their
teams, primarily because field inspectors went elsewhere in FAA to
accept promotions that are not available to them under Flight
Standards' current staffing structure. For example, of the 28
inspectors assigned to one ATOS team, 11 had been reassigned,
leaving only 17 available. Because some replacements lacked ATOS
training, training on the airline's policies and procedures, or
both, they were ineligible to conduct inspections under ATOS.
Principal inspectors were concerned about how, if turnover
continues, to meet the requirement that inspectors be trained on
the policies and procedures of the airline they oversee. In many
cases, the airlines provided this training in cooperation with
FAA. Several principal inspectors said that repeated requests for
training would be burdensome to the airlines. Principal inspectors
noted that the turnover of field inspectors would probably
continue unless changes are made in Flight Standards' grade and
pay levels to permit field inspectors to be promoted. Inspectors'
work locations do not always match inspection needs. Because the
work locations of some inspectors assigned to ATOS teams do not
match inspection needs, the inspectors will have to travel to
complete their work. This has made it difficult to assemble
effective teams for several reasons. First, many of the principal
inspectors we interviewed told us that field inspectors were
assigned to locations where they are not needed, while parts of
the country where the airlines have substantial activity have no
field presence. For example, one principal inspector told us that
he had a field inspector assigned to his team who was located in
Page 14 GAO/RCED-99-183 FAA's New Aviation
Inspection System B-281502 Boise, Idaho, where the airline he
oversees does not fly, but that he had no one in Kansas City,
where he needs an inspector to oversee maintenance operations.
Because of these problems, a number of principal inspectors
suggested that they be given a role in identifying the needed
qualifications and work locations for field inspectors assigned to
the ATOS teams. Furthermore, in the past, teams overseeing an
airline drew on field inspectors in FAA's international offices to
inspect foreign maintenance facilities and other overseas
operations of airlines with international routes. Because FAA has
not trained inspectors in its international offices to perform
ATOS inspections, ATOS team members will have to travel overseas
to complete planned inspections, despite FAA's already limited
travel funds. ATOS does not resolve resource constraints. FAA
designed ATOS to address the inspection portion of its inspectors'
workload and did not address the inspectors' other
responsibilities. In addition to inspections, the oversight of an
airline includes ongoing activities referred to as "demand" work.
Demand work includes certification and approvals for initiatives
taken by the airline, such as adding new aircraft types to the
fleet, adding new destinations, implementing computerized record
keeping, and restructuring by management. The ATOS internal audit
staff confirmed that a conflict exists between demand work and
ATOS inspection work. Inspectors based near the offices of the
airlines they oversee are especially prone to this conflict
because they perform both demand work and ATOS inspections. Many
principal inspectors said that these inspectors do not have
sufficient time to complete both their demand work and ATOS
inspection work. Field inspectors, who are assigned only part time
to ATOS, are subject to different pressures. ATOS field inspectors
must also investigate accidents and perform other inspection work
for the local offices to which they are assigned. Principal
inspectors expressed concern that field inspectors might not be
available for ATOS work when needed because of demands from their
local offices. In addition, several principal inspectors and
managers questioned whether the emphasis on ATOS has shifted
resources away from other areas of concern, such as repair
stations, troubled smaller carriers, and general aviation safety.
Managers contended that such conflicts between demand work and
inspection activities will persist as long as ATOS focuses solely
on inspections, rather than having a broader view of the work that
inspectors perform. These conflicting demands on inspectors' time
and on resources such as travel funds are exacerbated because
managers do not know what resources will be needed to perform ATOS
inspections. The ATOS guidance Page 15
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 does
not include estimates of how long various inspection tasks should
take. Without such information, it is difficult to estimate how
many inspectors will be needed to perform the tasks or what travel
funds they may need to accomplish the planned work. Both
inspectors and their managers told us that they are unable to
estimate how much time many of the ATOS inspection tasks will
require. Two principal inspectors told us that managers of some
local offices will not approve ATOS work plans for field
inspectors unless they know how much time will be needed to
complete the work. Testing and validation of the ATOS guidance
would have provided preliminary estimates for the time needed to
complete various ATOS inspection tasks. The ATOS Data Collection
Although ATOS includes initiatives to address past problems with
the Process Limits the usefulness of FAA's
inspection data for analysis and targeting of resources Usefulness
of Data for to the greatest safety risks, this goal may
not be fully realized for several Analysis and Targeting
reasons. First, because of the problems with the ATOS guidance
that we have already discussed, the data collected from ATOS
inspections are not likely to be reliable enough to support
meaningful analyses. Furthermore, in translating ATOS from concept
to design and eventual implementation, FAA did not adequately
determine its data analysis needs. The volume of inspection data
available for analysis has also dropped dramatically under ATOS
because few inspections have been completed. Finally, principal
inspectors responsible for overseeing airlines did not have timely
access even to the limited data available until FAA granted them
access in May 1999. FAA Did Not Adequately While ATOS
calls for structured inspections intended to result in more
Determine Data Analysis Needs thorough and consistent data, the
way ATOS collects data limits the potential of its database as a
tool for analysis and targeting. The data limitations reflect the
fact that FAA did not sufficiently analyze its data needs before
developing ATOS inspection guidance and its automated database.
The development of an effective automated system begins with a
thorough analysis of the data required to meet the needs of those
using the database. In the case of ATOS, a thorough analysis would
describe in detail the questions that need to be asked to improve
aviation safety, determine precisely what data are needed to
answer those questions, and plan the appropriate analyses to be
conducted on those data to answer the questions. After the
preliminary analysis of the users' data needs is completed, the
prototype system must be tested, validated, and revised in an
iterative process between data users and automation developers.
Staff from Sandia National Laboratories reviewed ATOS and
concluded that its Page 16 GAO/RCED-99-183
FAA's New Aviation Inspection System B-281502 developers did not
go beyond abstract, high-level statements about users' needs to
the level of detail essential to ensure data quality. They added
that without more detailed information it will be difficult to
identify the data and information needed to answer questions about
aviation safety and impossible to support the detailed automated
analysis of an airline. Because FAA did not sufficiently analyze
or list the data needed by inspectors to make determinations about
aviation safety and did not adequately test or validate ATOS,
features that would maximize the usefulness of data for analysis
and targeting were not built into the system. We found four
specific limitations with the ATOS database. Key information is
not required. ATOS does not currently require inspectors to record
inspection data that are essential for effective analyses. To
perform effective analyses of safety data, basic information is
needed, such as the airline's name; the make, model, and series of
the aircraft; the aircraft and pilot identification numbers; and
where the inspection was performed. However, an ATOS inspection
record can be closed without any of this basic information having
been entered because the system requires only that inspectors
indicate whether an airline is in compliance and explain any
violations. For example, our review of completed ATOS inspections
found 18 completed inspection activities related to de-icing
operations for which the location had been entered, not in the
appropriate field, but in a comment field. Data contained in
comment fields cannot be used for automated analyses to determine,
for example, which airports are experiencing problems with de-
icing. In addition, the ATOS inspection guidance does not provide
inspectors with an index or clear instructions on where in the
ATOS database to report findings. If an inspector cannot easily
figure out where to report a finding, it may go unreported. For
example, an inspector who found incorrect safety placards on an
aircraft said he had to search ATOS for nearly 4 hours to record
this violation. Inspectors' underreporting of violations that they
observed has jeopardized the completeness and quality of
inspection data in the past. ATOS does not resolve the problems
that contributed to underreporting, and the difficulty in finding
where to report violations in ATOS may exacerbate this problem.
Response options preclude meaningful analyses. Many ATOS
inspection activities are to be summed up in a report with a
single "yes" or "no" response to the items on the inspection
checklist for a given area. If an inspector finds problems during
a single inspection activity, it results in a Page 17
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 "no"
for the entire question. Regardless of whether an inspector finds
one problem in 10 inspections or one in 100, a "no" results.
Without knowing the proportion of "no" responses for a given item,
FAA cannot use the inspection results to conduct quantitative
analyses on ATOS data, for example, a trend analysis to determine
whether the airline's compliance in a given area has improved.
Minimum number of inspection activities needed is not specified.
ATOS currently does not specify the minimum number of activities
needed to complete each inspection. Instead, individual inspectors
decide independently how many times inspection observations should
be performed to determine whether the airline follows its
procedures and complies with regulations. This determination is
subjective. In our review of the ATOS data available as of March
31, 1999, we found that four inspectors conducted anywhere from 2
to 12 observations to complete the record for the same type of
inspection. Because these reports result in "yes" or "no" answers
on the completed inspection report, rather than in a quantifiable
report of the proportion of "no" answers, the data submitted by
these inspectors cannot be compared or analyzed. Because
inspectors will never have enough time to observe every safety-
related component of an airline's system, it is important to
define the minimum number of inspections to be conducted and to
report accurately the proportion of instances of noncompliance.
ATOS does not link to Flight Standards' existing data analysis
system. The information in the ATOS database cannot be analyzed by
Flight Standards' existing aviation risk analysis system, the
Safety Performance Analysis System (SPAS). FAA has spent $95
million developing SPAS to analyze key aviation safety data,
identify trends and potential safety concerns, and target
inspection resources accordingly. In January 1998, the ATOS
development team noted that ATOS inspection data could not be
analyzed by SPAS. The development team recommended that ATOS not
be implemented until FAA could develop appropriate links between
ATOS and SPAS. However, FAA, in its efforts to meet the October 1,
1998, implementation date, went forward without addressing this
recommendation. SPAS program officials told us that significant
work remains to link the two systems. An ongoing FAA work group
studying ways to incorporate system safety into Flight Standards'
programs has been given responsibility for linking ATOS and SPAS.
This work group has not yet established a timetable for how or
when the link will be completed. Page 18
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
Availability of ATOS Data The usefulness of ATOS data for
analysis and targeting has been limited by Limits Analysis
the overall lack of information in the ATOS database as well as by
initial problems with accessing what information is available. FAA
devoted much of the early part of fiscal year 1999 to planning
ATOS inspections and training inspectors in the new system. As a
result, very few data on the performance of the airlines covered
by ATOS have been available for analysis. By the end of the second
quarter, fewer than 60 of approximately 5,400 planned inspections
had been completed, entered into the ATOS database, and made
available to principal inspectors. These completed inspections
comprised fewer than 1,200 individual inspection activities,
substantially fewer than those recorded in previous years.12 The
flow of inspection activity results into the ATOS database has
since accelerated, with an additional 267 inspections and 1,808
individual inspection activities recorded over the first 6 weeks
of the third quarter. Nevertheless, only 326 inspections,
consisting of 3,079 inspection activities, had been completed by
May 11, 1999, and for one airline no inspections had been
completed. In contrast, during the first two quarters of fiscal
year 1998, over 37,000 inspection activities had been recorded in
the Program Reporting and Tracking System for the 10 airlines now
covered by ATOS.13 We also found that access to the information
available in the ATOS database was limited until inspections were
completed. FAA designed ATOS so that neither principal inspectors
nor analysts could access data until inspectors completed all
inspection activities and the team's data evaluation program
manager reviewed and approved the data. The lack of access to key
safety data created problems for principal inspectors, who are
responsible for overseeing operations, maintenance, and avionics
inspections at each airline. They could not view inspection
results until the data evaluation program manager reviewed the
data for clarity and consistency. Several principal inspectors
expressed concern about not having timely access to key safety
data, which they use as an early warning of potential safety
risks. In February 1999, we briefed FAA on the concerns expressed
about delayed access to inspection results in the ATOS database.
In March 1999, FAA directed its automation contractor to take the
steps necessary to grant principal inspectors immediate read-only
access to inspection findings. Until this access was granted,
principal inspectors remained dependent on 12Under ATOS, each
inspection recorded in the database summarizes the results of
multiple observations called inspection activities. 13While the
3,079 ATOS inspection activities may not be directly comparable to
the 37,000 inspections conducted under the previous inspection
system, a substantial drop-off in the inspection information
available to managers and analysts has clearly occurred over the
first half of fiscal year 1999. Page 19
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502 such
informal means as e-mail and telephone calls to learn of any
problems or violations found during inspections. Principal
inspectors now have access to all final inspection activity
reports related to their specialty for the airline that they
oversee. This change made data on 2,724 ATOS inspection activities
available to principal inspectors as of May 11, 1999. ATOS'
Targeting The previously mentioned inadequacies in
the ATOS guidance and problems Capabilities Are Limited by with
the usefulness of the resulting data for analysis limit FAA's
ability to Inadequate Inspection use ATOS for targeting
inspection resources to the problems that pose the Guidance and
Problems greatest safety risks. Prior to the
implementation of ATOS, the primary With the Usefulness of
purpose of inspections was to identify individual safety problems
and ensure their correction. Under ATOS, the primary objective is
to provide Data reliable data to enable
FAA to identify the highest-priority safety concerns and target
the agency's resources to reduce these concerns or risks as well
as to ensure that individual safety problems are corrected. As
implemented, ATOS falls short of this goal. Significant revisions
will be needed to the ATOS guidance and database before the
potential of this new inspection system can be realized. FAA Did
Not Take Although FAA has many ongoing initiatives
with the aviation industry, Advantage of Industry
coordination with industry was lacking in the design of ATOS.
Although Expertise in Developing several aspects of ATOS
involve areas in which industry safety experts have ATOS
experience and similar goals, the design process did not include
airline or industry representatives, who were briefed on ATOS
after the concept was developed. The industry's input is missing,
for instance, in the risk weights ATOS uses in its planning
guidance to help determine the number of inspections a team is to
conduct, thus directing resources to areas that require additional
oversight. For example, the weight ATOS gives to an airline's
screening, boarding, and briefing procedures for passengers is
greater than that assigned to cockpit procedures. FAA neither
validated these weights nor discussed them with airline safety
officials. FAA Has Initiated We briefed FAA in
December 1998 and again in February 1999 on the Improvements to
problems we had found with the implementation of ATOS. Because of
the magnitude and seriousness of the problems associated with the
current ATOS ATOS guidance and database,
we suggested that FAA not expand ATOS to additional airlines,
repair stations, or other aviation operations until these problems
are resolved. Acknowledging that there were significant Page 20
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
challenges to further implementing the program, FAA agreed that
expansion of the program needs to be delayed.14 In March 1999, the
Director of Flight Standards and key ATOS program officials met
with the principal inspectors who lead the 10 ATOS teams to brief
them on our findings and to obtain their views on the system's
implementation. This meeting resulted in a list of immediate and
future actions to be undertaken to address the concerns raised.
First, FAA has taken steps to provide principal inspectors with
immediate access to completed inspection reports. FAA also
provided partial funding for planned overseas inspections, which
will allow inspectors to complete some of the more critical
overseas inspections. Finally, FAA will incorporate additional
information on the inspection guidance task lists and database
tracking system into ATOS training beginning in July. These
initiatives will help alleviate several of the more immediate
problems with ATOS by (1) making key safety information available
to principal inspectors as soon as inspections are completed, (2)
allowing at least some of the planned overseas inspections to take
place, and (3) providing better training for inspectors on how to
conduct inspections and record their results. In addition, both
the ATOS Program Office and the ATOS internal audit group will
continue to monitor the implementation closely to ensure that
these problems and others that may arise are addressed. Funding
has not yet been allocated to support the needed improvements to
ATOS or to link ATOS with FAA's existing data targeting and
analysis system, SPAS. Conclusions The ATOS concept offers
significant promise for helping FAA overcome deficiencies in its
past approach to aviation safety inspections. However, its
potential will not be fully realized until FAA resolves the
problems resulting from the ambitious schedule it followed in
implementing ATOS. These problems limit both FAA's ability to
conduct more systematic, structured inspections and analyze the
resulting data to identify safety trends and its ability to target
its resources to the greatest risks. The ATOS guidance is not
clear and detailed enough to ensure more systematic, structured
inspections that will result in more usable data. In addition, FAA
has not adequately analyzed the data needs of ATOS users to ensure
that the system collects the information that will enable the
agency to perform critical trend and safety analyses. Such
analyses are also limited because ATOS does not link to FAA's
other major database for safety analyses (SPAS). FAA has
recognized the need for significant improvements before ATOS will
14FAA already had plans to bring two additional airlines into ATOS
in the near future. These airlines have recently completed
certification activities and will begin ATOS inspections upon
completion of a transition phase. Page 21
GAO/RCED-99-183 FAA's New Aviation Inspection System B-281502
achieve its full potential as a system for overseeing commercial
airline safety, and the agency has taken actions to correct some
of the problems we found. Because of the challenges involved in
making the needed improvements to ATOS, FAA officials have
postponed plans to expand ATOS to other airlines until these
problems are corrected. In addition, FAA is aware that the
resolution of some problems with the implementation of ATOS
involve broader issues that concern staffing decisions and
workload issues that can affect all of Flight Standards'
inspectors. Recommendations To strengthen the efforts to
improve FAA's aviation safety inspections and the usefulness of
the data that result from these inspections for analysis and for
targeting the agency's resources to the greatest potential safety
threats, we recommend that the Secretary of Transportation direct
the FAA Administrator to take the following actions: * Develop a
structured process and timeline for working with inspectors to
revise the Air Transportation Oversight System's planning and
inspection guidance. The process should involve the inspectors now
using this guidance to (1) identify problems with the clarity of
the guidance, (2) revise the inspection guidance to include tasks
related to all applicable Federal Aviation Regulations, and (3)
define the tasks to be completed during inspections. * Revise the
inspection guidance to (1) include guidelines on the minimum
number of times to perform various inspection tasks and (2)
distinguish between tasks based on regulatory requirements and
those based on handbook or other guidance. * Develop a plan that
involves both inspectors and experts in risk assessment and
database development in revising and refining the analysis of the
data needs of users of the new inspection program. The
requirements analysis should describe in detail the questions that
need to be asked to improve safety, determine precisely what data
are needed to answer those questions, and plan the appropriate
analyses to be conducted on those data to answer the questions. *
Restructure the inspection database to (1) require that essential
data fields be completed before inspection reports can be closed
out and (2) clearly indicate the proportion of inspection
observations in which an airline complies with regulations. Page
22 GAO/RCED-99-183 FAA's New Aviation
Inspection System B-281502 * Determine what revisions will be
needed to the Air Transportation Oversight System database and the
agency's existing Safety Performance Analysis System database to
maximize the potential of these two systems by coordinating their
trend analyses to identify potential safety risks. * Test and
validate the revised guidance and database for the new inspection
program. Agency Comments We provided FAA with a draft of
this report for review and comment. We and Our Evaluation met
with the Deputy Associate Administrator for Regulation and
Certification, the Director of the Office of Flight Standards
Service, the Acting Manager of the Air Transportation Oversight
System Program Office, the Manager of the System Process Audit
Group, and other FAA officials. The agency agreed with the
substance of the report but commented that the tone was
unnecessarily negative and could leave the impression that we
believe that the program should be abandoned. Agency officials
also said that they made a conscious decision to implement the new
system aggressively, rather than in stages, and recognized that
this approach would result in some implementation problems.
However, they believe that ultimately the new system will be fully
operational sooner than if they followed a more conservative
implementation approach. The agency also commented that our review
of the program was premature and suggested that an evaluation of
the program in another year would find that most of the problems
we reported had been resolved. We do not believe the program
should be abandoned. We believe that our report clearly supports
the Air Transportation Oversight System and acknowledges its
potential for significantly strengthening FAA's inspection
process. However, we continue to believe that serious challenges
need to be overcome before this program can achieve its potential.
Because our review coincided with the program's implementation, we
were able to identify serious problems early and to promote
constructive action by FAA to begin resolving them. As we
reported, FAA has begun to address some of these problems. In
commenting on this report, FAA also provided some updated
information on its inspection activities and suggested wording
revisions that we incorporated as appropriate. We conducted our
work from September 1998 through June 1999 in accordance with
generally accepted government auditing standards. Appendix II
contains details of the scope and methodology of our review. Page
23 GAO/RCED-99-183 FAA's New Aviation
Inspection System B-281502 As you requested, unless you publicly
announce its contents earlier, we plan no further distribution of
this report until 10 days from the date of this letter. We will
then send copies to the appropriate congressional committees;
Rodney E. Slater, the Secretary of Transportation; Jane F. Garvey,
the Administrator, FAA; Jacob J. Lew, the Director, Office of
Management and Budget; and other interested parties. We will also
make copies available to others upon request. If you have any
questions about this report or need additional information, please
call me at (202) 512-2834. Major contributors to this report are
listed in appendix III. John H. Anderson, Jr. Director,
Transportation Issues Page 24 GAO/RCED-99-183
FAA's New Aviation Inspection System Page 25 GAO/RCED-99-183
FAA's New Aviation Inspection System Contents Letter
1 Appendix I
28 ATOS Guidance and The System Safety Assessment Tool
30 The Air Carrier Assessment Tool
32 the Comprehensive The Comprehensive Surveillance Plan
33 Surveillance Plan Safety Attribute Inspections
33 Element Performance Inspections
34 Appendix II
35 Objectives, Scope, and Methodology Appendix III
37 GAO Contacts and Staff Acknowledgments Bibliography
38 Related GAO Products
39 Tables Table I.1: The Seven Airline Systems
Defined in ATOS 30 Table I.2:
Categories of System Safety Used With SSAT
31 Table I.3: ACAT Risk Indicators
32 Figure Figure I.1: How FAA Designs
Comprehensive Surveillance Plans 29 Through
ATOS Abbreviations ACAT air carrier assessment tool ATOS
Air Transportation Oversight System EPI element
performance inspection FAA Federal Aviation
Administration GAO General Accounting Office SAI
safety attribute inspection SPAS Safety Performance
Analysis System SSAT system safety analysis tool Page 26
GAO/RCED-99-183 FAA's New Aviation Inspection System Page 27
GAO/RCED-99-183 FAA's New Aviation Inspection System Appendix I
ATOS Guidance and the Comprehensive Surveillance Plan The Air
Transportation Oversight System (ATOS) is a process designed to
improve the Federal Aviation Administration's (FAA) oversight of
airlines. FAA's Office of Flight Standards Service developed ATOS
with the support of Sandia National Laboratories. ATOS uses system
safety principles created for the nuclear industry and risk
management to ensure that airlines have safety built into their
operating systems. A systems safety approach means that FAA's
inspection efforts will cover all aspects of an airline's
performance that can affect safety and will focus on preventing
accidents. A certificate management team oversees each of the 10
airlines under ATOS. Each team is led by three principal
inspectors, one for each major area of inspections (operations,
maintenance, and avionics). Additional team members include those
based at the FAA office that holds the airline's operating
certificate and field inspectors in other FAA offices at locations
to which the airline flies. The team uses automated planning
guidance to develop a comprehensive surveillance plan for the
airline. The planning guidance consists of two automated tools-the
system safety analysis tool (SSAT) and the air carrier assessment
tool (ACAT). The principal inspectors complete the SSAT and ACAT
prior to an annual planning meeting. During the meeting, team
members discuss the SSAT and ACAT, and their feedback is included
in the final version. The results of the ACAT help define
inspection activities that the team will include in the airline's
comprehensive surveillance plan. The SSAT, ACAT, and comprehensive
surveillance plan are described in more detail below. Figure I.1
shows how these ATOS components relate to one another. Page 28
GAO/RCED-99-183 FAA's New Aviation Inspection System Appendix I
ATOS Guidance and the Comprehensive Surveillance Plan Figure I.1:
How FAA Designs Comprehensive Surveillance Plans Through ATOS
Source: Federal Aviation Administration. Page 29
GAO/RCED-99-183 FAA's New Aviation Inspection System Appendix I
ATOS Guidance and the Comprehensive Surveillance Plan Using the
automated planning guidance, the team analyzes the airline's
operations, which are divided into seven systems (see table I.1).
These systems are made up of 14 subsystems and of 79 elements. For
example, the flight operations system is made up of two
subsystems, which encompass elements such as aircraft dispatch and
flight deck procedures. FAA designed the SSAT to help the team
assess the systems in place at an airline to ensure safe
operations. The ACAT applies a set of risk indicators to the
airline's subsystems and elements to generate a comprehensive
surveillance plan. These risk indicators for the ACAT are based on
safety and performance information that reflects areas of
potential risk for an airline's operations. Hence, the
comprehensive surveillance plan will target those areas most
likely to have safety problems. Table I.1: The Seven Airline
Systems Defined in ATOS System
System's purpose Aircraft configuration and control
Maintains the physical condition of the aircraft and associated
components. Manuals
Controls the information and instructions that define and govern
an airline's activities. Flight operations
Governs aircraft movement. Personnel training and qualifications
Ensures that an airline's personnel are trained and qualified.
Route structures Maintains an
airline's facilities on approved routes. Airman/crew member
flight, rest, and duty Prescribes time limitations for
airline time
employees. Technical administration
Addresses all other aspects of an airline's certification and
operations. Source: FAA Order 8400.10, Air Transportation
Operations Inspector's Handbook, appendix 6. The System Safety
The SSAT is a computerized tool designed to focus the inspection
team's Assessment Tool attention on the
systems that an airline has in place. It poses questions to the
team covering six categories: safety attributes, safety culture,
communications, accountability, training programs, and potential
problem areas. (See table I.2.) The principal inspectors complete
the SSAT prior to a yearly meeting to plan inspections to oversee
the airline's operations. To complete the SSAT, the principal
inspectors rely on their knowledge of the airline and on the data
available through FAA's Safety Performance Analysis System (SPAS),
the Flight Standards Automated System, or other sources. Before
the annual meeting, the team members review the SSAT Page 30
GAO/RCED-99-183 FAA's New Aviation Inspection System Appendix I
ATOS Guidance and the Comprehensive Surveillance Plan completed by
the principal inspectors and provide feedback. The SSAT is
finalized at the annual meeting but may be revised during the year
to retarget inspection resources. Table I.2: Categories of System
Safety Used With SSAT Category
Definition Safety attributes ATOS
identifies six safety attributes: Responsibility: The unit or
person in the airline that determines the course of action for a
process. Authority: The unit or person in the airline that has the
authority to establish or modify a process. Procedures: A
documented method of accomplishing a process. Controls: A check or
restraint that is designed into a process to ensure a desired
result. Process measurements: The unit or person in the airline
that measures and assesses information to identify, detect,
analyze, and document problems or potential problems. Interfaces:
Points at which independent processes interact. Safety culture
The priority given to safety by the airline's systems, including
the airline's identification and response to safety risks, and the
effectiveness of internal evaluation systems. Communications
The communication and feedback channels within the airline to
report and respond to safety risks as well as open and timely
communication with FAA and equipment manufacturers. Accountability
The extent to which the airline holds its management and employees
accountable for their assigned responsibility and authority.
Training programs The priority an
airline places on training as well as the effectiveness of initial
and recurrent training programs. Potential problem areas
The existence of concerns based on previous accidents or
incidents, hotline complaints, or trends revealed in safety data.
Source: FAA Order 8400.10, Air Transportation Operations
Inspector's Handbook, appendix 6. Page 31
GAO/RCED-99-183 FAA's New Aviation Inspection System Appendix I
ATOS Guidance and the Comprehensive Surveillance Plan By
completing the SSAT, the ATOS inspection team assesses how well an
airline addresses system safety issues. Using this information,
the team determines whether to inspect any of the systems more or
less frequently than suggested by the ATOS guidance and
incorporates the inspections in the comprehensive surveillance
plan for the airline. The Air Carrier In
completing the ACAT, principal inspectors use the results of the
SSAT, Assessment Tool their knowledge of the
airline their team oversees, and any other available data to
indicate concerns about any real or potential problem that could
contribute to the failure of one of the airline's elements,
subsystems, or systems. The ACAT applies risk indicators to each
of the airline's systems. Table I.3 shows the types of risk
indicators that are assessed when inspectors complete an ACAT.
Table I.3: ACAT Risk Indicators Type of risk indicator
Definition and examples Operational stability
Those aspects of an airline's organization and environment over
which it has no direct control and that, when managed effectively,
could enhance system safety and stability (e.g., turnover in
personnel, or a merger or takeover). Airline dynamics
Aspects of an airline's environment that it directly controls and
that could be used to enhance system safety and stability (e.g.,
an internal evaluation program, and risk management). Performance
history The results of an airline's
operations over time (e.g., enforcement actions, self-disclosure
reports to FAA). Environmental criticality
Those aspects of an airline's surroundings that could lead to or
trigger a failure of one of its systems, subsystems, or elements
and potentially create an unsafe condition (e.g., age of the
fleet, outsourcing of maintenance). Source: FAA Order 8400.10, Air
Transportation Operations Inspector's Handbook, appendix 6. The
principal inspectors complete the ACAT prior to the annual
planning meeting. During the annual planning meeting, the team
members provide feedback on the ACAT that is included in the final
version. Page 32 GAO/RCED-99-183 FAA's
New Aviation Inspection System Appendix I ATOS Guidance and the
Comprehensive Surveillance Plan The Comprehensive The
comprehensive surveillance plan is automatically generated based
on Surveillance Plan the information the team enters into ACAT.
This provides a baseline surveillance plan that is tailored to the
airline, reflecting concerns indicated by the principal
inspectors. Each comprehensive surveillance plan incorporates two
types of inspections, safety attribute inspections (SAI) and
element performance inspections (EPI). SAIs appraise the quality
of an airline's safety attributes (see table I.2) for each system,
its subsystems, and its elements. A team of inspectors conducts
these system inspections. EPIs determine whether an airline
adheres to its written procedures and controls for each system
element and whether the established performance measures for each
element are met. Individual inspectors conduct these inspections.
ATOS allows the principal inspector to increase or, in some cases,
decrease the level of inspection generated by the surveillance
plan. This allows principal inspectors to use their expertise and
personal knowledge of the airline to target resources toward the
greatest safety risks. Although the comprehensive surveillance
plan is automatically generated based on the results of the SSAT
and ACAT, the plan is not finalized until the annual inspection
planning meeting, which is attended by all members of the team.
This allows the principal inspectors to discuss the completed SSAT
and ACAT and to make changes based on other inspectors' feedback.
In addition, work assignments are discussed and made for each of
the SAIs and EPIs that are planned. Principal inspectors complete
and approve the final plan. Safety Attribute The ACAT provides
information directly to the SAI planning system in ATOS,
Inspections which indicates an inspection priority for
each of the airline's subsystems. Considering the SAI priority, a
principal inspector enters the number of SAIs to be completed for
each of the airline's elements during the year. Automation of the
SAI also allows a principal inspector to assign teams for each of
the SAI activities. The principal inspector can also provide
specific instructions to the team regarding the inspections. An
SAI is an in-depth look at an airline's policies and procedures
for a system element. This inspection is structured to look at the
safety attributes shown in table I.2. An SAI is completed by a
team of inspectors, led by a team coordinator. This team assesses
the accuracy and completeness of written policies and procedures
governing each safety attribute associated with one of the airline
system elements. For example, Page 33
GAO/RCED-99-183 FAA's New Aviation Inspection System Appendix I
ATOS Guidance and the Comprehensive Surveillance Plan one element
of the route structure system is the line stations for servicing
and maintaining the airline's aircraft at each city it serves.
Teams performing line station SAIs determine if there are
qualified persons accountable for the line stations and if those
persons have the authority to change the processes governing those
facilities. The teams also review the procedures governing line
stations to determine if proper controls are in place-such as
standards for the maintenance conducted at the line station or for
the training of line station employees. Finally, the teams
determine if processes are in place to identify and correct
problems as well as to ensure that other processes, such as de-
icing and refueling, are coordinated. Element Performance FAA
established a baseline for how frequently each of the 79 ATOS
elements Inspections should be inspected. This
frequency baseline identifies whether a system element should be
inspected on an annual, semiannual, or quarterly basis within the
planning cycle. The ACAT calculates an assessment value that, when
applied to the frequency baseline, increases or decreases the
number of inspections based on the concerns the principal
inspectors have identified. Elements must be inspected at least
once a year. Once the principal inspectors have determined the
number of EPIs that will be conducted, the work is assigned to
other inspectors on the team using the automated system. An EPI
shows whether the airline follows the airline's procedures and
controls. Individual inspectors conduct EPIs, which most resemble
the routine inspections FAA conducted in the past. For example,
the line station EPI requires an inspector to visit a line station
to determine if procedures and controls in place are being
followed at that location. A line station inspection under ATOS
may include multiple visits to one location or to a variety of
other locations as well. All of these visits may be included in a
single EPI report. Page 34 GAO/RCED-99-183
FAA's New Aviation Inspection System Appendix II Objectives,
Scope, and Methodology In September 1998, the Chairman and Ranking
Democratic Member of the Subcommittee on Aviation, House Committee
on Transportation and Infrastructure, asked us to address
questions related to FAA's new Air Transportation Oversight
System: * To what extent does ATOS address past concerns about
FAA's aviation safety inspections? * What factors, if any,
surfaced during the implementation of ATOS that could impede its
success? * What is FAA doing to address any factors that could
impede the success of ATOS? To determine to what extent ATOS
addresses problems identified in the past with FAA's inspection
program, we reviewed previous reports by GAO, the Department of
Transportation's Inspector General, and internal FAA reports, such
as FAA 90 Day Safety Review. In addition, we attended ATOS
training provided to FAA inspectors and the annual inspection
planning meeting held by 1 of FAA's 10 certificate management
teams. We interviewed members of the ATOS work group and program
office to discuss how the new program was developed and the
agency's intentions for its implementation. In total, we
interviewed 68 FAA employees assigned to the airlines ATOS covers,
including 64 of 540 ATOS inspectors and 4 of the 10 unit
supervisors. The 64 inspectors included 28 of the 30 principal
inspectors who oversee ATOS airlines. In the two cases, we
interviewed the assistant principal inspectors because the
principal inspectors were not available. We discussed the ATOS
concept, training, and implementation with each inspector. In
addition to the 68 FAA employees assigned to oversee airlines
under ATOS, we interviewed five Flight Standards district office
managers and supervisors who oversee ATOS field inspectors to gain
a broader perspective on inspector workload issues beyond those
involving ATOS inspections. We also reviewed all 10 comprehensive
surveillance plans developed for the 10 airlines covered by ATOS,
as well as inspection findings reported through May 11, 1999. To
determine what factors, if any, surfaced during the implementation
of ATOS that could impede its success, we interviewed FAA's
principal inspectors for all 10 airlines covered by ATOS and also
inspectors (including field inspectors) assigned to the
certificate management teams. We interviewed staff from Sandia
National Laboratories who served as consultants on the ATOS
project and reviewed Sandia's reports on the Page 35
GAO/RCED-99-183 FAA's New Aviation Inspection System Appendix II
Objectives, Scope, and Methodology system's implementation. At FAA
headquarters, we interviewed data management and system officials
as well as staff assigned to the System Process Audit Group. We
also reviewed reports prepared by these FAA units on the
implementation of ATOS. In addition, we discussed ATOS specifics
with key safety officials at the 10 airlines covered by the
system. To determine what FAA is doing to address any factors that
could impede the success of ATOS, we reviewed internal reports on
ATOS and associated recommendations. We discussed these
recommendations and what impact FAA's budget shortfall will have
on ATOS inspection efforts with principal inspectors on the ATOS
teams and with Flight Standards headquarters officials. We
conducted our work from September 1998 through June 1999 in
accordance with generally accepted government auditing standards.
Page 36 GAO/RCED-99-183 FAA's New
Aviation Inspection System Appendix III GAO Contacts and Staff
Acknowledgments GAO Contacts John H. Anderson, Jr., (202)
512-2834 Robert White, (202) 512-5463 Acknowledgments In
addition to those named above, Leslie Albin, Bonnie A. Beckett,
David K. Hooper, Christopher M. Jones, Fran Featherston, Debra
Prescott, and Stan Stenerson made key contributions to this
report. Page 37 GAO/RCED-99-183 FAA's New
Aviation Inspection System Bibliography Booz-Allen & Hamilton Inc.
Challenge 2000: Recommendations for Future Aviation Safety
Regulation. Prepared for the Federal Aviation Administration. Apr.
1996. Federal Aviation Administration. FAA 90 Day Safety Review.
Sept. 16, 1996. Sandia National Laboratories. "Analysis and Data
Issues for the Implementation of a Systems Safety Focus in Air
Transportation Oversight." Oct. 6, 1998. U.S. Department of
Transportation, Office of Inspector General. Audit of Aviation
Inspection Program: Federal Aviation Administration (R6-FA-2-084).
Washington, D.C.: May 29, 1992. U.S. Department of Transportation,
Office of Inspector General. Final Report on Audit of Flight
Standards Inspection Program in Southwest Region (R6-FA-9-078).
Washington, D.C.: Mar. 17, 1989. U.S. Department of
Transportation, Office of Inspector General. Pilot Examiner
Program: Federal Aviation Administration (R2-FA-7-001).
Washington, D.C.: Oct. 22, 1996. U.S. Department of
Transportation, Office of Inspector General. Report on Audit of
the Certification and Surveillance of Domestic and Foreign Repair
Stations: Federal Aviation Administration (R4-FA-4-009).
Washington, D.C.: Mar. 7, 1994. U.S. Department of Transportation,
Office of Inspector General. Statement of Raymond J. DeCarli,
Associate Deputy Inspector General, U.S. Department of
Transportation. Washington, D.C.: May 21, 1997. White House
Commission on Aviation Safety and Security. Final Report of the
White House Commission on Aviation Safety and Security. Feb. 12,
1997. Page 38 GAO/RCED-99-183 FAA's New
Aviation Inspection System Related GAO Products Aviation Safety:
Weaknesses in Inspection and Enforcement Limit FAA in Identifying
and Responding to Risks (GAO/RCED-98-6, Feb. 27, 1998). Aviation
Safety: FAA Oversight of Repair Stations Needs Improvement
(GAO/RCED-98-21, Oct. 24, 1997). Aviation Safety and Security:
Challenges to Implementing the Recommendations of the White House
Commission on Aviation Safety and Security (GAO/T-RCED-97-90, Mar.
5, 1997). Aviation Safety: New Airlines Illustrate Long-Standing
Problems in FAA's Inspection Program (GAO/RCED-97-2, Oct. 17,
1996). Aviation Safety: Targeting and Training of FAA's Safety
Inspector Workforce (GAO/T-RCED-96-26, Apr. 30, 1996). Aviation
Safety: Data Problems Threaten FAA Strides on Safety Analysis
System (GAO/AIMD-95-27, Feb. 8, 1995). FAA Technical Training
(GAO/RCED-94-296R, Sept. 26, 1994). Aviation Safety: FAA and the
State Department Can Better Manage Foreign Enforcement Cases
(GAO/RCED-94-87, Mar. 17, 1994). Aviation Safety: Progress on FAA
Safety Indicators Program Slow and Challenges Remain (GAO/IMTEC-
92-57, Aug. 31, 1992). Aviation Safety: FAA Needs to More
Aggressively Manage Its Inspection Program (GAO/T-RCED-92-25, Feb.
6, 1992). Aviation Safety: Problems Persist in FAA's Inspection
Program (GAO/RCED-92-14, Nov. 20, 1991). Aviation Safety:
Emergency Revocation Orders of Air Carrier Certificates (GAO/RCED-
92-10, Oct. 17, 1991). Aviation Safety: FAA's Safety Inspection
Management System Lacks Adequate Oversight (GAO/RCED-90-36, Nov.
13, 1989). Aviation Training: FAA Aviation Safety Inspectors Are
Not Receiving Needed Training (GAO/RCED-89-168, Sept. 14, 1989).
Page 39 GAO/RCED-99-183 FAA's New Aviation
Inspection System Related GAO Products Aviation Safety: Needed
Improvements in FAA's Airline Inspection Program Are Underway
(GAO/RCED-87-62, May 19, 1987). (348129) Page 40
GAO/RCED-99-183 FAA's New Aviation Inspection System Ordering
Information The first copy of each GAO report and testimony is
free. Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made out
to the Superintendent of Documents, when necessary. VISA and
MasterCard credit cards are accepted, also. Orders for 100 or more
copies to be mailed to a single address are discounted 25 percent.
Orders by mail: U.S. General Accounting Office P.O. Box 37050
Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner
of 4th and G Sts. NW) U.S. General Accounting Office Washington,
DC Orders may also be placed by calling (202) 512-6000 or by using
fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO
issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the
past 30 days, please call (202) 512-6000 using a touchtone phone.
A recorded menu will provide information on how to obtain these
lists. For information on how to access GAO reports on the
INTERNET, send an e-mail message with "info" in the body to:
[email protected] or visit GAO's World Wide Web Home Page at:
http://www.gao.gov PRINTED ON RECYCLED PAPER United States
General Accounting Office Bulk Rate Washington, D.C.
20548-0001 Postage & Fees Paid GAO Permit No. G100 Official
Business Penalty for Private Use $300 Address Correction Requested
*** End of document. ***