Aviation Security: Measures for Testing the Impact of Using
Commercial Data for the Secure Flight Program (23-FEB-05,
GAO-05-324).
The Transportation Security Administration (TSA) is developing a
new passenger prescreening program, known as Secure Flight. Under
the Secure Flight program, TSA plans to take over, from
commercial airlines, the responsibility for comparing identifying
information of domestic airline passengers against information on
known or suspected terrorists. TSA is also considering using
commercial data as part of Secure Flight if the data are shown,
through testing, to improve the results of these comparisons. In
the 2005 Homeland Security Appropriations Act, Congress mandated
that, prior to testing the use of commercial data for Secure
Flight, TSA develop measures to assess the impacts of using
commercial data on aviation security, and that GAO review the
measures. In response to that mandate, we reviewed TSA's measures
for commercial data testing and briefed congressional staff on
January 11, 2005, on our findings. This report documents the
results of our review, which we presented in that briefing.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-324
ACCNO: A18145
TITLE: Aviation Security: Measures for Testing the Impact of
Using Commercial Data for the Secure Flight Program
DATE: 02/23/2005
SUBJECT: Airport security
Commercial aviation
National preparedness
Counterterrorism
Data collection
Emergency preparedness
Operational testing
Performance measures
Program evaluation
Terrorism
Aviation security
Homeland security
TSA Secure Flight Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-324
United States Government Accountability Office
GAO
Report to Congressional Committees
February 2005
AVIATION SECURITY
Measures for Testing the Impact of Using Commercial Data for the Secure Flight
Program
GAO-05-324
Contents
Letter 1
Appendix I Briefing Slides
Appendix II Comments from the Department of Homeland Security
Abbreviations
DHS Department of Homeland Security GAO Government Accountability Office
TSA Transportation Security Administration
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office Washington, DC 20548
February 23, 2005
Congressional Committees:
The Transportation Security Administration (TSA) is developing a new
passenger prescreening program, known as Secure Flight. Under the Secure
Flight program, TSA plans to take over, from commercial airlines, the
responsibility for comparing identifying information of domestic airline
passengers against information on known or suspected terrorists. TSA is
also considering using commercial data as part of Secure Flight if the
data are shown, through testing, to improve the results of these
comparisons.1 In the 2005 Homeland Security Appropriations Act (Public Law
108-334, Section 522(d)), Congress mandated that, prior to testing the use
of commercial data for Secure Flight, TSA develop measures to assess the
impacts of using commercial data on aviation security, and that GAO review
the measures. In response to that mandate, we reviewed TSA's measures for
commercial data testing and briefed congressional staff on January 11,
2005, on our findings. This report documents the results of our review,
which we presented in that briefing.
Currently, commercial airlines are responsible for the prescreening of
passengers using terrorist watch lists provided by TSA, known as the nofly
and selectee lists.2 However, as noted by the National Commission on
Terrorist Attacks Upon the United States (9/11 Commission), the watch
lists used by the airlines do not include all terrorists or terrorism
suspects because of concerns about sharing intelligence information with
private firms and foreign countries.3 TSA expects that Secure Flight will
improve passenger prescreening as compared with the current
airline-operated process. For example, Secure Flight will utilize an
expanded terrorist watch list that includes information not currently
provided to air carriers
1Commercial data are maintained by private companies and can include
personally identifiable information that either identifies an individual
or is directly attributed to an individual, such as name, address, and
phone number.
2To conduct passenger prescreening, airlines also compare passenger data
against the Computer-Assisted Passenger Prescreening System (CAPPS I)
rules, which are behavioral characteristics associated with the way an
airline ticket is purchased. The CAPPS I rules are intended to identify
individuals who should receive additional security scrutiny.
3The 9/11 Commission Report: Final Report of the National Commission on
Terrorist Attacks Upon the United States (Washington, D.C.: July 2004).
for passenger prescreening. TSA also expects that by automating the
prescreening process and applying consistent procedures for comparing
passenger data against the expanded terrorist watch list, Secure Flight
will reduce the number of false positive matches against the terrorist
watch list as compared with the current process.
In preparing to take over passenger prescreening from domestic air
carriers, TSA has begun initial Secure Flight testing to determine the
ability of Secure Flight to effectively compare passenger-provided
information contained in air carrier reservation systems against the
expanded watch list in order to identify individuals known or reasonably
suspected to be engaged in terrorism.4 TSA expects that results from these
tests will be available in February 2005. In addition, TSA plans to
conduct a concept test to determine if the use of commercial data can
improve the matching of passenger-provided information against the
expanded watch list by identifying individuals who were incorrectly
identified as being on a terrorist watch list (referred to as false
positives) or who attempted to avoid detection by disguising their
identity (referred to as false negatives).5 The commercial data concept
test is also intended to determine if the accuracy of passenger-provided
data contained in passenger records can be verified using commercial
data.6 In January 2005, TSA issued a request for proposals in order to
obtain a contractor to conduct commercial data concept testing. TSA
expects to award the contract in late February 2005.
To determine the effectiveness of using commercial data, TSA developed
initial measures for commercial data concept testing, such as the overall
percentage of passenger-provided records from which identity can be
verified using commercial data, and plans to refine the measures
throughout the testing process. TSA expects to obtain the results of
commercial data concept testing in April 2005. On the basis of these test
results, the Department of Homeland Security (DHS) and TSA plan to
4These reservation systems contain detailed information about an
individual's travel on a particular flight, including information provided
by the passenger when making a flight reservation. Such information can
include (1) passenger name, (2) reservation date, (3) travel agency or
agent, (4) travel itinerary information, (5) form of payment, (6) flight
number, and (7) seating location.
5The purpose of the concept test is limited to identifying the utility of
using commercial data in improving the effectiveness of comparing
passenger information against the terrorist watch list in a test
environment.
6To obtain data for Secure Flight testing, TSA issued an order in November
2004 requiring domestic airlines to provide passenger records for the
month of June 2004.
make policy decisions regarding the use of commercial data as part of the
overall Secure Flight program. TSA also plans to subsequently test
additional functionality and the operations of Secure Flight before
implementation, regardless of whether TSA incorporates the use of
commercial data as part of Secure Flight.
To determine if the measures developed by TSA for commercial data testing
are designed to identify impacts on aviation security, we reviewed and
analyzed TSA's draft statement of work for commercial data concept
testing, which includes the initial measures developed by TSA. Since the
purpose of our review was to determine whether the measures identify
impacts on aviation security, we assessed the measures against performance
measurement criteria developed by GAO based on best practices.7 On the
basis of our knowledge of the Secure Flight program and GAO performance
measurement criteria, we determined whether TSA's measures are designed to
reflect relevant impacts on aviation security and are consistent with
attributes of successful performance measures. We also interviewed TSA
officials responsible for Secure Flight development and oversight. The
briefing slides, contained in appendix I, include the specific attributes
that we used as criteria for evaluating TSA's measures, detailed
information on our scope and methodology, and the results of our review of
TSA's measures for commercial data testing. Appendix I also includes a
list of TSA's initial measures for commercial data testing. We conducted
our work in accordance with generally accepted government auditing
standards from December 2004 to February 2005. GAO is also continuing to
review TSA's measures for commercial data testing based on a follow-on
congressional request.8
In January 2005, we briefed your offices on the results of our review of
TSA's measures for commercial data concept testing. In summary, we made
the following key points in our briefing:
o TSA developed a concept test to determine the utility of using
commercial data for Secure Flight as a first step in determining its
impact on aviation security. The results of this test are intended to
provide TSA the basis for
7Performance measurement is used to provide information on the achievement
of program accomplishments, particularly progress toward meeting
preestablished goals or targets, and the impacts of those accomplishments.
8TSA's final statement of work for commercial data testing, issued
subsequent to our briefing, includes a revised set of measures for the use
of commercial data. We will assess these revised measures as part of our
follow-on review of TSA's commercial data test.
refining performance measures identifying impacts on aviation security
prior to subsequent testing, should DHS and TSA decide to pursue the use
of commercial data.
o TSA developed initial measures for commercial data concept testing
that are intended to provide information related to impacts on aviation
security, including improvements in false positive and false negative
rates. TSA, in coordination with the contractor, plans to refine these
measures during concept testing-to include the establishment of
performance targets-and prior to operationally testing the system, should
DHS and TSA decide to pursue the use of commercial data.
o TSA measures developed to date for commercial data testing do not, and
were not designed to, provide information on overall Secure Flight system
operations (i.e., system response time, connectivity with air carriers,
security, and privacy) or identify impacts of using commercial data on
aviation security in an operational environment. Accordingly, the measures
do not generally reflect attributes of successful performance measures for
this purpose.
o Additional work reviewing TSA's refined measures, should DHS and TSA
decide to pursue the use of commercial data for Secure Flight, would be
needed to determine if the measures are designed to identify relevant
impacts on aviation security, and reflect attributes of successful
performance measures for that purpose.
We provided a draft of this report to DHS for its review and comment. In
commenting on the draft report, DHS generally agreed with our findings.
DHS's written comments are presented in appendix II. TSA also provided
technical comments which we have incorporated into this report where
appropriate.
We are sending copies of this report to the Secretary of the Department of
Homeland Security and the Administrator of the Transportation Security
Administration. We will also make copies available to others upon request.
In addition, the report will be available at no charge on GAO's Web site
at http://www.gao.gov.
If you or your staff have any questions about this report, please contact
me at (202) 512-3404 ([email protected]) or Christine Fossett, Assistant
Director, at (202) 512-2956 ([email protected]). Other key contributors to
this report were R. Denton Herring, Adam Hoffman, David Hooper, Tom
Lombardi, and David Plocher.
Sincerely yours,
Cathleen A. Berrick Director, Homeland Security and Justice Issues
List of Congressional Committees
The Honorable Thad Cochran
Chairman
The Honorable Robert C. Byrd
Ranking Minority Member
Committee on Appropriations
United States Senate
The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Committee on Commerce, Science, and Transportation
United States Senate
The Honorable Jerry Lewis
Chairman
The Honorable David R. Obey
Ranking Minority Member
Committee on Appropriations
House of Representatives
The Honorable Don Young
Chairman
The Honorable James L. Oberstar
Ranking Minority Member
Committee on Transportation and Infrastructure
House of Representatives
The Honorable Tom Davis
Chairman
Committee on Government Reform
House of Representatives
The Honorable Adam H. Putnam
House of Representatives
Briefing Outline
o Background
o Objective, Scope, and Methodology
o Summary
o TSA Secure Flight Testing Approach
o Concept Testing for Use of Commercial Data
o Attributes of Successful Performance Measures
o TSA Measures for Commercial Data Testing
o List of TSA Measures for Commercial Data Testing from Draft Statement
of Work
2
Background
The Transportation Security Administration (TSA) is developing a new
passenger prescreening program, known as Secure Flight. Under this
program:
o TSA will assume responsibility for checking airline passengers' names
against no-fly and selectee lists and the Computer-Assisted Passenger
Prescreening System (CAPPS I) rules, a task that is currently performed by
air carriers.1 TSA will also check passenger names against an expanded
terrorist watch list.
o TSA is also considering the use of commercial data (e.g., personally
identifiable information that either identifies an individual or is
directly attributed to an individual, such as name, address, and phone
number) if it is shown, through testing, to be effective in mitigating
false positives, identifying false negatives, or verifying passenger
identification.2
1 CAPPS I rules are behavioral characteristics used to select passengers
who require additional security scrutinyat airport security checkpoints.
2 A false positive is an individual who was misidentified as a positive
match when matching passenger data against a terrorist watch list. A false
negative is an individual on a terrorist watch list who avoids detection
when passenger data are matched against a terrorist watch list by
disguising his or her true identity.
3
Background (cont'd.)
TSA has two testing efforts for the Secure Flight program:
o watch list/CAPPS I testing-a test to match historical passenger data
against an expanded government watch list and CAPPS I rules.
o commercial data concept testing-a test of a methodology to determine if
the use of commercial data can improve on the results of watch list/CAPPS
I testing.
At the conclusion of both tests, TSA plans to conduct additional Secure
Flight system testing.
4
Objective, Scope, and Methodology (cont'd.)