Homeland Security: Guidance and Standards Are Needed for	 
Measuring the Effectiveness of Agencies' Facility Protection	 
Efforts (31-MAY-06, GAO-06-612).				 
                                                                 
The need to better protect federal facilities, coupled with	 
federal budget constraints and the increased scrutiny of homeland
security funding and programs, has prompted the need for U.S.	 
agencies to measure the performance of their facility protection 
efforts. In this environment, it is important for these agencies 
to ensure that investments in facility protection are providing  
adequate returns in terms of better protecting real property	 
assets against terrorism. In addition, the U.S. government's	 
national strategy, Presidential directive, and guidance on	 
protecting critical infrastructures--including facilities--have  
identified the use of performance measurement as a key means of  
assessing the effectiveness of protection programs. Given that	 
protection of critical infrastructures is an important issue for 
organizations outside of the federal government as well, it is	 
beneficial to look to the experiences of these organizations to  
identify lessons learned. As such, our objectives for this review
were (1) to identify examples of performance measures for	 
facility protection being used by selected organizations outside 
of the federal government--including private-sector entities,	 
state and local governments, and foreign governments, and (2) to 
determine the status of U.S. federal agencies' efforts to develop
and use performance measures as part of their facility protection
programs.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-06-612 					        
    ACCNO:   A54905						        
  TITLE:     Homeland Security: Guidance and Standards Are Needed for 
Measuring the Effectiveness of Agencies' Facility Protection	 
Efforts 							 
     DATE:   05/31/2006 
  SUBJECT:   Antiterrorism					 
	     Facility management				 
	     Facility security					 
	     Federal facilities 				 
	     Federal property					 
	     Homeland security					 
	     Human capital management				 
	     Interagency relations				 
	     Lessons learned					 
	     Performance measures				 
	     Physical security					 
	     Real property					 
	     Risk management					 
	     Strategic planning 				 
	     National Infrastructure Protection Plan		 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-06-612

     

     * Results in Brief
     * Background
     * Organizations outside of the U.S. Government Use Security Pe
          * Selected Organizations Use a Range of Output, Outcome, and P
               * Case Example: A Financial Service Organization's Performance
          * Security Officials Use Performance Measure Results for Risk
               * Case Example: An Australian Agency's Risk Model
          * Performance Measures Can Be Used to Hold Security Officials
               * Case Example: The District of Columbia's Alignment of Securi
               * Departmental Security Officers and Individual Accountability
               * Case Example: Individual Accountability in Two Financial Ser
               * Case Example: An Australian Agency's Security Certification
          * Organizations Cited Challenges in Developing and Using Perfo
     * U.S. Agencies Have Made Some Progress in Developing and Usin
          * Agencies Use Output Measures and Some Outcome Measures to In
               * Case Example: DHS's Federal Protective Service in GSA Facili
               * Case Example: Interior's Bureau of Reclamation and National
               * Case Example: USPS Inspection Service
               * Case Example: Department of Veterans Affairs
          * Federal Guidance for Developing and Using Performance Measur
               * Federal Agencies Have Received Minimal Guidance on Using Per
     * Conclusions
     * Recommendations for Executive Action
     * Agency Comments and Our Evaluation
     * GAO Comments
     * GAO Contact
     * Staff Acknowledgments
     * GAO's Mission
     * Obtaining Copies of GAO Reports and Testimony
          * Order by Mail or Phone
     * To Report Fraud, Waste, and Abuse in Federal Programs
     * Congressional Relations
     * Public Affairs

Report to the Chairman, Committee on Government Reform, House of
Representatives

United States Government Accountability Office

GAO

May 2006

HOMELAND SECURITY

Guidance and Standards Are Needed for Measuring the Effectiveness of
Agencies' Facility Protection Efforts

GAO-06-612

Contents

Letter 1

Results in Brief 4
Background 6
Organizations outside of the U.S. Government Use Security Performance
Measures to Enhance Decision Making and Help Ensure Accountability 12
U.S. Agencies Have Made Some Progress in Developing and Using Performance
Measures for Facility Protection Programs, but Lack Guidance and Standards
25
Conclusions 48
Recommendations for Executive Action 49
Agency Comments and Our Evaluation 49
Appendix I Objectives, Scope, and Methodology 52
Appendix II Examples of Performance Measures Used by Selected
Organizations outside of the Federal Government 58
Appendix III Comments from the Department of Homeland Security 62
Appendix IV Comments from the Department of the Interior 64
GAO Comments 67
Appendix V GAO Contact and Staff Acknowledgments 68

Tables

Table 1: Examples of Performance Measures for Facility Protection 13
Table 2: FPS's Performance Measures for Facility Protection 26
Table 3: BOR's Performance Measures for Facility Protection 32
Table 4: Inspection Service's Performance Measure for Facility Protection
34
Table 5: Types of Information Technology Security Performance Measures
Described by NIST 41
Table 6: U.S. State and Local Governments Contacted 54
Table 7: Foreign Government Agencies and Organizations Visited 55

Figures

Figure 1: Smart Card Access Portals at a Federal Building Entrance 9
Figure 2: Linkages between District of Columbia Strategic Goals and
Performance Measures for Facility Protection 20
Figure 3: Linkages between DHS Mission and FPS Performance Measures for
Facility Protection 29
Figure 4: Linkages between USPS Inspection Service Strategic Goals and
Performance Measure for Facility Protection 37
Figure 5: Sample Standardized Performance Measurement Data Form 44
Abbreviations

BOR Bureau of Reclamation

DHS Department of Homeland Security

DSO departmental security officer

FPS Federal Protective Service

GPRA Government Performance and Results Act of 1993

GSA General Services Administration

HSPD-7 Homeland Security Presidential Directive Number 7

ICE Immigration and Customs Enforcement

ISC Interagency Security Committee

IT information technology

NIPP National Infrastructure Protection Plan

NIST National Institute of Standards and Technology

NM&I National Monuments and Icons Assessment Methodology

OLES Office of Law Enforcement and Security

OMB Office of Management and Budget

PART Program Assessment Rating Tool

USPS United States Postal Service

VA Department of Veterans Affairs

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

May 31, 2006

The Honorable Tom Davis Chairman Committee on Government Reform House of
Representatives

Dear Mr. Chairman:

The threat of terrorism has increased the emphasis on physical security
for federal real property assets since the 1995 bombing of the Alfred P.
Murrah Federal Building in Oklahoma City; the 1998 embassy bombings in
Africa; the September 11, 2001, attacks on the World Trade Center and the
Pentagon; and the anthrax attacks in the fall of 2001. The federal
government owns or leases an estimated 3.2 billion square feet of space
within the United States in more than 450,000 buildings, which are
regularly accessed by millions of federal employees, contractors, and
citizens. Approximately 42 percent of this square footage is nonmilitary
property, and a majority of this is under the control or custody of the
General Services Administration (GSA), the United States Postal Service
(USPS), the Department of Veterans Affairs (VA), and the Department of the
Interior (Interior).1 Under the Homeland Security Act of 2002, the Federal
Protective Service (FPS), which protects GSA properties, was transferred
to the Department of Homeland Security (DHS). For agencies that aim to
ensure public access to their assets, protecting nonmilitary real property
assets can be complex and contentious because of the need to strike a
balance between public access and security.2 Federal agencies face
additional security-related challenges, such as securing federally leased
space and addressing conflicts with state, local, or private entities that
also have jurisdiction over, or input regarding, physical security
enhancements. The challenge of protecting federal facilities against the
threat of terrorism was a major reason GAO designated federal real
property as a high-risk area in January 2003.3

1GSA, Overview of the United States Government's Owned and Leased Real
Property: Federal Real Property Profile As of September 30, 2004
(Washington, D.C.). This property includes government-owned and leased
space.

2GAO, Homeland Security: Actions Needed to Better Protect National Icons
and Federal Office Buildings from Terrorism, GAO-05-790 (Washington, D.C.:
June 24, 2005), p. 1.

Although FPS is primarily responsible for protecting GSA properties, it
also has responsibility for broader efforts across the federal government
to enhance the protection of critical facilities and works closely with
the Interagency Security Committee (ISC) on these issues. The ISC, which
DHS chairs, is tasked with coordinating federal agencies' facility
protection efforts, developing protection standards, and overseeing
implementation of those standards.4 In November 2004, we recommended that
ISC develop an action plan for fulfilling its responsibilities and
establish a set of key practices for facility protection.5 We identified
several key practices in facility protection, which included using risk
management to allocate resources;6 leveraging security technology;
coordinating protection efforts and sharing information; realigning real
property assets to an agency's mission, thereby reducing vulnerabilities;
strategically managing human capital; and measuring program performance
and testing security initiatives.7 With regard to measuring performance,
performance measures can be classified as output measures, which focus on
the quantity of direct products and services a program delivers; outcome
measures, which offer information on the results of the direct products
and services a program has delivered; or process/input measures, which
address the type or level of program activity an organization conducts and
the resources used by the program. Outcome measures are particularly
useful because they indicate what program activities are accomplishing. At
the time of our November 2004 report, agencies were only in the early
stages of implementing security performance measures.

3GAO, High-Risk Series: Federal Real Property, GAO-03-122 (Washington,
D.C.: January 2003).

4In this report, facility protection denotes the protection of not only
the facilities but also the people, equipment, and other assets within
them. Additionally, this report focuses largely on protecting facilities
from threats and acts of terrorism. However, it is important to note that
facilities are also vulnerable to other types of hazards, such as natural
disasters and workplace violence, and information in this report may be
applicable to those hazards as well.

5GAO, Homeland Security: Further Actions Needed to Coordinate Federal
Agencies' Facility Protection Efforts and Promote Key Practices, GAO-05-49
(Washington, D.C.: Nov. 30, 2004). Since the time of that report, the ISC
Chair noted that he is in the process of creating and establishing an
action plan with the ISC membership, although little progress has been
made because of limited resources. The Chair anticipates that this action
plan, which will articulate a roadmap for the ISC to follow in meeting its
responsibilities, will incorporate portions of the material and related
concepts contained in GAO reports.

6Risk management is a tool for assessing risks, evaluating alternatives,
making decisions, and implementing and monitoring protective measures.
More specifically, risk can be calculated as follows: risk = (threat x
vulnerability) x consequence. Threat is the probability that a specific
type of attack will be initiated against a particular target or class of
targets. The vulnerability of an asset is the probability that a
particular attempted attack will succeed against a particular target or
class of targets. It is usually measured against some set of standards,
such as availability/predictability, accessibility, countermeasures in
place, and target hardness (the material construction characteristics of
the asset). The consequence of a terrorist attack is characterized as the
expected worst case or worst reasonable adverse impact of a successful
attack.

The need to better protect federal facilities, coupled with federal budget
constraints and the increased scrutiny of homeland security funding and
programs, has prompted the need for U.S. agencies to measure the
performance of their facility protection efforts. In this environment, it
is important for these agencies to ensure that investments in facility
protection are providing adequate returns in terms of better protecting
real property assets against terrorism. In addition, the U.S. government's
national strategy, Presidential directive, and guidance on protecting
critical infrastructures-including facilities-have identified the use of
performance measurement as a key means of assessing the effectiveness of
protection programs. Given that protection of critical infrastructures is
an important issue for organizations outside of the federal government as
well, it is beneficial to look to the experiences of these organizations
to identify lessons learned. As such, our objectives for this review were
(1) to identify examples of performance measures for facility protection
being used by selected organizations outside of the federal
government-including private-sector entities, state and local governments,
and foreign governments, and (2) to determine the status of U.S. federal
agencies' efforts to develop and use performance measures as part of their
facility protection programs. To address the first objective, we
interviewed private-sector representatives from four entities in the
gaming industry and from five major financial services entities, because
these industries were identified as having invested in security and likely
to have developed performance measures. We also interviewed officials from
17 of the 20 state and local governments that received the most funding
from two security-related DHS grant programs in fiscal year 2005.8
Finally, we interviewed government officials from multiple agencies in
Australia, Canada, and the United Kingdom, because these countries have
experience with threats of terrorism and have performance measurement
initiatives. We also reviewed relevant documents we obtained from these
organizations, related GAO reports, and literature on performance
measurement. To address the second objective, we interviewed federal
officials from DHS, GSA, USPS, VA, and Interior-the agencies that hold, or
are responsible for the security of, the majority of the domestic,
nonmilitary property. We also reviewed pertinent documents and policies
obtained from these agencies, in addition to related laws and directives.
A detailed discussion of our scope and methodology, including more
information on how we selected the organizations we contacted, is
contained in appendix I. We conducted our work between June 2005 and April
2006 in accordance with generally accepted government auditing standards.

7Performance measurement is the ongoing monitoring and reporting of
program accomplishments, particularly progress toward preestablished
goals. It is typically conducted by program or agency management.

                                Results in Brief

We found a range of examples of performance measures that organizations
outside the U.S. government, including private-sector firms, state and
local governments, and foreign government agencies, use to help improve
the security of facilities, inform risk-management and resource-allocation
decisions, and hold security officials and others in their organizations
accountable for security performance. These included output measures, such
as the average time to process background screenings, and outcome
measures, such as the change in the total number of security incidents
relating to thefts, vandalism, and acts of terrorism. For example, an
agency in Australia monitors an outcome measure concerning the impact of
additional security expenditures on a facility's risk rating, while
controlling for existing security enhancements that mitigate the risk,
such as the number of guard patrols and the adequacy of access control
systems (e.g., electronic locks). In another example, each business line
in one financial services organization conducts security compliance
reviews of its facilities, including confirming the presence of required
key security equipment and determining whether staff are following
security policies. Senior security officials review the results to
determine where problems exist and hold each business manager accountable
for addressing them. Despite some organizations' use of these measures,
less than one-quarter of the organizations we contacted had developed
performance measures for facility protection, and there was widespread
acknowledgement among the organizations that effectiveness in facility
protection is challenging to measure. For example, security officials do
not necessarily know whether a potential security threat or incident has
been prevented, even after perceived security weaknesses have been
addressed. Since security is so challenging to measure, some of the
organizations that we interviewed told us that they rely on U.S. federal
agencies for support and leadership in developing security standards and
performance measures, and one foreign government agency said it was
interested in developing guidance for security performance measurement but
was looking to U.S. federal agencies for assistance in this area.

8Of the 20 state and local governments we attempted to contact, we were
able to obtain information from officials from 17 of them.

We found that some bureaus and services within three of the agencies we
reviewed-DHS (for GSA properties), USPS, and Interior-are using output
measures, and, to a lesser extent, outcome measures, while VA and some
bureaus and services within the other three agencies are not. The agencies
that have developed performance measures use them to evaluate and improve
program effectiveness, make risk management decisions, and help ensure
adequate protection at individual facilities. For example, within DHS, FPS
has established an output-oriented performance measure to monitor the
timely deployment of security enhancements such as x-ray machines. Such a
measure provides a basis for FPS to compare planned versus actual
performance. Several bureaus and services within USPS and Interior have
developed methodologies to rank and monitor the relative risk ratings of
their respective facilities over time-these ratings are then used as
outcome measures for determining the change in the effectiveness of
facility protection efforts. VA and the bureaus and services that did not
have security performance measures generate data on ongoing protection
activities, such as monitoring the numbers and types of security breaches
at a given facility. This information could provide useful feedback about
the agency's effectiveness in mitigating building security risks and
therefore could be used for measuring performance. Although agencies have
placed an emphasis on performance measurement and initiatives are under
way, agency security officials said it has been challenging to measure the
actual impact of various approaches on improving security and that
resources for measurement initiatives have been scarce. Furthermore, while
importance has been placed on performance measures in national homeland
security policies and broad guidance exists for measuring the performance
of critical infrastructure protection programs, agencies have not
established specific guidance and standards for developing and using
performance measures for facility protection programs in particular. This
differs from the information technology security area, where agencies not
only are required to measure performance, but also have detailed guidance
and standards for developing and implementing performance measures.
Without effective performance measurement data, especially data on program
outcomes, decision makers may have insufficient information to evaluate
whether the benefits of security investments justify their costs, to
determine the effectiveness of security activities, to know the extent to
which security enhancements have improved security or reduced federal
facilities' vulnerability to acts of terrorism or other forms of violence,
or to determine funding priorities within and across agencies.

Because ISC was established to enhance the quality and effectiveness of
security in buildings and facilities in the United States and to provide a
permanent body to address continuing governmentwide security in federal
facilities, we are recommending that the Secretary of DHS direct ISC to
(1) establish guidance and standards for measuring the performance of
facility protection efforts, particularly for program outcomes; (2)
communicate the established guidance and standards to relevant federal
agencies; and (3) ensure that the guidance and standards are regularly
reviewed and updated. In commenting on a draft of this report, DHS, USPS,
VA, and Interior generally concurred with the findings, and DHS concurred
with the recommendations. DHS, USPS, and Interior also provided comments,
which were incorporated as appropriate to ensure accuracy. GSA said they
did not have any comments on the draft report.

                                   Background

The protection of federal facilities gained importance after the 1995
bombing of the Alfred P. Murrah Federal Building in Oklahoma City, and
this issue became even more critical after the 1998 embassy bombings in
Africa; the September 11, 2001, attacks on the World Trade Center and the
Pentagon; and the anthrax attacks in the fall of 2001. Shortly after the
1995 bombing, the President signed Executive Order 12977, establishing the
Interagency Security Committee (ISC). ISC-which has representation from
all major federal departments, agencies, and key offices-was charged with
enhancing the quality and effectiveness of security in, and protection of,
nonmilitary facilities occupied by federal employees in the United
States.9 Furthermore, ISC was tasked to serve as a permanent body to
address continuing governmentwide security issues for federal facilities.
Under the order, ISC became responsible for developing policies and
standards, ensuring compliance and overseeing implementation, and sharing
and maintaining information. Around the same time that ISC was created,
the Department of Justice categorized all federal facilities into security
levels I through V based on factors such as facility size and number of
employees, and it established recommended minimum security standards for
each of the five levels. These standards covered perimeter, entry, and
interior security and security planning.10

The 2001 terrorist attacks prompted additional policies concerning
facility protection and a variety of security enhancements at federal
facilities. The Homeland Security Act of 2002 and a number of national
strategies, including the National Strategy for Homeland Security,11
assigned DHS specific duties associated with coordinating the nation's
efforts to protect critical infrastructures and key assets. Government
facilities (at the federal, state, and local levels) were identified as
key assets and therefore were included in this effort.12 Furthermore, the
2002 Act transferred FPS from GSA to DHS and, as a result, made DHS
responsible for ISC.13 A related directive, the Homeland Security
Presidential Directive Number 7 (HSPD-7), stated that DHS's Secretary was
responsible for coordinating the overall national effort to identify,
prioritize, and protect critical infrastructures and key assets.14 To meet
this responsibility, DHS developed a National Infrastructure Protection
Plan (NIPP), which is currently in draft form. FPS is responsible for
implementing the NIPP for the government facilities sector. HSPD-7 also
required each federal agency to develop plans to address identification,
prioritization, protection, and contingency planning for physical and
cyber critical infrastructures, along with key assets that they hold or
operate. As the governmentwide emphasis on protecting critical
infrastructures mounted, the federal agencies' facility protection efforts
continued to intensify. In addition to implementing such activities as
searching vehicles that enter federal facilities, restricting parking, and
installing concrete bollards, federal agencies also implemented various
security technologies, such as smart cards for access control. Figure 1
shows smart card technologies that are utilized at a federal building.

9ISC membership includes the Departments of State, Treasury, Defense,
Justice, Interior, Agriculture, Commerce, Labor, Health and Human
Services, Housing and Urban Development, Transportation, Energy,
Education, and Veterans Affairs; GSA; Environmental Protection Agency;
Central Intelligence Agency; and the Office of Management and Budget.
Other members of ISC include the Director, U.S. Marshals Service; the
Director, Security Policy Board; and the Assistant to the President for
National Security Affairs. As a member of ISC, the Department of Defense
participates in meetings to ensure that its physical security policies are
consistent with ISC security standards and policy guidance, according to
the Executive Director of ISC.

10U.S. Department of Justice, Vulnerability Assessment of Federal
Facilities, June 28, 1995.

11Office of Homeland Security, The National Strategy for Homeland
Security, July 2002.

12The other critical infrastructure sectors and key assets identified in
the National Strategy include agriculture and food, water, public health,
emergency services, defense industrial base, telecommunications, energy,
transportation, banking and finance, chemical industry and hazardous
materials, postal and shipping, national monuments and icons, nuclear
power plants, dams, and key commercial assets.

13Executive Order 13286, dated February 28, 2003, amended numerous
executive orders to reflect the transfer of certain functions and
responsibilities to the Secretary of Homeland Security. Section 23 of the
Executive Order transferred the ISC chairmanship responsibility from GSA
to DHS.

14Homeland Security Presidential Directive Number 7, Critical
Infrastructure Identification Prioritization and Protection, Dec. 17,
2003.

Figure 1: Smart Card Access Portals at a Federal Building Entrance

While it is evident from the policies and strategies outlined above that
the protection of key assets, including federal facilities, has become an
important issue for the U.S. government, the protection of such assets has
also gained attention in state, local, and foreign governments, as well as
the private sector. State and local governments in the United States, for
instance, have taken steps to ensure the protection of critical
infrastructures and key assets within their jurisdictions, often receiving
resources for such efforts from the federal government. For example, DHS's
Homeland Security Grant Program provides funding to state and local
governments to prevent, deter, respond to, and recover from acts of
terrorism. Funding from this grant program can be used for, among other
things, critical infrastructure protection activities. The protection of
critical infrastructures and key assets has also gained momentum in
foreign governments, particularly in countries like the United Kingdom
that have recently faced terrorist attacks. Furthermore, because many U.S.
critical infrastructures are owned and operated by the private sector, and
because some of these infrastructures have been targeted by terrorists in
the past, many private-sector entities have increased their investments in
security efforts.

Due in part to the growing attention to facility protection, we designated
federal real property as a high-risk area in January 2003 and have since
published a number of reports on this issue.15 In a November 2004 report,
we identified six key practices in protecting federal facilities, one of
which was measuring performance to help achieve broad program goals and to
improve security at individual facilities. We reported that, for broader
program goals, performance measures could indicate whether organizations
establish timelines and adhere to budgets. And, at the individual facility
level, on-site security assessments and other active testing could provide
data on the effectiveness of efforts to reduce a facility's vulnerability
to attack. Training exercises and drills are also useful in assessing
preparedness.16

The need for agencies to measure performance stemmed from the Government
Performance and Results Act of 1993 (GPRA),17 which was intended to
improve federal program effectiveness, accountability, and service
delivery. This act required federal agencies to develop strategic plans,
link them with outcome-oriented goals, and measure agency performance in
achieving these goals. Likewise, in the security context, a number of
national strategies called for federal agencies to use performance
measures to, among other things, assist in the planning and budgeting of
protection activities for critical infrastructures and key assets.

We have previously reported that successful performance measures should
(1) be linked to an agency's mission and goals; (2) be clearly stated; (3)
have quantifiable targets or other measurable values; (4) be reasonably
free of significant bias or manipulation that would distort the accurate
assessment of performance; (5) provide a reliable way to assess progress;
(6) sufficiently cover a program's core activities; (7) have limited
overlap with other measures; (8) have balance, or not emphasize one or two
priorities at the expense of others; and (9) address governmentwide
priorities.18

15For example, see GAO, High-Risk Series: An Update, GAO-05-207
(Washington, D.C.: January 2005); GAO-05-790 ; and GAO-05-49 .

16 GAO-05-49 .

17Pub.L. No. 103-62, 107 Stat. 285 (1993).

Managers can use performance measures in a number of ways to improve
programs and allocate resources more efficiently and effectively. Decision
makers can use results from performance measurement to identify problems
or weaknesses in programs, identify factors causing the problems, and
modify services or processes to try to address problems. Conversely,
results from performance measurement can be used to identify and increase
the use of program approaches that are working well and to consider
alternative processes in areas where goals are not met. Separately,
performance measures can also be used to identify priorities and allocate
resources. Decision makers can compare performance measure results with
program goals and subsequently determine where to target resources to
improve performance. Furthermore, in a risk management process, agencies
can use performance measurement to assess progress towards meeting
homeland security goals. The intended effect of assessing such progress,
when coupled with other aspects of the risk management process, is the
reduction of risk.19 Finally, when performance information is used to
reward individuals, these measures can hold individuals accountable for
certain work activities and related goals and, as a result, create an
incentive for achieving results. A greater focus on performance results
can be achieved by creating a cascade from an organization's goals and
objectives down to the individual performance level. Such alignment
facilitates the linking of individual performance to organizational
performance.20

18See GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing
Season Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 22, 2002),
pp. 2-3, 46-53.

19GAO, Risk Management: Further Refinements Needed to Assess Risks and
Prioritize Protective Measures at Ports and Other Critical Infrastructure,
GAO-06-91 (Washington, D.C.: Dec. 15, 2005), pp. 24, 105.

20See GAO, Managing For Results: Enhancing Agency Use of Performance
Information for Management Decision Making, GAO-05-927 (Washington, D.C.:
Sept. 9, 2005), pp. 7-17 and 21.

 Organizations outside of the U.S. Government Use Security Performance Measures
           to Enhance Decision Making and Help Ensure Accountability

We found a range of examples of performance measures that organizations
outside the U.S. government-including private-sector firms, state and
local governments, and foreign government agencies-used to track the
number and types of security activities conducted, the quantity of
security equipment and services delivered, and the outcomes of these
security efforts.21 Security officials within these organizations
recognized that performance measures helped them better assess how
effective they were in protecting against threats to and vulnerabilities
of their facilities. Organizations then used the results of these
performance measures to improve security, inform the risk management
process, make resource allocation decisions, and hold security officials
and others in the organization accountable for security performance.
Despite efforts by some organizations to use performance measures as an
additional decision-making tool, some security officials told us that they
faced some challenges in developing and implementing performance measures.
The challenges include limited guidance and expertise in the performance
measurement area.

Selected Organizations Use a Range of Output, Outcome, and Process/Input
Measures to Assess the Effectiveness of Facility Protection Efforts

Security officials recognized that performance measurement is important
for improving facility protection and ensuring accountability. They also
acknowledged that performance measures would allow them to take a more
strategic, outcome-based approach to managing their security programs and
to better prepare their facilities against terrorism and other threats.
However, less than a quarter of the organizations we interviewed told us
that they have developed and used various performance measures for their
security programs, and several of those that did have performance measures
said that the measures are still a work in progress. Table 1 provides
examples of the output, outcome, and process/input measures these
organizations have developed. Appendix II provides additional examples of
performance measures.

21For this report, we categorized the District of Columbia as a local
government.

Table 1: Examples of Performance Measures for Facility Protection

Type of measure Example                                                    
Output             o  Number of risk assessments performed                 
                      o  Average time to process background screenings        
                      o  Compliance with security policies                    
                      o  Client/customer satisfaction with security services  
Outcome            o  Evidence of damage to buildings and facilities       
                      o  Change in risk rating resulting from countermeasures 
                      deployed                                                
                      o  Change in the total number of security-related       
                      incidents                                               
Process/Input      o  Number of security clearances undertaken             
                      o  Number of training courses and drills conducted      
                      o  Number of security guards                            

Source: GAO.

Note: GAO analysis of data from selected state, local, and foreign
government agencies and private-sector organizations.

In some of the organizations we interviewed, some security officials use
output measures to monitor the direct products and services delivered by a
program and the characteristics of those outputs, including efficiency,
cost-effectiveness, timeliness, quality, and customer service. Some
security officials use outcome measures to compare the results of those
products and services with the goals security officials are trying to
achieve, such as reducing the total number of security incidents relating
to thefts, vandalism, and acts of terrorism. In addition, some security
officials use outcome measures to assess whether their security program is
operating efficiently and to determine the quality of the services and
products they are trying to provide. Separately, security officials use
various process/input measures to provide a descriptive overview of the
program activities and the resources of their security program, including
the types and numbers of facilities they manage and the level of
countermeasures,22 such as entry control security systems, they have
installed. Input measures are used for resource allocation and monitoring
and do little to reflect the effectiveness of the security program.

As an additional output measure, some of the organizations we interviewed
determine whether their security efforts comply with their security
policies, standards, and guidance. For example, some of the government
agencies in the three foreign countries we visited use performance
measures to evaluate whether their security activities are compliant with
their government's protective security policies. Several security
officials in these agencies told us that they use this measure to
demonstrate compliance with established government standards. Some of
these foreign government agencies indicated that they measure compliance
based on the results of security audits completed internally-by the
security department or other departments within the organization-or
externally. Some of these security officials then use the results of the
audits to identify security weaknesses and make corrections to improve
security. Other foreign government agencies use surveys to measure the
degree of security policy compliance. For example, Australian government
agencies are required to adhere to the minimum protective security
standards contained in the Australian government's Protective Security
Manual.23 Ministers and agency heads are accountable for their agency's
compliance with these standards. Agencies are surveyed annually for
compliance with the security manual standards. The survey results are
assessed and reported to the central government.

22A countermeasure is any action taken or physical equipment used
principally to reduce or eliminate one or more vulnerabilities.

Some of the nonfederal organizations we interviewed also measure the
effectiveness of their countermeasures by determining whether the services
and security equipment they provide are adequate under both real and
simulated conditions. Some of the organizations we interviewed stated that
they test security equipment, such as perimeter alarms and x-ray machines,
and conduct simulated attacks and penetration exercises on a periodic
basis. One official from the gaming industry said that it is important to
test equipment to ensure it is being used properly, because the technology
itself is not as important as how it is used. For example, a facility
could have a sophisticated card access system but still be vulnerable if
someone props the door open. To help government agencies select effective
security equipment, a central agency in the United Kingdom tests security
equipment and provides those in the security community with information to
help the user match the appropriate equipment to the requirement.
Similarly, an agency in Australia conducts tests on security equipment and
provides agencies with a catalog of approved products. Security officials
from the gaming industry also told us that they are members of an external
group that tests security equipment and shares the results of the testing
with security officials in other industries, such as the chemical,
petrochemical, and pharmaceutical industries.

23The Australian government's Protective Security Manual contains
governmentwide policies and guidelines that establish the minimum
standards for the protection of Australian government resources (including
information, personnel, and assets) that all agencies governed by the
country's Financial Management and Accountability Act of 1997 must meet.

In some organizations, the selection of useful performance measures has
evolved through a trial-and-error process. For example, one financial
services organization went through several iterations of its security
performance measures over a 1-1/2 year period in order to determine which
performance measures were important to monitor and would provide them the
right information needed to achieve the organization's security
objectives. For example, they initially reported on the number of security
alarms, and then changed the measure to a more useful measure-the number
of alarms with unique responses (i.e., alarms that required a guard to
respond in person)-so that they could better understand how security staff
were interacting with the security equipment. One security official
acknowledged that, although they were satisfied with their current
performance measures, it would still be helpful to measure performance in
other areas, such as employee satisfaction with security services.

  Case Example: A Financial Service Organization's Performance Measures

Security officials at a large, well-known financial services organization
use a number of output and outcome measures to regularly monitor the
performance of their security program. In addition, they use process/input
measures to assist them with resource allocation decisions. The security
officials emphasized that there is a constant need to measure and evaluate
what their security program does in order to educate business
professionals on the importance of a security investment. While the
organization assesses all of its facilities using a baseline set of
security standards and risk assessments, performance measures provide
security officials with information to understand whether these standards
and risk assessments are actually improving their security situation. The
security officials told us that they use the following performance
measures:

           o  Outputs-Security officials use output measures relating to
           their operational readiness (i.e., how prepared the security
           program is against potential security threats), which includes the
           number of risk assessments performed. They also measure the number
           of non-security related incidents such as false alarms or broken
           security cameras. In addition, security officials monitor the
           number of policy exceptions that exist when a business line or
           facility cannot comply with the standards set forth in their
           security policy manual. If many exceptions to a particular section
           of the policy manual occur in a given month, a policy working
           group reviews the issue and determines whether additional
           assistance will be required to bring the facilities into
           compliance.
           o  Outcomes-One outcome measure is the monetary savings resulting
           from less costly, more efficient security processes and new
           technologies. Security officials use this outcome measure to
           demonstrate savings from the security program's budget as well as
           from the budgets external to the security division, such as
           operations. Officials are also able to prorate contract-related
           savings over the lifetime of the contract to better understand how
           the savings affect the organization over time. To understand the
           effectiveness of their security efforts, security officials use
           data on the responses to security incidents, which are classified
           by type (e.g., assault, burglary, terrorism). Security officials
           then analyze the data to help them make recommendations for
           additional security improvements.
           o  Process/Input-The financial organization tracks guard levels,
           security expenditures, and security activities across all its
           facilities. Security officials use these measures to compare the
           different levels of service required, given the risk associated at
           each facility or region. In a given month, they also measure the
           number of training sessions and drills conducted. The performance
           measure for training identifies the specialized fields in which
           the security staff are being trained and the type of training the
           security staff are providing to others.

           Security officials at this financial services organization told us
           that they monitor their performance measures on a monthly basis,
           and that the data are aggregated for the entire organization and
           also broken out by region. They developed, and have continued to
           modify, their performance measures based on the analysis of
           incidents and other activities in a particular region as well as
           trends across regional facilities. They also obtained feedback
           from regional offices and from their own security staff. Security
           officials noted that they tried to select performance measures
           that represented common threads and were not biased in favor of
           one particular region. They also continuously evaluate the
           usefulness of their performance measures, adding a measure if they
           determine that information is needed on a particular subject or
           dropping a measure if it does not seem to be informative.

           We have previously reported that organizations can use the results
           of performance measures to make various types of management
           decisions to improve programs.24 Security professionals also
           recognize the benefits of using performance measurement within the
           security industry. At a major security industry conference in
           2005, a conference presenter indicated that the ability to compare
           past performance and the performance of others contributes to the
           goal of continuous improvement, the result of which is a stronger,
           more mature security program with security processes that can
           better protect facilities and staff from harm. Performance
           measures also provide management with the tools to verify that the
           organization's resources are used responsibly and security risks
           are managed appropriately.

           In some of the organizations we interviewed, security officials
           and other decision makers use performance measures to manage risk,
           allocate resources, and improve the quality of the security
           services they provide for their facilities. For example, at one
           financial services organization, security officials installed
           protective security equipment at some of their facilities and then
           compared the number of security incidents and the level of
           customer satisfaction before and after the equipment was
           installed. In this particular case, security officials used this
           performance measurement data to demonstrate the value of that
           security investment to their corporate management and the business
           lines they supported. The performance measures also allowed
           security officials to compare individual facility performance to
           the average within the industry, which they use to demonstrate the
           risk level of a particular facility and take appropriate action to
           address the risk.

           Where security goals and objectives were not achieved, some
           security officials also used performance measurement results to
           identify problem areas and take corrective action. Several
           organizations mentioned that they measure the quality of their
           security efforts through an output measure by soliciting feedback
           from employees and clients through customer satisfaction surveys.
           For instance, one Canadian organization periodically surveys
           clients about their satisfaction with the security services the
           organization provides to government agencies. The survey questions
           range from how often the client saw security managers to how
           satisfied they were with the services they received. The responses
           to the surveys provide feedback that allows security officials to
           improve their provision of security services to both private and
           public sector clients.

           Performance measures helped security officials in one government
           agency in Australia become better risk managers and allocate
           resources more efficiently across facilities. The agency uses a
           security plan that includes security objectives that are linked to
           its strategic goals. The plan also lists strategies and actions
           for achieving these objectives, along with performance measures
           that assess the extent to which objectives are being achieved. For
           example, the performance measures monitor the extent to which
           security practices are in accordance with the agency's security
           policies, any evidence of harm to agency staff or facilities, and
           the extent to which agency stakeholders view the agency's
           facilities as safe for their resources and assets. To monitor
           performance, security officials use two different review
           processes. First, security officials can access the audit function
           of a computer-based risk assessment model to monitor the outcomes
           of the performance measures contained in their security plan and
           to understand how well their security efforts are performing
           within individual facilities. For example, the risk-assessment
           model allows security officials to monitor the impact of
           additional security expenditures on a facility's risk rating while
           controlling for existing security enhancements that mitigate the
           risk, such as the number of guard patrols and the adequacy of
           access control systems (e.g., electronic locks). Security
           officials can then use the results to justify spending decisions
           and prioritize security investments. For example, one facility
           requested a perimeter fence, and security officials were able to
           use the risk-assessment model to demonstrate that the facility's
           risk was adequately managed without the fence since there were no
           known risks in that location and since the facility already had
           guards and an alarm system. Second, the agency's audit unit also
           conducts its own independent measurement of the security
           activities so that security officials can compare across
           facilities to guide them in determining where they need to make
           adjustments. Together, these two security reviews provide the
           security program with enough information to assess their security
           position, according to one agency security official.

           Security officials recognized the value of performance measures to
           help ensure the accountability of security officials, management,
           and other employees throughout the organization. Many of the
           organizations we interviewed had security policies and procedures
           in place, and some of these organizations were able to link these
           plans directly to performance measures that demonstrated
           achievement of both the security-related strategic goals and the
           organization's broader strategic goals. We have previously
           reported that aligning the goals at the executive level with the
           goals and objectives at each operational level reinforces the
           connection between strategic goals and the day-to-day activities
           of managers and staff.25 For example, an annual goal that is
           linked to a program and also to a long-term goal can be used to
           hold agencies and program offices accountable for achieving those
           goals.26 Furthermore, we reported that such alignment increases
           the usefulness of performance information to decision makers at
           each level.27

           One agency within the District of Columbia (D.C.) government uses
           performance measures and targets to hold agency management and
           security officials responsible for its security-related
           activities. D.C.'s Office of Property Management is responsible
           for D.C. government buildings, and the Protective Services
           Division, which falls under Property Management, is responsible
           for security at these buildings. Protective Services faces a
           unique environment in protecting the facilities that it is
           responsible for because of the proximity of these assets to
           federal facilities, which are considered to be attractive targets
           for terrorist attacks. To help ensure that their security concerns
           are addressed, security officials in Protective Services noted
           that they have linked their security goals and related performance
           measures with the Property Management's goals and citywide
           strategic goals (see fig. 2). Specifically, Protective Services'
           goals, performance measures, and related targets support the goal
           of Property Management to provide a high-quality work environment
           and user-friendly facilities, and also support the broader
           citywide strategic goal of making government work. The security
           officials pointed out that this alignment is very deliberate and
           can help hold officials accountable for a security-related
           activity. For example, the Director of Property Management can use
           security-related performance measures and corresponding targets to
           hold the Protective Services Division accountable for its
           activity. If Protective Services does not meet the targets, it is
           required to submit justifications to senior management as to why
           they were not met. The officials explained, however, that in
           situations where there are unforeseen circumstances, their targets
           can be realigned, with the consent of senior management. For
           example, following Hurricane Katrina, Protective Services was
           required to provide security services for Katrina victims housed
           at a D.C. arena. The human resources required for this task made
           it impossible for Protective Services to meet all the targets, and
           the D.C. mayor's office allowed for adjustments to the target for
           that time. Separately, the mayor's office can also use the
           security-related performance measures and targets in conjunction
           with other Property Management performance measures and targets to
           monitor the work of the entire agency and hold the Director of
           Property Management accountable for agencywide activity.

           Figure 2: Linkages between District of Columbia Strategic Goals
           and Performance Measures for Facility Protection

           We also recognized in a previous report that the establishment of
           a chief security officer position is essential in organizations
           that own and operate large numbers of mission-critical
           facilities.28 Such a high-level position is important for
           coordinating security responsibilities across facilities and
           ensuring accountability for security results, including
           establishing linkages between security performance and outcomes.
           We found that government agencies in all three countries we
           visited are required to designate a departmental security officer
           (DSO) or an agency security executive to oversee security matters
           across all agency facilities and implement government security
           policies. For example, in the United Kingdom, security officials
           told us that the DSOs are sufficiently senior within each agency
           department to have an effective voice and to put security issues
           on the management agenda. These security officials also told us
           that the DSOs are playing a greater role in coordinating with
           other agency departments to enhance their security. The financial
           services and gaming organizations we interviewed also have
           directors or vice-presidents of security who have a direct line of
           communication to their corporate management. They said that this
           arrangement promotes a good working relationship with management
           and allows them to identify and fix security problems efficiently.

           Some of the organizations we interviewed also used performance
           measures to hold security officials accountable for program
           performance. For example, some organizations hold their security
           officials accountable for results through the use of customer
           satisfaction surveys. Security officials at one financial services
           organization indicated that they conduct quality surveys with
           their business-line clients, which allows clients to provide input
           to security officials on whether the security program is effective
           and whether the security program met the client's expectations.

           Two major financial services organizations we interviewed use
           performance measures to help ensure accountability for investments
           in security improvements and compliance with security policies and
           regulations. Security officials in one financial services
           organization told us that they work in a security culture that is
           very performance driven. While their security budget is fully
           separate from other corporate expenditures, regional security
           directors are responsible for determining how to spend security
           funds. Regional security directors use performance measures to
           justify security expenditures to all of the individual business
           lines they support and to demonstrate a return on investment for
           their security expenditures. For example, the organization uses
           output and outcome performance measures to monitor monetary
           savings, the number of security incidents, and the impact of new
           technologies and processes. When security officials want to invest
           in a new security technology, they use these performance measures
           to demonstrate to the business lines that they have investigated
           all of the alternatives and determined the cost and potential
           savings of the purchase. For example, they used past data on the
           cost and performance of security equipment and guards to calculate
           the cost of installing some security equipment versus hiring a
           security guard to protect one of its facilities. They were able to
           demonstrate that the security equipment would be more
           cost-efficient over time and be more effective in deterring
           certain crimes.

           Another financial services organization uses performance measures
           to help ensure that all facilities are complying with its security
           policies and regulations. The security policies for each of the
           organization's business lines differ based on their level of risk.
           As a form of quality control for its security operations, each
           business line is expected to conduct compliance reviews of all of
           its facilities, including confirming the presence of required key
           security equipment and determining whether staff members are
           following security policies. Each business manager is held
           accountable for the results of these reviews: senior security
           officials receive and review monthly compliance reports, and the
           financial services organization's central audit department ensures
           that the reviews were properly conducted. According to security
           officials, the data in the monthly reports are used to determine
           where problems exist and look for emerging security trends.

           One Australian government agency uses performance measures to hold
           its security executives accountable for identifying and addressing
           security risks. Officials from the agency noted that they have
           historically had a strong security and risk management culture
           that emphasizes executive accountability for performance. The
           agency holds its security executives accountable by requiring them
           to produce a certificate of assurance that includes physical and
           personal security. The purpose of the certificate, which is signed
           by a senior agency executive, is to assure the chief executive
           that the agency is meeting its security obligations, and that
           action plans are in place to address any problems. It covers
           compliance with external requirements, including government
           regulations, and internal conformance with corporate security
           policies. The assurances given must be underpinned by evidence,
           which includes the results of physical security reviews that are
           conducted periodically at each facility. These reviews measure and
           report on the standard of physical security, including perimeter
           security, access control, alarm systems, and surveillance and
           communication systems. The certificate uses a color code to
           indicate the overall status of the security function-red, amber,
           or green. Certificates rated red or amber are reviewed and
           resubmitted every 6 months. Green certificates are reviewed
           annually. If the certificate identifies a security problem, it
           must be accompanied with an action plan for addressing the risks
           involved.

           Although performance measurement is seen as an important tool for
           assessing the effectiveness of security programs, developing and
           using performance measures can be challenging, according to
           security officials we interviewed at selected organizations. A
           difficulty with developing performance measures is determining
           whether the measures that are used are suitable, given a
           constantly changing threat environment. Some security officials
           said that it was difficult to know what to measure because
           security is intangible or difficult to quantify. Others also
           acknowledged that it is difficult to determine whether a potential
           security threat or incident has been prevented, even after
           additional countermeasures or security staff are introduced to
           address perceived security weaknesses, because deterrence is
           immeasurable. Several security officials cited the difficulty in
           determining a causal relationship between security efforts and
           changes in the number of security incidents. For example, a
           security official from an Australian government agency indicated
           that an increase in the number of breaches in a particular
           facility may result because an organization is being targeted at
           that particular point in time rather than because it lacks
           adequate security measures. Organizations also find it hard to
           measure the impact of some security actions, such as the potential
           financial savings resulting from attacks that have been
           discouraged. Organizations told us that they recognize the need to
           draw linkages between security incidents and security investments,
           but some organizations find it difficult to measure the benefit of
           a particular security process or piece of equipment in the absence
           of a security breach.

           A number of organizations also told us that other priorities and
           considerations might hinder their ability to effectively use
           performance measures for making security decisions. Some security
           officials pointed out that the ultimate decision on how to
           allocate security resources can be based on priorities other than
           performance. For example, several private sector and foreign
           government agencies we interviewed noted that they have to balance
           their security needs with their goals of maintaining sufficient
           public access to their facilities. Some security officials are
           also reluctant to use performance measures because they do not
           want to be held accountable for not meeting their performance
           targets. Several organizations mentioned that potential liability
           could be seen as a disincentive for using performance measurement
           data, because an organization may be seen as negligent if the
           performance data were to show that an organization could have done
           something to prevent an incident but chose not to. One security
           official told us that having established performance targets could
           also discourage organizations from accurately collecting data
           because security officials may be reluctant to report an incident
           if a decline in the number of incidents is one of the performance
           goals.

           Some organizations we interviewed cited the lack of knowledge and
           expertise available to collect and analyze security data as a
           limitation to overcoming some of the challenges of using
           performance measures. One financial services organization
           indicated that some of its security officials did not see the
           benefits of using performance measures until after they saw that
           their business line managers responded favorably to the use of
           performance measures to demonstrate a return on investment for
           security expenditures. Several state, local, and foreign
           government agency officials noted that they had limited management
           staff available to develop and monitor performance measures for
           physical security. According to one state government agency
           official, without staff expertise in this area, security staff
           tend to approach security initiatives like a project-they monitor
           the initiative's progress to make sure that it is delivered on
           time and on budget, but they do not necessarily measure the
           effectiveness of the security equipment once it is installed.

           Many organizations we interviewed said that they face the
           aforementioned challenges, and we noted that some of the entities
           outside the U.S. government rely on U.S. agencies for support and
           leadership in developing security standards and performance
           measures. One state government agency we interviewed expressed an
           interest in developing performance measures in the future and
           mentioned that it often looks to the federal government for
           guidance on security efforts. DHS officials told us that their
           agency was providing assistance to several foreign government
           agencies in the United Kingdom in measuring performance and
           allocating security resources. One foreign government agency said
           that it was interested in developing governmentwide guidance for
           measuring security performance but was looking to U.S. agencies
           for assistance in this area.

           Responding to the requirements in 2002 by the National Homeland
           Security Strategy and subsequent federal policies, agencies have
           paid greater attention to facility protection and have begun using
           key practices-such as performance measurement-to varying degrees.
           Agency officials noted that developing performance measures for
           facility protection was a difficult undertaking, since the results
           are not always readily observable. We found that some bureaus and
           services within three of the agencies we reviewed-DHS, USPS, and
           Interior-are using output measures and, to a lesser extent,
           outcome measures, while the VA and some bureaus and services
           within the other three agencies are not. Despite the lack of
           security performance measures, we found that ongoing protection
           activities within these bureaus and services and the VA, such as
           monitoring the numbers and types of security breaches at a given
           facility, generate a wealth of data that could provide useful
           feedback about the agency's effectiveness in mitigating building
           security risks, and therefore could be used as measures of
           performance. While the agencies have demonstrated some progress in
           applying performance measurement to facility protection, with
           limited precedent for how to do this, more work remains to
           identify measures-particularly outcome measures-that assess the
           impact of facility protection efforts. Output measures do not
           provide an indication of what security activities are
           accomplishing, while outcome measures that are clearly tied to
           results indicate the extent of progress made and help identify the
           security gaps that still remain. Officials expressed concerns
           about the lack of resources and the limitations of existing
           guidance in providing direction about how to measure progress and
           evaluate the effectiveness of physical security programs.

           In general, the agencies we reviewed have made some progress in
           collecting and using performance-related data for their facility
           protection program activities, but many of the measures are of
           program outputs rather than outcomes. While output measures are an
           important part of performance measurement, outcome measures could
           provide information to evaluate whether the benefits of security
           investments outweigh their costs and to determine the
           effectiveness of security activities. The agencies we reviewed use
           output measures, such as the timely completion of risk assessments
           and whether countermeasures work as intended once deployed, to
           inform risk management decisions and to help ensure adequate
           protection at the individual facility. Additionally, several
           bureaus and services within DHS, USPS, and Interior have developed
           outcome measures to rank and monitor the relative risk ratings of
           their respective facilities over time or to otherwise assess the
           effectiveness of their facility protection efforts.

           The effectiveness of security programs at GSA facilities is
           evaluated using performance measures developed by the Federal
           Protective Service (FPS) and a physical security testing program
           developed by GSA. FPS has identified four performance
           measures-both output and outcome measures-to assess its efforts to
           reduce or mitigate building security risks. These four performance
           measures, detailed in table 2, are at varying stages of
           implementation and are still evolving. Under the Homeland Security
           Act of 2002, DHS, through FPS, is directly responsible for law
           enforcement and security-related functions at facilities under
           GSA's control or custody. FPS delivers security and law
           enforcement services for approximately 8,000 facilities that fall
           under GSA's jurisdiction.

Security Officials Use Performance Measure Results for Risk Management and
Resource Allocation

24See GAO-05-927 .

  Case Example: An Australian Agency's Risk Model

Performance Measures Can Be Used to Hold Security Officials Accountable for
Achieving Goals and Results

  Case Example: The District of Columbia's Alignment of Security Goals and
  Measures

25 GAO-05-927 .

26 GAO-03-143 .

27 GAO-05-927 .

  Departmental Security Officers and Individual Accountability

  Case Example: Individual Accountability in Two Financial Services
  Organizations

28 GAO-05-790 .

  Case Example: An Australian Agency's Security Certification Process

Organizations Cited Challenges in Developing and Using Performance Measures

U.S. Agencies Have Made Some Progress in Developing and Using Performance
Measures for Facility Protection Programs, but Lack Guidance and Standards

Agencies Use Output Measures and Some Outcome Measures to Inform Risk
Management, Help Ensure Adequate Protection, and Assess Effectiveness of
Facility Protection Efforts

  Case Example: DHS's Federal Protective Service in GSA Facilities

Table 2: FPS's Performance Measures for Facility Protection

Type of                                                                    
measure Performance measure         Purpose
Output  Timely deployment of        To compare actual deployment dates     
           countermeasures             with planned deployment dates          
Output  Countermeasure              To gauge whether those security        
           functionality (e.g.,        countermeasures for which FPS is       
           surveillance cameras, x-ray contractually responsible are working  
           machines)                   as intended, once deployed             
Output  Patrol and response time    To assess FPS's ability to respond to  
                                       calls for service within certain time  
                                       limit goals                            
Outcome Facility security index     To calculate FPS's average success     
                                       rate for the above three performance   
                                       measures                               

Source: GAO.

Note: GAO analysis of FPS data.

The first measure-monitoring the deployment of countermeasures-focuses on
the timeliness of implementation and serves as a measure of program
output. Once approval and funding have been obtained to implement a
recommended countermeasure, FPS personnel record planned deployment dates
so that they can compare them with actual implementation dates. An FPS
working group decided that the initial baseline for this measure,
developed in fiscal year 2005, would be 90-percent success, which is
calculated as the number of countermeasures actually deployed by the
scheduled due date, divided by the number planned. FPS officials noted
that they will not know how well they are progressing on this measure
until the end of fiscal year 2006 because they are still automating the
process and training regional staff. For fiscal year 2007 and subsequent
years, FPS expects the annual goal to be some increment above the
preceding year's results until the long-term goal of 98 percent is
achieved and maintained.

Countermeasure functionality, FPS's second measure, gauges whether a
countermeasure works as intended once it is deployed. Specifically, it
assesses the operational capability of five major groups of
countermeasures for which FPS is contractually responsible: closed circuit
television surveillance, contract security guards, intrusion detection
systems, magnetometers, and x-ray machines. In some instances, contract
guards are routinely evaluated to determine whether they are performing
effectively. Performance includes the guards' knowledge of and compliance
with relevant operations for their security post. Based on FPS testing
results in fiscal year 2005, the baseline for this measure is 90-percent
success, which is calculated as the number of countermeasures working and
performing as intended divided by the number tested. According to FPS
officials, FPS currently has about a 92-percent success rate for this
measure. The long-term goal for this measure is 100-percent effectiveness.
Related to facility protection, this output measure reflects the
functionality of a program element, but not its effect.

Patrol and response, the third measure, assesses FPS's ability to respond
to calls for service within certain time limit goals. The initial baseline
for this measure was established in October 2005 and was about 17.5
minutes. This baseline represents an average response time for all of
FPS's 11 regions, and is calculated using dispatch and arrival time
information from FPS's online incident reporting system. The time
parameters for data collection fell between FPS's core duty hours of 6:00
a.m. and 6:00 p.m. The goal for this measure is to reduce response times
by 10 percent, although FPS noted that this goal could increase or
decrease depending on staffing levels or deployments. At the time of this
report, FPS noted that they have collected statistics on response times
for this measure and are in the process of evaluating whether they have
achieved their goal.

Finally, the facility security index-an outcome measure29-calculates the
overall effectiveness of FPS operations in meeting the performance goals
of the three output measures described above (timely deployment of
countermeasures, countermeasure functionality, and patrol and response
time). An index score of 1 indicates that FPS has met its performance
goals, a score of greater than 1 indicates that FPS has exceeded the
goals, and a score of less than 1 indicates that it has not met the goals.

29Although FPS considers this an outcome measure, it is intended to
reflect the composite level of performance of its three output measures.

Taken together, these four FPS performance measures provide insight into
activities designed to support FPS's efforts to prevent and respond to
security and criminal incidents, including terrorist threats. In addition
to assessing FPS's performance in fulfilling its facility protection
responsibilities, the measures also serve as a baseline for making
decisions about deploying existing resources or requesting additional
resources. FPS officials told us that these measures are derived from
strategic objectives established by DHS's Immigration and Customs
Enforcement (ICE), of which FPS is a component. These objectives include
implementing appropriate countermeasures to reduce vulnerabilities facing
buildings under GSA's jurisdiction (see fig. 3). Aligning facility
protection performance measures and targets with broader DHS and ICE
mission, goals, and objectives helps hold employees accountable for
security activity and allows them to observe how day-to-day security
activities contribute to the broader mission, goals, and objectives.
Similar to organizations outside the federal government, FPS provides its
financial management staff with quarterly and annual reports that document
the accomplishments for each measure in order to support planning and
budgeting efforts included in DHS's Future Years Homeland Security Program
document.30

30The Homeland Security Act requires that, beginning in fiscal year 2005,
DHS prepare the Future Years Homeland Security Program document-a 5-year
resource plan that outlines departmental priorities and the ramifications
of program and budget decisions. See GAO, Results Oriented Government:
Improvements to DHS's Planning Process Would Enhance Usefulness and
Accountability, GAO-05-300 (Washington, D.C.: Mar. 31, 2005).

Figure 3: Linkages between DHS Mission and FPS Performance Measures for
Facility Protection

It is important to note that when FPS was a part of GSA, we reported on
GSA's lack of performance goals and measures for its building security
program. In June 1998, we testified that GSA had not established key
program evaluation mechanisms for its building security program that could
help determine how effective its security program has been in reducing or
mitigating building security risks or in shaping new security programs.31
At the time, we reported on features that would support program
evaluation, including: (1) developing specific goals, outcomes, and
performance indicators for the security program, such as reducing the
number of unauthorized entries; (2) establishing and implementing
systematic security program evaluations that provide feedback on how well
the security program is achieving its objectives and contributing to GSA's
strategic goals; and (3) ensuring that a reliable performance data
information system is in place. While we found that GSA had established
goals and measures for its security program both apart from and in
connection with GPRA, we noted that these goals and measures were output
oriented and did not address the outcomes or results the building security
program was expected to achieve. Consequently, we recommended that GSA
develop outcome-oriented goals and measures for its building security
program. As previously noted, FPS has demonstrated some progress in moving
beyond the use of output measures that monitor program activity in
carrying out its responsibilities within the Department of Homeland
Security (DHS).

In addition to FPS's performance measures for assessing the security of
properties under GSA's control, GSA's Office of the Chief Architect also
has a program for testing the physical security of GSA buildings. Under
this program, GSA performs explosive testing of various window systems;
identifies gaps in protective design and security technologies; and
provides criteria and tools for blast resistant design, progressive
collapse in new and existing facilities, and upgrading walls to reduce
fragmentation and hazards resulting from an explosion, among other things.
The program team is also developing a tool to identify gaps in security
planning, ensure consistency with GSA policies and ISC's security design
criteria, and provide a consistent foundation and methodology for making
security design decisions.

  Case Example: Interior's Bureau of Reclamation and National Park Service

One bureau within Interior-the Bureau of Reclamation (BOR)-has identified
performance measures for its facility protection programs, while the
National Park Service (Park Service) generates information that could be
used to monitor the effectiveness of its physical security efforts. Each
of Interior's eight bureaus independently manages the protection program
for the facilities that fall under its respective purview, and each bureau
has developed broad security goals derived from the agency's overall
mission.32 In general, Interior's program evaluation methods are based on
GPRA and the Office of Management and Budget's (OMB) Program Assessment
Rating Tool (PART).33 Several of the bureaus have had their programs
reviewed under the PART system, and some security performance measures
were identified as part of this effort. Over time, Interior intends to
have all of its law enforcement programs assessed under the PART system.
However, an agency official from the Park Service reported difficulty in
developing formal performance measures because GPRA is directed toward
evaluating federal programs and does not provide guidance on developing
goals and measures specifically for security activities.

31GAO, General Services Administration: Many Building Security Upgrades
Made But Problems Have Hindered Program Information, GAO/T-GGD-98-141
(Washington, D.C.: June 4, 1998).

Within Interior, BOR has an important role in protecting critical
infrastructures because of its responsibilities related to dams. BOR is
responsible for managing and protecting well-known assets such as Hoover
Dam in Arizona and Nevada, which receives approximately 1 million paying
visitors each year. In 2005, the security program administered by BOR was
selected for review under the PART system. To demonstrate its progress in
meeting the long-term goal of reducing security-related risks at its
critical facilities, BOR developed several output and outcome performance
measures, including (1) timely completion of risk assessments, (2) the
cost per active background investigation, (3) the percentage of
recommendations that have been implemented based on the results of risk
assessments, (4) the number of updated regional threat assessments, and
(5) changes in the risk ratings as countermeasures are implemented for an
individual asset (see table 3). Although these measures were developed for
the protection of dams and related facilities, they could be applied to
building security because there is some similarity in the protection
activities. In all but one instance, BOR had achieved or exceeded its
performance target for each measure established for fiscal year 2005.
According to OMB's PART assessment, BOR's facility protection program was
rated moderately effective and its performance measures were described as
creative and useful measures that will help monitor program
accomplishments and efficiency.34

32However, to centrally manage Interior's security initiatives, the
department established in 2002 a central coordination and oversight office
for activities related to homeland security. This office-the Office of Law
Enforcement and Security-has worked within Interior to identify assets
that are likely targets, conduct risk assessments, and coordinate efforts
by Interior's bureaus to enhance security at individual locations. See
GAO-05-790 .

33OMB developed PART to support the integration of performance information
and budgeting. OMB describes it as a diagnostic tool meant to provide a
consistent approach to evaluating federal programs as part of the
executive budget formulation process.

Table 3: BOR's Performance Measures for Facility Protection

Type of                                                                    
measure Performance measure      Purpose
Output  Timely completion of     To compare actual completion dates with   
           risk assessments         planned completion dates                  
Output  Cost per active          To monitor the cost efficiency of the     
           background investigation personnel security program, including     
           file                     processing of background investigations,  
                                    issuance and verification of clearances,  
                                    and case file maintenance                 
Output  Status of                To indicate the percentage of recommended 
           recommendations designed security enhancements that have been      
           to mitigate risk         funded and implemented, and are           
                                    operational                               
Output  Number of updated        To assess the frequency with which        
           regional threat          assessments are conducted and help ensure 
           assessments              that current threat intelligence is       
                                    incorporated as part of risk assessments  
                                    and risk-reduction strategies             
Outcome Change in risk ratings   To assess the risk-reduction benefits     
                                    associated with implementing              
                                    countermeasures at an individual asset    

Source: GAO.

Note: GAO analysis of BOR data.

The Park Service is responsible for managing and protecting some of the
nation's most treasured icons, including the Washington Monument, the
Lincoln and Jefferson Memorials, and the Statue of Liberty. The Park
Service manages more than 375 park units, covering more than 84 million
acres, which provide recreational and educational opportunities and
numerous other benefits to millions of visitors each year. From 2001 to
2005, park units averaged a total of about 274 million recreation visits
per year. While a Park Service official stated that they did not have any
formal performance measures for facility protection, we found that their
risk management methodology provides useful feedback about the bureau's
effectiveness in reducing or mitigating security risks for facilities
under its jurisdiction. In June 2005, we reported that Interior had made
significant progress in the risk assessment area, in large part due to its
new National Monuments and Icons Assessment Methodology (NM&I).35 NM&I-a
uniform risk assessment and ranking methodology-is specifically designed
to quantify risk, identify needed countermeasures, and measure
risk-reduction benefits at icon and monument assets. According to an
Interior official, Interior's Office of Law Enforcement and Security
(OLES) developed NM&I to assist bureaus in quantifying risk levels and
identifying needed security enhancements, initially at critical
infrastructures and key assets, but eventually at all departmental
facilities. The NM&I methodology has a consequence assessment phase and a
risk assessment phase. First, during the consequence assessment phase,
senior officials from the Park Service and OLES determine which icons are
considered nationally significant.36 Specific attack scenarios-such as
chemical/biological, aircraft, or improvised explosive device-are used to
evaluate security at each asset and score attack consequences.37 During
the risk assessment phase, a group of security professionals from the Park
Service and OLES, assisted by the site security supervisor and the site
manager, collectively determine the effectiveness of existing security
systems using DHS guidelines. Using risk values calculated from this
evaluation, OLES assigns asset risk ratings of high, medium, or low, and
specific mitigation recommendations are formulated. As part of its annual
review, OLES routinely monitors the security enhancements that have been
implemented to reduce the risk rating designations. OLES has not had
formal performance measures and targets for reducing risk ratings in the
past. However, in April 2006, according to Interior officials, OLES
developed and submitted for inclusion in the departmental strategic plan
performance measures related to the reduction in the percentage of
physical security vulnerabilities identified at departmental facilities.
If adopted, such outcome measures could provide valuable feedback about
the Park Service's progress and overall effectiveness in protecting its
physical assets.

34According to OMB, a moderately effective rating means that a program is
well managed and has established ambitious goals. Programs with this
rating likely need to improve their efficiency or address other problems
in design or management to achieve better results. See www.expectmore.gov,
which is a Web site that was developed by OMB and federal agencies to
provide information on PART ratings.

35See GAO-05-790 . Before the development of this approach, Interior did
not have a uniform comprehensive risk management approach for national
icons and monuments-most of which are highly visible and tend to have
public access. It relied instead on the judgment of senior officials in
determining where resources should be directed, and the risk assessments
completed at individual sites were done by a number of external experts
using different methodologies. In our June 2005 report, we recognized that
Interior had made progress in addressing this concern but recommended that
the agency link the results of its risk assessments and related risk
rankings to its funding priorities and develop guiding principles for
balancing security initiatives with its core mission. Regarding the
recommendation to develop guiding principles, Interior officials told us
that they have not made any progress on this effort, in large part because
resources have been dedicated to meeting the requirements of a
presidential directive that calls for governmentwide identification
standards and processes for federal employees and contractors.

36Interior officials said that they consider the following characteristics
in determining which monuments and icons are nationally significant: (1)
asset is widely recognized to represent the nation's heritage, tradition,
or values or is widely recognized to represent important national
cultural, religious, historical or political significance; (2) asset's
primary purpose is to memorialize or represent some significant aspect of
the nation's heritage, tradition, or values, and to serve as a point of
interest for visitors and educational activities; (3) if asset were
successfully attacked, it would damage the American psyche and/or
international confidence in the United States; and (4) asset is a
monument, physical structure, or geographic site.

37Consequence categories include casualties, economic impact, and length
of disruption.

  Case Example: USPS Inspection Service

The USPS Inspection Service utilizes an outcome-oriented performance
measure to help ensure that it is progressing towards its strategic goal.
USPS has over 38,000 facilities nationwide that collectively handle about
700 million pieces of mail every day, and the agency serves over 7.5
million customers daily in its post offices. Postal facilities are a
compelling target for criminal and terrorist attacks, as evidenced by the
anthrax attacks in 2001, which put at risk the integrity of the mail and
the safety of USPS's employees, customers, and assets. Within USPS, the
Inspection Service-an investigative branch whose mission is to protect the
nation's mail system and its critical assets (i.e., employees, customers,
and facilities)-established its first performance measure related to
facility protection: the percentage of facilities that have high-risk
ratings (see table 4).38 This outcome measure allows the Inspection
Service to monitor progress toward achieving its strategic goal of
ensuring a safe, secure, and drug-free environment.

Table 4: Inspection Service's Performance Measure for Facility Protection

Type of measure Performance measure Purpose                                
Outcome         Percentage of USPS  To monitor the effectiveness of        
                   facilities with     countermeasures through the percentage 
                   high-risk ratings   of USPS facilities that score more     
                                       than 800 points                        

Source: GAO.

Note: GAO analysis of USPS data.

Specifically, this effort involves annual security surveys of facilities
conducted by facility protection control officers, as well as periodic
comprehensive reviews of larger core postal facilities performed by the
Inspection Service. The data from these surveys and reviews are maintained
in a database and used by the Inspection Service to tabulate a risk score
based on USPS's Facility Risk Rating Model. Several data elements are
considered to compute the composite risk score for a given facility,
including:

38In addition to the Inspection Service, USPS also has an Emergency
Preparedness group that works in close conjunction with the Inspection
Service to integrate emergency preparedness training and awareness from an
operational perspective.

           o  crime statistics;
           o  building characteristics (e.g., the absence or presence of
           customer parking, whether the facility is attached to an adjoining
           structure);
           o  location information (e.g., the number of federal buildings
           within a 1-mile radius of the post office);
           o  operational policies and procedures (e.g., the absence or
           presence of policies related to visitors, the timely completion of
           the facility security survey within the last 12 months); and
           o  countermeasures (e.g., the absence or presence of closed
           circuit television surveillance cameras).

           Using these data elements, the maximum risk score that can be
           computed for a facility is 2,854 points. After each element at a
           particular facility is assigned a risk score, the system ranks the
           facilities according to the designated composite risk score. The
           scoring and ranking system is national and is applied to all USPS
           facilities, which allows officials to compare facilities across
           the country using standardized data to identify which buildings
           are at the highest risk level. Facilities with scores at or above
           the threshold score of 800 are considered to be high-risk.39 The
           Inspection Service reassesses its facilities every 3 years or when
           a facility undergoes any major renovations or expansions. However,
           if a facility receives a high-risk score, the facility can be
           reassessed more often to help ensure that countermeasures are
           effective and that USPS has lowered the security risks. For
           example, if a facility received a high-risk score in fiscal year
           2005, the Inspection Service will revisit that facility again in
           fiscal year 2006 to try to lower the risk score. The target is to
           reduce facility risk scores for 90 percent of the facilities that
           have a high-risk designation. At the time of our review, USPS was
           successful in meeting its performance target, according to
           Inspection Service officials.

           The Inspection Service's outcome performance measure, outlined
           above, is closely aligned with its strategic goal-to ensure a
           safe, secure, and drug-free environment-and with its strategic
           objective-to enhance the security of USPS facilities. Linking
           their performance measures and targets with their strategic goals
           and objectives in this way provides managers and staff in the
           Inspection Service with a roadmap that shows how their day-to-day
           activities contribute to achieving broader Inspection Service
           goals (see fig. 4). Inspection Service officials told us that they
           designed their security-related strategic goal and objective to
           support USPS's broader strategic goal of improving services, which
           includes activities that protect mail, employees, and customers in
           order to improve services.40

           Figure 4: Linkages between USPS Inspection Service Strategic Goals
           and Performance Measure for Facility Protection

           Although it does not use security performance measures, VA
           collects data that could be used to assess the effectiveness of
           the agency's facility protection program. VA manages a large
           health system for veterans that now includes 154 medical centers,
           875 ambulatory care and community-based outpatient clinics, and
           136 nursing homes. In 2005, more than 5 million people received
           care in VA health care facilities, and VA's outpatient clinics
           registered nearly 58 million visits. VA also operates 61 major
           veterans' benefits facilities, including 57 regional offices, 3
           records centers, and headquarters.41 While VA officials noted the
           absence of performance measures for facility protection, we found
           that the Veterans Health Administration and the Veterans Benefit
           Administration rely on physical security assessments to inform
           risk-management and resource-allocation decisions, just as other
           federal agencies and nonfederal entities do. The phases of the
           physical security assessment include defining the criticality of
           VA facilities, identifying and analyzing the vulnerabilities of
           VA's critical facilities, and identifying appropriate
           countermeasures. VA determines vulnerability based on factors such
           as facility population, building characteristics (e.g., the number
           of floors in the facility), and the presence or absence of armed
           officers and surveillance cameras. VA's assessment includes a
           procedure for scoring and prioritizing identified vulnerabilities
           at each assessed site. The objective of the security assessment is
           to identify shortcomings in the physical security of functional
           areas within critical facilities and to estimate the cost of
           mitigating the risk of disruption or termination of the facility's
           ability to provide services to veterans. For example, they assess
           the vulnerability of a facility's air system to a criminal attack.
           For each assessed functional area, a composite score and
           corresponding risk rating is assigned. The risk-rating system is
           based on a color-coded "traffic light" scheme to designate low-,
           medium-, and high-risk functional areas. The results from the
           security assessment-in particular, the risk-rating designation-are
           used to develop recommendations to mitigate the security risk and
           to prioritize and justify resource-allocation decisions. VA
           officials said that they had conducted full assessments at 18
           critical facilities and revisited these facilities a year later to
           determine progress since the assessment. At the time,
           approximately 16 percent of recommended mitigation items had been
           completed, were in progress, or had been planned for. VA officials
           said they are finalizing a database and software that would
           facilitate the tracking of facilities' responses to assessment
           recommendations. The officials said that they expect to roll out
           the database and software within a few months.

           Besides conducting security assessments, organizations can
           mitigate risk by testing their facility protection
           countermeasures. Like FPS, VA conducts inspections and tests to
           evaluate compliance with security policies and procedures and to
           help ensure that adequate levels of protection are employed. In
           some instances, such as in the VA headquarters building,
           inspections can include simulated attempts to gain unauthorized
           access to a building or to smuggle fake weapons into a building.42
           For example, within VA, scenario-based tests that are derived from
           emerging security threats are commonly used to assess police
           officers' knowledge of, and compliance with, policies and
           procedures and to evaluate preparedness in the event of an attack.
           Earlier in this report, we noted that FPS has developed a
           performance measure using similar tests in order to assess the
           effectiveness of security countermeasures, such as contract
           security guards, in mitigating risk. In addition, both VA and FPS
           conduct biannual inspections of compliance with standards and
           policies, including for physical security.

           Such measurable activity could enable the measurement of program
           outcomes, including changes in the number of unauthorized building
           entries or the number of weapons and other prohibited items
           detected as part of facility intrusion tests. Although VA
           officials told us they had not developed performance measures, we
           believe they have valuable data that can be used to measure the
           overall effectiveness of the agency's facility protection program.
           For VA, security assessments and testing activity provide useful
           feedback on how well security measures have operated and whether
           they continue to be appropriate for the future. Further, these
           evaluations could form the basis for overall evaluations of VA's
           building security program and could provide data for performance
           measurement initiatives.

           While performance measures have been used to monitor many federal
           programs, little has been done to apply performance measurement to
           physical security programs-a complex and challenging undertaking,
           since outcomes may not be quickly achieved or readily observable.
           Although we found that physical security performance measurement
           is a challenge for many organizations in the public and private
           sector, we found that the information technology (IT) security
           area has performance measurement initiatives under way. Similar to
           facility protection, IT security has been a considerable concern
           in large part because computer systems are vital to many of our
           nation's critical operations and infrastructure. The dependency on
           these systems prompted a number of congressional actions,
           including various mandates for agencies to implement security
           controls to protect information systems within the federal
           government. In compliance with these federal requirements,
           agencies must demonstrate their progress in meeting requisite
           information security requirements and report on their actual level
           of performance based on the results of annual program reviews.

           In its role as a leader on technology issues, the National
           Institute of Standards and Technology (NIST), a subagency within
           the Department of Commerce, issued a report in 2003-Security
           Metrics Guide for Information Technology Systems-to provide
           guidance on how an organization can use performance measures to
           determine the adequacy of in-place security controls, policies,
           and procedures intended to mitigate security risks.43 More
           specifically, the report provides an approach that helps managers
           decide where to invest additional security protection resources or
           how to identify and evaluate controls that are not effective. The
           guidance is the culmination of several efforts to identify a
           suitable method for measuring security and supplemented ongoing
           initiatives by OMB to help agencies develop workable measures of
           job and program performance that would hold federal employees
           accountable for their IT security responsibilities. In addition to
           providing practical examples of security performance measures that
           can be readily used or modified to meet agency-specific needs, the
           report provides a detailed description of how performance
           measurement is being approached in the IT security area and
           addresses the following areas: (1) the roles and responsibilities
           of agency staff at all levels, (2) the benefits of using
           performance measures, and (3) an overview of the performance
           measures development and implementation process.

           The NIST report advocates the use of measurable performance
           measures based on IT security performance goals and objectives. In
           turn, the report describes performance measures as tools designed
           to facilitate decision making and improve performance and
           accountability through the collection, analysis, and reporting of
           relevant performance-related data. NIST describes three types of
           performance measures-implementation, efficiency and effectiveness,
           and impact-that can be used to measure progress (see table 5).
           Although NIST uses different terminology to describe the three
           types of performance measures, they are similar to the output and
           outcome measures that we have advocated for use in monitoring and
           reporting program accomplishments. The NIST report cautions that
           the type of performance measures that can realistically be
           obtained and used for performance improvement depends on the
           maturity of the security program. According to NIST, in the early
           stages of establishing a security program, the focus tends to be
           on developing security policies and procedures, and beginning to
           ensure that security controls are implemented. In such an
           environment, an appropriate performance measure would be one that
           focuses on implementation, such as the percentage of information
           systems with approved security plans. In contrast, a more mature
           security program may evolve to measure the efficiency and
           effectiveness of security controls and the impact of these
           controls on the organization's mission. In such cases, the
           performance measures may concentrate on the evidence and results
           of testing.

39Inspection Service officials told us that they chose 800 as the
threshold score because they wanted to further review the security of the
top 10 percent of the most vulnerable facilities. When this performance
measure was implemented, the top 10 percent of most vulnerable facilities
scored above 800. While this threshold remains the same today, the
threshold score may decrease or increase over time due to implementation
of countermeasures and changes in risk elements. To date, the Inspection
Service has decided not to change the threshold score in order to keep the
scoring methodology consistent.

40In its Strategic Transformation Plan 2006-2010, USPS has identified four
strategic goals: (1) generate revenue; (2) reduce costs; (3) achieve
results with a customer-focused, performance-based culture; and (4)
improve service.

  Case Example: Department of Veterans Affairs

41VA officials noted that the majority of the space occupied by VA's
Veterans Benefit Administration is in GSA-held buildings. As such, FPS is
responsible for security at these facilities.

Federal Guidance for Developing and Using Performance Measures Exists for IT
Security, but Not for Physical Security

42VA officials noted that most Veterans Health Administration buildings
are designed for maximum public access and therefore do not have
magnetometers or metal detectors, so such tests are not conducted in those
facilities. In addition, many Veterans Benefit Administration facilities
are in GSA buildings, so FPS is responsible for providing security and
conducting related tests.

43National Institute of Standards and Technology, Security Metrics Guide
for Information Technology Systems, NIST Special Publication 800-55 (July
2003).

Table 5: Types of Information Technology Security Performance Measures
Described by NIST

Type of measure Performance measure             Purpose                    
Implementation  Percentage of systems with      Assess the extent to which 
                   approved security plans and the security plans and         
                   percentage of systems with      password policies have     
                   password policies configured as been documented and        
                   required                        implemented to support the 
                                                   security program           
Efficiency and  Percentage of crackable         Evaluate the results of    
effectiveness   passwords within a predefined   security controls that     
                   time threshold                  have been implemented;     
                                                   validate whether security  
                                                   controls, as described in  
                                                   the security plan, are     
                                                   effective in protecting    
                                                   the organization's assets  
Impact          Quantify incidents by type      Measure the impact of      
                   (e.g., root compromise,         training on security       
                   password compromise, malicious  
                   code, denial of service) and    
                   correlate incident data with    
                   the percentage of trained users 
                   and system administrators       

Source: NIST.

The guidance goes beyond extolling the virtues of using performance
measures and illustrates the place of IT security within a larger
organizational context, provides a roadmap for how to develop and
implement a performance measurement program, and includes practical
examples of performance measures. According to NIST, the performance
measures that are ultimately selected can be useful not only for measuring
performance, identifying causes of unsatisfactory measurements, and
pinpointing improvement areas, but also for facilitating continuous policy
implementation, effecting security policy changes, and redefining goals
and objectives. NIST notes that successful implementation of a security
performance measurement program can also assist agencies in meeting OMB's
annual requirements to report the status of agency IT security programs.
In addition to providing examples of performance measures, some of which
are required by OMB, the report also includes a standardized template that
describes the various data elements that should be documented (see fig.
5). The data elements include:

           o  Performance goal: States the desired results of implementing
           security control objectives that are measured by the metric.
           o  Performance objective: States the actions that are required to
           accomplish the performance goal.
           o  Metric: Defines the metric by describing the quantitative
           measurements it provides.
           o  Purpose: Describes the overall functionality obtained by
           collecting the metric; includes whether a metric will be used for
           internal performance measurement or for external reporting, what
           insights are hoped to be gained from the metric, and whether
           regulatory or legal lessons exist for collecting a specific metric
           if applicable.
           o  Implementation evidence: Includes indirect indicators that
           validate that the activity is being performed and causation
           factors that may point to the causes of unsatisfactory results for
           a specific metric.
           o  Frequency: Establishes time periods for collecting data that is
           used for measuring changes over time.
           o  Formula: Describes the calculation to be performed that results
           in a numeric expression of the metric.
           o  Data source: Identifies the location of the data to be used in
           calculating the metric (e.g., databases, tracking tools,
           organizations, or specific roles within the organization that can
           provide required information).
           o  Indicators: Provide information about the meaning of the metric
           and its performance trend; state the performance target and
           indicate what trends would be considered positive in relation to
           the performance target.

           The NIST report notes that the universe of possible performance
           measures, based on policies and procedures in place in the
           organization, will be quite substantial and that the final
           performance measurement set selected for initial implementation
           should relate to high-priority areas, use data that can be
           realistically obtained, and measure processes that already exist
           and are relatively stable. The guidance further states that
           performance measures can be developed and selected using a phased
           approach. This approach identifies short-, mid-, and long-term
           measures where the time frame in which these measures are
           implemented depends on a combination of system-level
           effectiveness, performance measure priority, data availability,
           and process stability. The NIST report also notes that, once
           applicable performance measures have been identified, they should
           be documented using a standardized template (see figure 5).
           Standardizing the reporting process is particularly useful in
           cases where the reporting process within an organization is
           inconsistent. Such practices, among others, can help ensure the
           success of a performance measurement program.

Figure 5: Sample Standardized Performance Measurement Data Form

  Federal Agencies Have Received Minimal Guidance on Using Performance
  Measurement for Facility Protection Programs

We have previously reported that, at the agencywide level, agencies face
obstacles in developing meaningful, outcome-oriented performance goals and
in collecting data that can be used to assess the true impact of facility
protection efforts. GPRA emphasizes measuring the results of products and
services delivered by a federal program (i.e., outcomes). For programs
that have readily observable results or outcomes, performance measurement
may provide sufficient information to evaluate the effectiveness of
facility protection efforts. Yet in some programs, such as facility
protection, outcomes are not quickly achieved or readily observable, or
their relationship to the program is often not clearly defined. In such
cases, more in-depth program evaluations, in addition to performance
measurement, may be needed to examine the extent to which a program is
achieving its objectives.

While federal agencies have made some progress developing performance
measures for facility protection, we noted that the emphasis is on using
output measures that monitor program activity rather than outcome measures
that assess the overall impact of program activity. This lack of outcome
measures leaves agencies with insufficient information to determine
whether security activities are effective and to evaluate whether the
benefits of security investments justify their costs. We have previously
reported that various security program outputs-such as conducting
patrols-may have contributed to improved security, but that using them as
performance measures may not systematically target areas of higher risk
and may not result in the most effective use of resources, because these
measures are not pointed toward outcomes. Such output measures do not
provide an indication of what these activities are accomplishing. By
contrast, outcome measures that are clearly tied to results would indicate
the extent of progress made and help identify the security gaps that still
remain.44 Without more information on security program outcomes, agencies
do not know the extent to which security enhancements have improved
security or reduced federal facilities' vulnerability to acts of terrorism
or other forms of violence. In addition, there is some inconsistency in
the types of activities that are being monitored and used as indicators of
an agency's progress in fulfilling its facility protection
responsibilities. If agencies use inconsistent approaches to performance
measurement, decision makers could be at risk of having incomparable
performance information to determine funding priorities within and across
agencies.

44 GAO-06-91 .

Echoing what organizations outside the U.S. federal government told us,
some agency security officials said it was challenging to measure the
impact that various approaches have on actually improving security. Some
agency officials also noted that resources for performance measurement
initiatives were scarce. Additionally, the availability of information
needed for applying performance measurement to facility protection is
somewhat limited. More generally, with the exception of DHS, the agencies
that we reviewed do not view security as their primary mission, and some
agencies are faced with competing demands for limited resources to
accomplish broader agency goals. In such an environment, security must be
integrated using scarce resources.

In spite of the inherent difficulty in measuring facility protection
performance, and the considerable emphasis on doing so, agencies have
minimal guidance on how to accomplish this. There is, however, broad
guidance for the protection of critical infrastructures, which includes
government facilities. Using a risk-based approach, the Draft National
Infrastructure Protection Plan (NIPP) was developed to provide an
integrated, comprehensive approach to addressing physical, cyber, and
human threats and vulnerabilities.45 As part of the NIPP, DHS officials
have provided guidance and collected information on core performance
measures-which are common measures that can be broadly applied to all
protection programs for critical infrastructures and key assets. These
measures are mostly process/input and output oriented, and DHS officials
noted that they hope to develop outcome measures as the program matures.
The NIPP, however, does not provide or collect information on specific
performance measures related to the protection of federal facilities.
Rather, it notes that FPS-the agency assigned responsibility for
implementing the NIPP framework and guidance in the government facilities
sector-will develop such performance measures. Separately, OMB issued a
memorandum in June 2004 that reported it was working with agencies on
initiatives related to physical security reporting requirements noted in
Homeland Security Presidential Directive Number 7 (HSPD-7).46 The
memorandum instructed each agency to disclose the performance measures it
had designed and implemented to measure outputs and outcomes. However, OMB
did not provide specific guidance or standards and instead directed
agencies to use DHS guidance-related to the NIPP-that does not specify
measures for facility protection.

45DHS released the first Draft NIPP for public comment in November 2005.
In January 2006, DHS released a revised Draft NIPP that incorporated some
of the comments it had already received.

46As mentioned earlier, HSPD-7 establishes a national policy for federal
departments and agencies to identify and prioritize U.S. critical
infrastructures and key assets so that they can be protected from
terrorist attacks.

By contrast, the IT security performance measurement guidance issued by
NIST includes information on: (1) clearly defining roles and
responsibilities for relevant stakeholders; (2) establishing security
goals and objectives; (3) identifying and implementing performance
measures and performance targets; and (4) using measures that are unique
to IT security to assess the impact of IT security efforts. One security
official from the gaming industry said that IT security performance was
somewhat easier to evaluate than physical security performance because it
is possible to directly monitor the number of attempted IT security
breaches. A foreign government agency we interviewed is farther along in
developing standards and performance measures for IT security than for
physical security. In general, IT security approaches are slightly more
standardized than physical security because the field is newer than
physical security and because organizations had to work together to
prepare for possible complications in the year 2000 (Y2K). Despite such
differences between IT and physical security performance measurement, some
of the performance measurement guidance could be applicable to physical
security situations.

ISC is a body that addresses governmentwide security policy issues and,
like NIST, is well positioned to develop guidance and promote performance
measurement. Executive Order 12977 calls for ISC to play an oversight role
in implementing appropriate security measures in federal facilities and
taking actions that would enhance the quality and effectiveness of
security in federal facilities. As we reported in November 2004, ISC has
already made progress in coordinating the federal government's facility
protection efforts through activities such as developing security policies
and standards for leased space, improving information sharing, and
coordinating the development of a security database of all federal
facilities.47 The ISC Chair told us that he supports the use of
performance measurement as a means of strengthening federal facility
protection efforts.

47See GAO-05-49 .

                                  Conclusions

Given their competing priorities and limited security resources, U.S.
federal agencies could benefit from specific performance measurement
guidance and standards for facility protection to help them address the
challenges they face and help ensure that their physical security efforts
are achieving the desired results. While some of these agencies have
implemented performance measures to monitor their security programs'
outputs, fewer have developed outcome measures to assess the extent to
which security enhancements have improved security or reduced their
facilities' vulnerability to acts of terrorism or other forms of violence.
Without a means of comparing security effectiveness across facilities,
particularly program outcomes, the U.S. government is open to the risk of
either spending more money for less effective physical security or
investing in the wrong areas. The output measures that federal agencies
have developed provide an indication of what their security activities are
accomplishing but do not indicate the extent of progress made or help
identify the security gaps that still remain, as outcome measures would.
Fundamentally, performance measurement helps ensure accountability, since
it enables decision makers to isolate certain activities that are
hindering an agency's ability to achieve its strategic goals. Performance
measurement can also be used to prioritize security needs and justify
investment decisions so that an agency can maximize available resources.
Over time, a thorough performance measurement approach could allow the
federal government to manage the risks to federal facilities both within
and across agencies. Recognizing the unique nature of U.S. federal
agencies' missions, some uniformity in measuring performance in facility
protection efforts could facilitate comparisons across agencies.

Organizations outside of the U.S. government-including private-sector
entities as well as state, local, and foreign government agencies-have
developed and are using performance measures for facility protection, and
their knowledge and experience could be helpful to U.S. federal agencies
in developing and refining their own performance measures. Likewise,
because the application of performance measures to facility protection can
be challenging, many nonfederal organizations are looking to U.S.
government agencies for assistance and leadership. Some U.S. federal
agencies are already collecting data that could be used for measuring
security performance, and they currently have guidance for measuring
information technology security, but not physical security. The U.S.
federal government has provided guidance and collected information on a
set of common measures that can be broadly applied to all protection
programs for critical infrastructures and key assets, and agencies will be
required to report on additional security performance measures that are
sector-specific. With regard to federal facilities, the ISC, in serving as
the central coordinator for U.S. agencies' federal facility protection
efforts, is well positioned to develop and promote performance measurement
guidance and standards for physical security, and could look to
information technology security as a model to follow. In turn, it could
draw from examples of performance measurement we identified in the private
sector and foreign government agencies. Federal agencies could
subsequently follow the guidance and standards to evaluate their actions,
identify lessons learned, and develop strategies for overcoming any
challenges in developing and using performance measures for facility
protection. Because of the ever-changing nature of security threats and
new security technologies and countermeasures, such guidance and standards
would need to be periodically reviewed and updated. The development of
guidance and standards for facility protection could help ensure uniform
application of performance measurement so that the U.S. federal
government, particularly its largest real-property-holding agencies, would
be accountable for its facility protection programs and would be able to
demonstrate that security investments are producing a return, both within
and across agencies, in terms of better-protected facilities.

                      Recommendations for Executive Action

To ensure that useful information is available for making decisions about
the allocation of resources for, and the effectiveness of investments in,
the protection of federal facilities, we recommend that the Secretary of
Homeland Security direct the Chair of ISC to do the following:

           o  as part of ISC's efforts to support DHS in developing
           sector-specific performance measures for the security of federal
           government facilities, establish guidance and standards, with
           input from ISC member agencies, for measuring performance in
           facility protection-with a particular focus on developing outcome
           measures;
           o  communicate the established guidance and standards to the
           relevant federal agencies; and
           o  ensure that the guidance and standards are regularly reviewed
           and updated.

           We provided a draft of this report to DHS, GSA, USPS, VA, and
           Interior for their official review and comment. DHS concurred with
           the report's overall findings and recommendations. DHS comments
           are contained in appendix III. USPS and VA concurred with the
           report's findings. In addition, DHS and USPS provided separate
           technical comments, which we incorporated into the final report
           where appropriate. GSA notified us that they had no comments on
           this report.

           Interior, while generally agreeing with the report's findings,
           suggested that an agency-by-agency assessment of each federal
           agency's facility vulnerabilities would be more effective than a
           cross-agency facility protection performance measure. We agree
           that identifying and monitoring vulnerabilities is important, but
           believe that it is also important for decision makers to have
           comparable information about the relative security performance of
           facilities within an agency as well as across the federal
           government. Interior also expressed concern that a more public
           viewing of agency facility protection performance could reveal
           weaknesses or vulnerabilities that could be exploited. We agree
           that this could be a concern but leave the development of
           guidelines for using and protecting this information to ISC and
           its member agencies. Interior also provided technical comments,
           which we incorporated. Comments from Interior and our evaluation
           can be found in appendix IV.

           As agreed with your office, unless you publicly announce the
           contents of this report earlier, we plan no further distribution
           until 30 days from the report date. At that time, we will send
           copies of this report to other interested congressional committees
           and the Secretaries of the Interior, Homeland Security, and
           Veterans Affairs; the Administrator of GSA; and the Postmaster
           General of the U.S. Postal Service. We will also make copies
           available to others upon request. In addition, the report will be
           available at no charge on the GAO Web site at http://www.gao.gov.

           If you have any questions regarding this report, please contact me
           on (202) 512-2834 or at [email protected] . Contact points for
           our Offices of Congressional Relations and Public Affairs may be
           found on the last page of this report. GAO staff who made major
           contributions to this report are listed in appendix V.

           Sincerely yours,

           Mark L. Goldstein Director, Physical Infrastructure Issues

           Appendix I: Objectives, Scope, and Methodology
			  
			  The objectives of our report were (1) to identify examples of
           performance measures for facility protection being used by
           selected organizations outside of the federal government-including
           private-sector entities, state and local governments, and foreign
           governments; and (2) to determine the status of U.S. federal
           agencies' efforts to develop and use performance measures as part
           of their facility protection programs.

           To identify examples of performance measures for facility
           protection being used by selected organizations outside the
           federal government, we interviewed representatives from the
           private sector, U.S. state and local governments, and foreign
           governments. With respect to the private sector, we asked a number
           of umbrella organizations to identify industries that are likely
           to utilize performance measures for facility protection and known
           leaders in the security performance measurement area. These
           umbrella organizations included ASIS International, Real Estate
           Roundtable, Financial Services Roundtable, Financial Services
           Information Sharing and Analysis Committee, International Facility
           Management Association, and National Association of Industrial and
           Office Properties. GAO staff also attended the annual ASIS
           International Conference in 2005. Some of these entities stated
           that the gaming and finance industries would be the most
           appropriate to review, since these industries have invested
           significantly in the quality of their security efforts. As a
           result, we interviewed officials from four gaming entities and
           five major financial services organizations. To maintain the
           organizations' security and the confidentiality of proprietary
           information, we do not identify specific organizations in this
           report.

           For the gaming industry, a member of the Real Estate Roundtable
           provided a contact who was known to be active in physical security
           testing and performance measurement. This individual then arranged
           a joint interview for us with a number of gaming entities. Some of
           the representatives present at the interview were also members of
           the Las Vegas Security Chiefs Association or ASIS International
           Gaming and Wagering Council. The five financial services
           organizations we interviewed were selected because they (1) were
           considered to be leaders in their industry; (2) were recommended
           by others within the industry; (3) were members of ASIS
           International, the largest organization supporting security
           professionals; or (4) have had prior security concerns related to
           threats of terrorism.

           To determine if U.S. state and local governments have developed
           performance measures for facility protection, we attempted to
           contact 10 state and 10 local governments. For state governments,
           we selected the 10 states receiving the most funding from the
           Department of Homeland Security's (DHS) State Homeland Security
           Program grant in fiscal year 2005. For local governments, we
           selected the 10 local governments/urban areas receiving the most
           funding from DHS's Urban Areas Security Initiative grant in fiscal
           year 2005.1 Of the 20 state and local governments we attempted to
           contact, we were able to obtain information from officials from 17
           of them. While all 17 of these state and local governments were
           engaged in facility protection efforts, only a few had developed
           performance measures to evaluate the effectiveness of these
           efforts. Table 6 shows a listing of these state and local
           governments. The agencies we approached within each of the state
           and local governments were often, but not always, the agencies
           responsible for real property or policing/security. Some of the
           state and local governments we attempted to contact were also
           identified by the Government Accounting Standards Board as having
           performance measurement initiatives on a variety of their
           organizations, departments, and projects.

           Table 6: U.S. State and Local Governments Contacted

           Source: GAO.

           aFor the purposes of this report, Washington, D.C., was treated as
           a local government.

           For our work with foreign governments, we conducted international
           site visits in three foreign countries-Australia, Canada, and the
           United Kingdom-where we interviewed a number of government
           agencies and organizations about their use of performance measures
           for facility protection. (Table 7 shows a listing of each of these
           agencies.) We selected these three countries for site visits
           because they are known to have experience with threats of
           terrorism and because they have been identified by the Government
           Accounting Standards Board as having performance measurement
           initiatives, not necessarily for facility protection but for
           government initiatives in general. We also spoke with
           representatives from a number of other foreign governments. While
           these other governments have facility protection efforts in place,
           they said they did not use performance measures to assess the
           effectiveness of these efforts. Furthermore, officials from some
           of these countries told us that they look to the United States for
           guidance on a number of issues relating to facility protection,
           including how to measure effectiveness. For such reasons, these
           countries were not highlighted in this report.

           Table 7: Foreign Government Agencies and Organizations Visited

           Source: GAO.

           In addition to interviewing officials from the nonfederal entities
           identified above, we reviewed relevant documentation obtained from
           these organizations, previous GAO reports, and performance
           measurement and facility protection literature from ASIS
           International and other sources.

           For the second objective-to determine the status of U.S. federal
           agencies' efforts to develop and use performance measures as part
           of their facility protection programs-we interviewed selected
           officials from the major civilian real property holding agencies.
           These agencies include the General Services Administration (GSA),
           the United States Postal Service (USPS), the Department of
           Veterans Affairs (VA), and the Department of Interior (Interior).
           GSA acknowledged the need to measure the performance of facility
           protection efforts; however, for most facility protection issues,
           they defer to the Federal Protective Service (FPS) within DHS.
           Because FPS is responsible for protecting all GSA buildings, we
           also interviewed officials from FPS. For each of the selected
           federal agencies, we reviewed agency strategic and performance
           plans, security goals, performance reports, and other relevant
           documentation provided to us. We also interviewed the Executive
           Director of the Interagency Security Committee (ISC)-a DHS-led
           committee that is tasked with coordinating federal agencies'
           facility protection efforts. Finally, we reviewed a number of
           national strategies and presidential directives; previous GAO
           reports; and relevant reports by the Office of Management and
           Budget, the Congressional Budget Office, the Congressional
           Research Service, and other government entities. We also reviewed
           laws and authorities related to facility protection.

           It is important to note that the private-sector entities, U.S.
           state and local governments, and foreign governments selected for
           our review are not representative of the universe of such
           organizations. Furthermore, GAO has not evaluated the robustness
           and quality of the performance measures cited in this report.
           Rather, these measures are simply a compilation of what we have
           gathered from the nonfederal and federal entities we have
           interviewed. Additionally, the performance measures identified in
           this report may not include all performance measures relating to
           the protection of federal facilities. We used our judgment to
           classify the performance measures into process/input, output, and
           outcome measures according to our definitions, but these
           performance measures could be classified differently depending on
           the performance measurement goals or objectives used by an
           organization.

           Also, ISC has identified GAO as an associate member, which
           includes the ability to serve on ISC subcommittees. No GAO staff
           member, however, serves on any subcommittee. Furthermore, no GAO
           staff member actively participates in ISC meetings or contributes
           to decisions. Rather, GAO's role on ISC is only to observe
           proceedings and obtain ISC information distributed to the other
           ISC members. Because of GAO's observational role, our independence
           in making recommendations involving ISC and in completing this
           engagement was maintained.

           Officials from nonfederal and federal entities provided much of
           the information used in this report. In most cases where officials
           provided their views as representatives of their organizations, we
           corroborated the information with other officials or with
           documentation provided to us. We requested official comments on
           this report from DHS, GSA, USPS, VA, and Interior. Furthermore,
           when we used examples from the private sector, state and local
           governments, foreign governments, and the National Institute of
           Standards and Technology (NIST), we provided the respective entity
           an opportunity to review relevant portions of the report and offer
           comments, thus ensuring the validity of our reporting. We
           conducted site visits and interviews from July 2005 through
           January 2006. We conducted our work from May 2005 through April
           2006 in accordance with generally accepted government auditing
           standards.

           Appendix II: Examples of Performance Measures Used by Selected
			  Organizations outside of the Federal Government
			  
			  The performance measures below were provided by the selected
           organizations we interviewed outside of the federal government. We
           did not evaluate the quality of the performance measures, and we
           used our judgment to classify them according to the following
           definitions of performance measures:

           o  Output measures focus on the quantity of direct products and
           services a program delivers and the characteristics of those
           outputs, including efficiency, cost-effectiveness, timeliness,
           quality, and customer service.
           o  Outcome measures provide information on the results of the
           direct products and services a program has delivered.
           o  Process/input measures address the type or level of program
           activities an organization conducts and the resources used by the
           program.

           The performance measures could be classified differently depending
           on the performance measurement goals or objectives used by an
           organization.

           Source: GAO.

           Note: GAO analysis of data from state, local, and foreign
           government agencies and private-sector organizations.

           .

           The following are GAO's comments on Interior's letter dated May
           15, 2006.

           Appendix V: GAO Contact and Staff Acknowledgments
			  
			  GAO Contact
			  
			  Staff Acknowledgments
			  
			  Mark Goldstein (202) 512-2834 or [email protected]

           Other key contributors to this report were Roshni Dave, Tamera
           Dorland, Brandon Haller, Anne Izod, Jessica Lucas-Judy, Susan
           Michal-Smith, David Sausville, Scott Tessier, Candice Wright, and
           Dorothy Yee

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site ( www.gao.gov ). Each weekday,
           GAO posts newly released reports, testimony, and correspondence on
           its Web site. To have GAO e-mail you a list of newly posted
           products every afternoon, go to www.gao.gov and select "Subscribe
           to Updates."

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
           [email protected] Automated answering system: (800) 424-5454 or
           (202) 512-7470

           Gloria Jarmon, Managing Director, [email protected] (202) 512-4400
           U.S. Government Accountability Office, 441 G Street NW, Room 7125
           Washington, D.C. 20548

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548
			  
			  GAO Comments

                        1. Interior suggested that an agency-by-agency
                        assessment of each federal agency's facility
                        vulnerabilities would be more effective than a
                        cross-agency facility protection performance measure.
                        We agree that identifying vulnerabilities and
                        monitoring efforts to address those vulnerabilities
                        is a useful part of an agency's comprehensive
                        facility protection program. For example, the
                        Department of Veterans Affairs conducts vulnerability
                        assessments, and one Australian government agency we
                        interviewed monitors the effect of different security
                        investments on its facilities' risk ratings (which
                        typically involve threat and vulnerability factors).
                        However, we believe it is also important for decision
                        makers to have comparable information about the
                        relative security performance of facilities within an
                        agency, rather than just in one bureau or service, as
                        well as across the federal government. Such
                        information could help reduce the risk of spending
                        more money for less effective physical security or
                        investing in the wrong areas.
                        2. Interior expressed concern that a more public
                        viewing of agency facility protection performance
                        could reveal weaknesses or vulnerabilities that could
                        be exploited. We agree that this could be a concern,
                        but choose to leave the development of guidelines for
                        using and protecting such information to the
                        Interagency Security Committee and its member
                        agencies.

                       Agency Comments and Our Evaluation

1The State Homeland Security Program and Urban Areas Security Initiative
grants can be applied to a number of homeland security efforts, including
facility protection. See U.S. Department of Homeland Security, Fiscal Year
2005 Homeland Security Grant Program, Program Guidelines and Application
Kit.

Organization           Location              
U.S. state governments California            
                                                
                          Florida               
                                                
                          Georgia               
                                                
                          Illinois              
                                                
                          Michigan              
                                                
                          New Jersey            
                                                
                          New York              
                                                
                          Ohio                  
                                                
                          Pennsylvania          
                                                
                          Texas                 
U.S. local governments Boston, Mass.         
                                                
                          Detroit, Mich.        
                                                
                          Washington, D.C.a     
                                                
                          Los Angeles, Calif.   
                                                
                          New York, N.Y.        
                                                
                          Philadelphia, Pa.     
                                                
                          San Francisco, Calif. 

Location       Organization                                                
Australia      Airservices Australia                                       
                                                                              
                  Attorney-General's Department                               
                                                                              
                  Commonwealth Scientific and Industrial Research             
                  Organization                                                
                                                                              
                  Customs Service                                             
                                                                              
                  Department of Defence                                       
                                                                              
                  Department of Foreign Affairs and Trade                     
                                                                              
                  Federal Police                                              
                                                                              
                  National Audit Office                                       
                                                                              
                  Taxation Office                                             
Canada         Bank of Canada                                              
                                                                              
                  Corps of Commissionaires                                    
                                                                              
                  Department of National Defence                              
                                                                              
                  National Gallery                                            
                                                                              
                  Office of Auditor General                                   
                                                                              
                  Public Works and Government Services Canada                 
                                                                              
                  Royal Canadian Mounted Police                               
                                                                              
                  Treasury Board                                              
United Kingdom Cabinet Office                                              
                                                                              
                  Department for Transport                                    
                                                                              
                  Foreign and Commonwealth Office                             
                                                                              
                  Home Office                                                 
                                                                              
                  National Infrastructure Security Coordination Centre,       
                  Security Service                                            
                                                                              
                  National Security Advice Centre, Security Service           
                                                                              
                  Office for Civil Nuclear Security                           

Appendix II: Examples of Performance Measures Used by Selected
Organizations outside of the Federal Government Appendix II: Examples of
Performance Measures Used by Selected Organizations outside of the Federal
Government

Output                                                                     
Number of risk assessments performed                                       
                                                                              
New security projects                                                      
                                                                              
      o  Security checklist completed during planning stages                  
      o  Security officials consulted                                         
                                                                              
Number of security requests received                                       
                                                                              
      o  Security report requests                                             
      o  New access badge requests                                            
      o  Requests for changes to existing badges                              
                                                                              
Security clearance                                                         
                                                                              
      o  Number of background screenings completed                            
      o  Average time to process background screenings                        
      o  Average number of days to process security clearances                
      o  Number of overdue security clearances by more than 4 weeks           
      o  Cost per security clearance                                          
      o  Percentage of officers/contractors who hold sufficient level of      
      security clearance when compared to their duties                        
                                                                              
Alarm systems                                                              
                                                                              
      o  Responded to and cleared                                             
      o  Alarms with unique responses (i.e., alarms requiring guards to       
      respond in person)                                                      
      o  Failed to arm Number of police incidents/reports filed Number of     
      threats                                                                 
      o  Against employees                                                    
      o  Against facilities                                                   
                                                                              
Security incident reaction/response                                        
                                                                              
      o  Number of avoidable incidents detected                               
      o  All significant investigations completed within 45 days              
                                                                              
Compliance with security policies and standards                            
                                                                              
      o  Number of exceptions reviewed                                        
      o  Number of significant policy breaches                                
      o  Surveillance and communication systems are compliant with standards  
      o  Entry/access control systems are compliant with standards            
      o  Security staff are fulfilling their contract obligations             
                                                                              
Customer/client satisfaction                                               
                                                                              
      o  Staffing-training, professional appearance, professional behavior,   
      turnover rate, supervision                                              
      o  Security reporting-accuracy, timeliness, use of correct forms        
      o  Management-responsiveness, understanding of issues, availability,    
      number of personal contacts                                             
                                                                              
Timely delivery of security alerts and briefings                           
                                                                              
Percentage of alarms responded to within 20 minutes during nonpublic       
service hours                                                              
                                                                              
Increased attendance at training courses for security officers             
                                                                              
Number of new employees, contractors, and consultants who have not         
attended a security awareness session within 4 weeks of receiving their    
identification pass                                                        
                                                                              
Percentage of security guards in compliance with licensing standards       
within a 7-day period                                                      
                                                                              
All scheduled audit and compliance reports completed in 14 days            
Outcome                                                                    
Change in the total number of security-related incidents                   
                                                                              
      o  Accident                                                             
      o  Assault                                                              
      o  Burglary                                                             
      o  Organization assets                                                  
      o  Personal assets                                                      
      o  Drugs/Alcohol                                                        
      o  Extortion                                                            
      o  Fire                                                                 
      o  Fraud Referral                                                       
      o  Harassment                                                           
      o  Larceny/Theft                                                        
      o  Malicious damage                                                     
      o  Public disorder                                                      
      o  Robbery                                                              
      o  Suspicious activity                                                  
      o  Terrorism                                                            
      o  Vandalism                                                            
      o  Workplace violence                                                   
                                                                              
Evidence of damage to building and facilities                              
                                                                              
Evidence of harm to staff or tenants                                       
                                                                              
Change in risk rating resulting from countermeasures deployed              
                                                                              
Security policies and practices receive favorable comment from security    
audit program                                                              
                                                                              
Agency stakeholders view agency as a safe custodian of allocated resources 
and assets                                                                 
Process/Input                                                              
Number of facilities being protected (including types and locations)       
                                                                              
Number of security staff                                                   
                                                                              
Number of security guards/security escorts                                 
                                                                              
Personal security arrangements for after-hours entry/access                
                                                                              
Perimeter security                                                         
                                                                              
      o  Assessment of entry/exit points                                      
      o  Serviceability of perimeter security equipment (locks, door frames,  
      security signs)                                                         
      o  Sufficiency of perimeter lighting                                    
      o  Presence of obstructions, waste containers/material, combustibles,   
      other risk factors                                                      
      o  Evidence of vandalism, malicious damage, or other criminal activity  
      o  Maintenance schedules                                                
                                                                              
Number of security clearances undertaken Number of training courses and    
drills conducted                                                           
                                                                              
Security threats and general risks discussed at management forum and       
disseminated to all levels of agency staff Security spending per square    
foot                                                                       

Appendix III: Comments from the Department of Homeland Security Appendix
III: Comments from the Department of Homeland Security

Note: GAO comments supplementing those in the report text appear at the
end of this appendix.

Appendix IV: Comments from the Department of the Interior Appendix IV:
Comments from the Department of the Interior

See comment 1.

See p. 34.

References to BLM have been deleted.

See comment 2.

                                
(543129)

GAO's Mission

Obtaining Copies of GAO Reports and Testimony

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs

www.gao.gov/cgi-bin/getrpt? GAO-06-612 .

To view the full product, including the scope

and methodology, click on the link above.

For more information, contact Mark Goldstein at (202) 512-2834 or
[email protected].

Highlights of GAO-06-612 , a report to the Chairman, Committee on
Government Reform, House of Representatives

May 2006

HOMELAND SECURITY

Guidance and Standards Are Needed for Measuring the Effectiveness of
Agencies' Facility Protection Efforts

The protection of U.S. federal facilities has become an important concern
due to the ongoing threat of terrorism. The General Services
Administration (GSA), U.S. Postal Service (USPS), and the Departments of
Veterans Affairs (VA) and Interior (Interior) hold the most domestic,
nonmilitary property. Additionally, the Department of Homeland Security
(DHS) is responsible for the protection of GSA facilities. DHS chairs the
Interagency Security Committee (ISC), which is tasked with coordinating
federal agencies' facility protection efforts. The need to better protect
federal facilities, as well as federal budget constraints, have prompted
the need for these agencies to measure the performance of their facility
protection efforts. GAO's objectives were (1) to identify examples of
performance measures for facility protection being used by selected
organizations outside of the federal government; and (2) to determine the
status of U.S. federal agencies' efforts to develop and use performance
measures as a part of their facility protection programs.

What GAO Recommends

GAO is recommending that the Secretary of DHS direct ISC to establish
guidance and standards for measuring performance in federal government
facility protection. DHS agreed with the findings and recommendations in
this report.

GAO found a range of examples of performance measures that organizations
outside the U.S. government-including private-sector entities, state and
local governments, and foreign government agencies-have developed that,
collectively, indicate whether facility protection efforts are achieving
results (see figure below). These organizations use security-related
performance measures to help improve security, make decisions about risk
management and resource allocation, and hold employees accountable for
whether a program meets its security goals and objectives. However, many
of the organizations said that developing and using these measures can be
challenging and that they look to the U.S. government for assistance and
leadership in developing standards and guidance for facility protection.

Performance Measurement Types, Examples, Uses, and Results

Note: Output measures focus on the direct product/services delivered by a
program. Outcome measures provide information on the results of
products/services.

We found that some bureaus and services within DHS (for GSA properties),
USPS, and Interior are using security performance measures, while VA and
other bureaus and services within the three agencies collect data that
could be used to measure security performance. Agencies that have
performance measures use them to ensure adequate protection at individual
facilities, make risk management decisions, and evaluate program
effectiveness. However, agencies face challenges-similar to those cited by
nonfederal entities-in further developing and using security performance
measures. Currently, there is no governmentwide guidance or standards on
measuring facility protection performance to help federal agencies address
these challenges. This differs from information technology security, where
agencies have detailed, governmentwide guidance for developing and using
performance measures. Without effective performance measurement data,
decision makers may have insufficient information to evaluate whether
their investments have improved security or reduced federal facilities'
vulnerability to acts of terrorism or other forms of violence. ISC is
uniquely positioned to develop and disseminate guidance and standards for
measuring the performance of federal government facility protection
efforts.
*** End of document. ***