    Homeland Security: Guidance and Standards Are Needed for
    Measuring the Effectiveness of Agencies' Facility Protection
    Efforts (31-MAY-06, GAO-06-612).

    The need to better protect federal facilities, coupled with
    federal budget constraints and the increased scrutiny of homeland
    security funding and programs, has prompted the need for U.S.
    agencies to measure the performance of their facility protection
    efforts. In this environment, it is important for these agencies
    to ensure that investments in facility protection are providing
    adequate returns in terms of better protecting real property
    assets against terrorism. In addition, the U.S. government's
    national strategy, Presidential directive, and guidance on
    protecting critical infrastructures--including facilities--have
    identified the use of performance measurement as a key means of
    assessing the effectiveness of protection programs. Given that
    protection of critical infrastructures is an important issue for
    organizations outside of the federal government as well, it is
    beneficial to look to the experiences of these organizations to
    identify lessons learned. As such, our objectives for this review
    were (1) to identify examples of performance measures for
    facility protection being used by selected organizations outside
    of the federal government--including private-sector entities,
    state and local governments, and foreign governments, and (2) to
    determine the status of U.S. federal agencies' efforts to develop
    and use performance measures as part of their facility protection
    programs.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-612
        ACCNO:   A54905
      TITLE:     Homeland Security: Guidance and Standards Are Needed for
    Measuring the Effectiveness of Agencies' Facility Protection
    Efforts
         DATE:   05/31/2006
      SUBJECT:   Antiterrorism
    	     Facility management
    	     Facility security
    	     Federal facilities
    	     Federal property
    	     Homeland security
    	     Human capital management
    	     Interagency relations
    	     Lessons learned
    	     Performance measures
    	     Physical security
    	     Real property
    	     Risk management
    	     Strategic planning
    	     National Infrastructure Protection Plan

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-612

         * Results in Brief
         * Background
         * Organizations outside of the U.S. Government Use Security Pe
              * Selected Organizations Use a Range of Output, Outcome, and P
                   * Case Example: A Financial Service Organization's Performance
              * Security Officials Use Performance Measure Results for Risk
                   * Case Example: An Australian Agency's Risk Model
              * Performance Measures Can Be Used to Hold Security Officials
                   * Case Example: The District of Columbia's Alignment of Securi
                   * Departmental Security Officers and Individual Accountability
                   * Case Example: Individual Accountability in Two Financial Ser
                   * Case Example: An Australian Agency's Security Certification
              * Organizations Cited Challenges in Developing and Using Perfo
         * U.S. Agencies Have Made Some Progress in Developing and Usin
              * Agencies Use Output Measures and Some Outcome Measures to In
                   * Case Example: DHS's Federal Protective Service in GSA Facili
                   * Case Example: Interior's Bureau of Reclamation and National
                   * Case Example: USPS Inspection Service
                   * Case Example: Department of Veterans Affairs
              * Federal Guidance for Developing and Using Performance Measur
                   * Federal Agencies Have Received Minimal Guidance on Using Per
         * Conclusions
         * Recommendations for Executive Action
         * Agency Comments and Our Evaluation
         * GAO Comments
         * GAO Contact
         * Staff Acknowledgments
         * GAO's Mission
         * Obtaining Copies of GAO Reports and Testimony
              * Order by Mail or Phone
         * To Report Fraud, Waste, and Abuse in Federal Programs
         * Congressional Relations
         * Public Affairs

    Report to the Chairman, Committee on Government Reform, House of
    Representatives

    United States Government Accountability Office

    GAO

    May 2006

    HOMELAND SECURITY

    Guidance and Standards Are Needed for Measuring the Effectiveness of
    Agencies' Facility Protection Efforts

    GAO-06-612

    Contents

    Letter 1

    Results in Brief 4
    Background 6
    Organizations outside of the U.S. Government Use Security Performance
    Measures to Enhance Decision Making and Help Ensure Accountability 12
    U.S. Agencies Have Made Some Progress in Developing and Using Performance
    Measures for Facility Protection Programs, but Lack Guidance and Standards
    25
    Conclusions 48
    Recommendations for Executive Action 49
    Agency Comments and Our Evaluation 49
    Appendix I Objectives, Scope, and Methodology 52
    Appendix II Examples of Performance Measures Used by Selected
    Organizations outside of the Federal Government 58
    Appendix III Comments from the Department of Homeland Security 62
    Appendix IV Comments from the Department of the Interior 64
    GAO Comments 67
    Appendix V GAO Contact and Staff Acknowledgments 68

    Tables

    Table 1: Examples of Performance Measures for Facility Protection 13
    Table 2: FPS's Performance Measures for Facility Protection 26
    Table 3: BOR's Performance Measures for Facility Protection 32
    Table 4: Inspection Service's Performance Measure for Facility Protection
    34
    Table 5: Types of Information Technology Security Performance Measures
    Described by NIST 41
    Table 6: U.S. State and Local Governments Contacted 54
    Table 7: Foreign Government Agencies and Organizations Visited 55

    Figures

    Figure 1: Smart Card Access Portals at a Federal Building Entrance 9
    Figure 2: Linkages between District of Columbia Strategic Goals and
    Performance Measures for Facility Protection 20
    Figure 3: Linkages between DHS Mission and FPS Performance Measures for
    Facility Protection 29
    Figure 4: Linkages between USPS Inspection Service Strategic Goals and
    Performance Measure for Facility Protection 37
    Figure 5: Sample Standardized Performance Measurement Data Form 44
    Abbreviations

    BOR Bureau of Reclamation

    DHS Department of Homeland Security

    DSO departmental security officer

    FPS Federal Protective Service

    GPRA Government Performance and Results Act of 1993

    GSA General Services Administration

    HSPD-7 Homeland Security Presidential Directive Number 7

    ICE Immigration and Customs Enforcement

    ISC Interagency Security Committee

    IT information technology

    NIPP National Infrastructure Protection Plan

    NIST National Institute of Standards and Technology

    NM&I National Monuments and Icons Assessment Methodology

    OLES Office of Law Enforcement and Security

    OMB Office of Management and Budget

    PART Program Assessment Rating Tool

    USPS United States Postal Service

    VA Department of Veterans Affairs

    This is a work of the U.S. government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed in
    its entirety without further permission from GAO. However, because this
    work may contain copyrighted images or other material, permission from the
    copyright holder may be necessary if you wish to reproduce this material
    separately.

    United States Government Accountability Office

    Washington, DC 20548

    May 31, 2006

    The Honorable Tom Davis Chairman Committee on Government Reform House of
    Representatives

    Dear Mr. Chairman:

    The threat of terrorism has increased the emphasis on physical security
    for federal real property assets since the 1995 bombing of the Alfred P.
    Murrah Federal Building in Oklahoma City; the 1998 embassy bombings in
    Africa; the September 11, 2001, attacks on the World Trade Center and the
    Pentagon; and the anthrax attacks in the fall of 2001. The federal
    government owns or leases an estimated 3.2 billion square feet of space
    within the United States in more than 450,000 buildings, which are
    regularly accessed by millions of federal employees, contractors, and
    citizens. Approximately 42 percent of this square footage is nonmilitary
    property, and a majority of this is under the control or custody of the
    General Services Administration (GSA), the United States Postal Service
    (USPS), the Department of Veterans Affairs (VA), and the Department of the
    Interior (Interior).1 Under the Homeland Security Act of 2002, the Federal
    Protective Service (FPS), which protects GSA properties, was transferred
    to the Department of Homeland Security (DHS). For agencies that aim to
    ensure public access to their assets, protecting nonmilitary real property
    assets can be complex and contentious because of the need to strike a
    balance between public access and security.2 Federal agencies face
    additional security-related challenges, such as securing federally leased
    space and addressing conflicts with state, local, or private entities that
    also have jurisdiction over, or input regarding, physical security
    enhancements. The challenge of protecting federal facilities against the
    threat of terrorism was a major reason GAO designated federal real
    property as a high-risk area in January 2003.3

    1GSA, Overview of the United States Government's Owned and Leased Real
    Property: Federal Real Property Profile As of September 30, 2004
    (Washington, D.C.). This property includes government-owned and leased
    space.

    2GAO, Homeland Security: Actions Needed to Better Protect National Icons
    and Federal Office Buildings from Terrorism, GAO-05-790 (Washington, D.C.:
    June 24, 2005), p. 1.

    Although FPS is primarily responsible for protecting GSA properties, it
    also has responsibility for broader efforts across the federal government
    to enhance the protection of critical facilities and works closely with
    the Interagency Security Committee (ISC) on these issues. The ISC, which
    DHS chairs, is tasked with coordinating federal agencies' facility
    protection efforts, developing protection standards, and overseeing
    implementation of those standards.4 In November 2004, we recommended that
    ISC develop an action plan for fulfilling its responsibilities and
    establish a set of key practices for facility protection.5 We identified
    several key practices in facility protection, which included using risk
    management to allocate resources;6 leveraging security technology;
    coordinating protection efforts and sharing information; realigning real
    property assets to an agency's mission, thereby reducing vulnerabilities;
    strategically managing human capital; and measuring program performance
    and testing security initiatives.7 With regard to measuring performance,
    performance measures can be classified as output measures, which focus on
    the quantity of direct products and services a program delivers; outcome
    measures, which offer information on the results of the direct products
    and services a program has delivered; or process/input measures, which
    address the type or level of program activity an organization conducts and
    the resources used by the program. Outcome measures are particularly
    useful because they indicate what program activities are accomplishing. At
    the time of our November 2004 report, agencies were only in the early
    stages of implementing security performance measures.

    3GAO, High-Risk Series: Federal Real Property, GAO-03-122 (Washington,
    D.C.: January 2003).

    4In this report, facility protection denotes the protection of not only
    the facilities but also the people, equipment, and other assets within
    them. Additionally, this report focuses largely on protecting facilities
    from threats and acts of terrorism. However, it is important to note that
    facilities are also vulnerable to other types of hazards, such as natural
    disasters and workplace violence, and information in this report may be
    applicable to those hazards as well.

    5GAO, Homeland Security: Further Actions Needed to Coordinate Federal
    Agencies' Facility Protection Efforts and Promote Key Practices, GAO-05-49
    (Washington, D.C.: Nov. 30, 2004). Since the time of that report, the ISC
    Chair noted that he is in the process of creating and establishing an
    action plan with the ISC membership, although little progress has been
    made because of limited resources. The Chair anticipates that this action
    plan, which will articulate a roadmap for the ISC to follow in meeting its
    responsibilities, will incorporate portions of the material and related
    concepts contained in GAO reports.

    6Risk management is a tool for assessing risks, evaluating alternatives,
    making decisions, and implementing and monitoring protective measures.
    More specifically, risk can be calculated as follows: risk = (threat x
    vulnerability) x consequence. Threat is the probability that a specific
    type of attack will be initiated against a particular target or class of
    targets. The vulnerability of an asset is the probability that a
    particular attempted attack will succeed against a particular target or
    class of targets. It is usually measured against some set of standards,
    such as availability/predictability, accessibility, countermeasures in
    place, and target hardness (the material construction characteristics of
    the asset). The consequence of a terrorist attack is characterized as the
    expected worst case or worst reasonable adverse impact of a successful
    attack.

    The need to better protect federal facilities, coupled with federal budget
    constraints and the increased scrutiny of homeland security funding and
    programs, has prompted the need for U.S. agencies to measure the
    performance of their facility protection efforts. In this environment, it
    is important for these agencies to ensure that investments in facility
    protection are providing adequate returns in terms of better protecting
    real property assets against terrorism. In addition, the U.S. government's
    national strategy, Presidential directive, and guidance on protecting
    critical infrastructures-including facilities-have identified the use of
    performance measurement as a key means of assessing the effectiveness of
    protection programs. Given that protection of critical infrastructures is
    an important issue for organizations outside of the federal government as
    well, it is beneficial to look to the experiences of these organizations
    to identify lessons learned. As such, our objectives for this review were
    (1) to identify examples of performance measures for facility protection
    being used by selected organizations outside of the federal
    government-including private-sector entities, state and local governments,
    and foreign governments, and (2) to determine the status of U.S. federal
    agencies' efforts to develop and use performance measures as part of their
    facility protection programs. To address the first objective, we
    interviewed private-sector representatives from four entities in the
    gaming industry and from five major financial services entities, because
    these industries were identified as having invested in security and likely
    to have developed performance measures. We also interviewed officials from
    17 of the 20 state and local governments that received the most funding
    from two security-related DHS grant programs in fiscal year 2005.8
    Finally, we interviewed government officials from multiple agencies in
    Australia, Canada, and the United Kingdom, because these countries have
    experience with threats of terrorism and have performance measurement
    initiatives. We also reviewed relevant documents we obtained from these
    organizations, related GAO reports, and literature on performance
    measurement. To address the second objective, we interviewed federal
    officials from DHS, GSA, USPS, VA, and Interior-the agencies that hold, or
    are responsible for the security of, the majority of the domestic,
    nonmilitary property. We also reviewed pertinent documents and policies
    obtained from these agencies, in addition to related laws and directives.
    A detailed discussion of our scope and methodology, including more
    information on how we selected the organizations we contacted, is
    contained in appendix I. We conducted our work between June 2005 and April
    2006 in accordance with generally accepted government auditing standards.

    7Performance measurement is the ongoing monitoring and reporting of
    program accomplishments, particularly progress toward preestablished
    goals. It is typically conducted by program or agency management.

                                    Results in Brief

    We found a range of examples of performance measures that organizations
    outside the U.S. government, including private-sector firms, state and
    local governments, and foreign government agencies, use to help improve
    the security of facilities, inform risk-management and resource-allocation
    decisions, and hold security officials and others in their organizations
    accountable for security performance. These included output measures, such
    as the average time to process background screenings, and outcome
    measures, such as the change in the total number of security incidents
    relating to thefts, vandalism, and acts of terrorism. For example, an
    agency in Australia monitors an outcome measure concerning the impact of
    additional security expenditures on a facility's risk rating, while
    controlling for existing security enhancements that mitigate the risk,
    such as the number of guard patrols and the adequacy of access control
    systems (e.g., electronic locks). In another example, each business line
    in one financial services organization conducts security compliance
    reviews of its facilities, including confirming the presence of required
    key security equipment and determining whether staff are following
    security policies. Senior security officials review the results to
    determine where problems exist and hold each business manager accountable
    for addressing them. Despite some organizations' use of these measures,
    less than one-quarter of the organizations we contacted had developed
    performance measures for facility protection, and there was widespread
    acknowledgement among the organizations that effectiveness in facility
    protection is challenging to measure. For example, security officials do
    not necessarily know whether a potential security threat or incident has
    been prevented, even after perceived security weaknesses have been
    addressed. Since security is so challenging to measure, some of the
    organizations that we interviewed told us that they rely on U.S. federal
    agencies for support and leadership in developing security standards and
    performance measures, and one foreign government agency said it was
    interested in developing guidance for security performance measurement but
    was looking to U.S. federal agencies for assistance in this area.

    8Of the 20 state and local governments we attempted to contact, we were
    able to obtain information from officials from 17 of them.

    We found that some bureaus and services within three of the agencies we
    reviewed-DHS (for GSA properties), USPS, and Interior-are using output
    measures, and, to a lesser extent, outcome measures, while VA and some
    bureaus and services within the other three agencies are not. The agencies
    that have developed performance measures use them to evaluate and improve
    program effectiveness, make risk management decisions, and help ensure
    adequate protection at individual facilities. For example, within DHS, FPS
    has established an output-oriented performance measure to monitor the
    timely deployment of security enhancements such as x-ray machines. Such a
    measure provides a basis for FPS to compare planned versus actual
    performance. Several bureaus and services within USPS and Interior have
    developed methodologies to rank and monitor the relative risk ratings of
    their respective facilities over time-these ratings are then used as
    outcome measures for determining the change in the effectiveness of
    facility protection efforts. VA and the bureaus and services that did not
    have security performance measures generate data on ongoing protection
    activities, such as monitoring the numbers and types of security breaches
    at a given facility. This information could provide useful feedback about
    the agency's effectiveness in mitigating building security risks and
    therefore could be used for measuring performance. Although agencies have
    placed an emphasis on performance measurement and initiatives are under
    way, agency security officials said it has been challenging to measure the
    actual impact of various approaches on improving security and that
    resources for measurement initiatives have been scarce. Furthermore, while
    importance has been placed on performance measures in national homeland
    security policies and broad guidance exists for measuring the performance
    of critical infrastructure protection programs, agencies have not
    established specific guidance and standards for developing and using
    performance measures for facility protection programs in particular. This
    differs from the information technology security area, where agencies not
    only are required to measure performance, but also have detailed guidance
    and standards for developing and implementing performance measures.
    Without effective performance measurement data, especially data on program
    outcomes, decision makers may have insufficient information to evaluate
    whether the benefits of security investments justify their costs, to
    determine the effectiveness of security activities, to know the extent to
    which security enhancements have improved security or reduced federal
    facilities' vulnerability to acts of terrorism or other forms of violence,
    or to determine funding priorities within and across agencies.

    Because ISC was established to enhance the quality and effectiveness of
    security in buildings and facilities in the United States and to provide a
    permanent body to address continuing governmentwide security in federal
    facilities, we are recommending that the Secretary of DHS direct ISC to
    (1) establish guidance and standards for measuring the performance of
    facility protection efforts, particularly for program outcomes; (2)
    communicate the established guidance and standards to relevant federal
    agencies; and (3) ensure that the guidance and standards are regularly
    reviewed and updated. In commenting on a draft of this report, DHS, USPS,
    VA, and Interior generally concurred with the findings, and DHS concurred
    with the recommendations. DHS, USPS, and Interior also provided comments,
    which were incorporated as appropriate to ensure accuracy. GSA said they
    did not have any comments on the draft report.

                                       Background

    The protection of federal facilities gained importance after the 1995
    bombing of the Alfred P. Murrah Federal Building in Oklahoma City, and
    this issue became even more critical after the 1998 embassy bombings in
    Africa; the September 11, 2001, attacks on the World Trade Center and the
    Pentagon; and the anthrax attacks in the fall of 2001. Shortly after the
    1995 bombing, the President signed Executive Order 12977, establishing the
    Interagency Security Committee (ISC). ISC-which has representation from
    all major federal departments, agencies, and key offices-was charged with
    enhancing the quality and effectiveness of security in, and protection of,
    nonmilitary facilities occupied by federal employees in the United
    States.9 Furthermore, ISC was tasked to serve as a permanent body to
    address continuing governmentwide security issues for federal facilities.
    Under the order, ISC became responsible for developing policies and
    standards, ensuring compliance and overseeing implementation, and sharing
    and maintaining information. Around the same time that ISC was created,
    the Department of Justice categorized all federal facilities into security
    levels I through V based on factors such as facility size and number of
    employees, and it established recommended minimum security standards for
    each of the five levels. These standards covered perimeter, entry, and
    interior security and security planning.10

    The 2001 terrorist attacks prompted additional policies concerning
    facility protection and a variety of security enhancements at federal
    facilities. The Homeland Security Act of 2002 and a number of national
    strategies, including the National Strategy for Homeland Security,11
    assigned DHS specific duties associated with coordinating the nation's
    efforts to protect critical infrastructures and key assets. Government
    facilities (at the federal, state, and local levels) were identified as
    key assets and therefore were included in this effort.12 Furthermore, the
    2002 Act transferred FPS from GSA to DHS and, as a result, made DHS
    responsible for ISC.13 A related directive, the Homeland Security
    Presidential Directive Number 7 (HSPD-7), stated that DHS's Secretary was
    responsible for coordinating the overall national effort to identify,
    prioritize, and protect critical infrastructures and key assets.14 To meet
    this responsibility, DHS developed a National Infrastructure Protection
    Plan (NIPP), which is currently in draft form. FPS is responsible for
    implementing the NIPP for the government facilities sector. HSPD-7 also
    required each federal agency to develop plans to address identification,
    prioritization, protection, and contingency planning for physical and
    cyber critical infrastructures, along with key assets that they hold or
    operate. As the governmentwide emphasis on protecting critical
    infrastructures mounted, the federal agencies' facility protection efforts
    continued to intensify. In addition to implementing such activities as
    searching vehicles that enter federal facilities, restricting parking, and
    installing concrete bollards, federal agencies also implemented various
    security technologies, such as smart cards for access control. Figure 1
    shows smart card technologies that are utilized at a federal building.

    9ISC membership includes the Departments of State, Treasury, Defense,
    Justice, Interior, Agriculture, Commerce, Labor, Health and Human
    Services, Housing and Urban Development, Transportation, Energy,
    Education, and Veterans Affairs; GSA; Environmental Protection Agency;
    Central Intelligence Agency; and the Office of Management and Budget.
    Other members of ISC include the Director, U.S. Marshals Service; the
    Director, Security Policy Board; and the Assistant to the President for
    National Security Affairs. As a member of ISC, the Department of Defense
    participates in meetings to ensure that its physical security policies are
    consistent with ISC security standards and policy guidance, according to
    the Executive Director of ISC.

    10U.S. Department of Justice, Vulnerability Assessment of Federal
    Facilities, June 28, 1995.

    11Office of Homeland Security, The National Strategy for Homeland
    Security, July 2002.

    12The other critical infrastructure sectors and key assets identified in
    the National Strategy include agriculture and food, water, public health,
    emergency services, defense industrial base, telecommunications, energy,
    transportation, banking and finance, chemical industry and hazardous
    materials, postal and shipping, national monuments and icons, nuclear
    power plants, dams, and key commercial assets.

    13Executive Order 13286, dated February 28, 2003, amended numerous
    executive orders to reflect the transfer of certain functions and
    responsibilities to the Secretary of Homeland Security. Section 23 of the
    Executive Order transferred the ISC chairmanship responsibility from GSA
    to DHS.

    14Homeland Security Presidential Directive Number 7, Critical
    Infrastructure Identification Prioritization and Protection, Dec. 17,
    2003.

    Figure 1: Smart Card Access Portals at a Federal Building Entrance

    While it is evident from the policies and strategies outlined above that
    the protection of key assets, including federal facilities, has become an
    important issue for the U.S. government, the protection of such assets has
    also gained attention in state, local, and foreign governments, as well as
    the private sector. State and local governments in the United States, for
    instance, have taken steps to ensure the protection of critical
    infrastructures and key assets within their jurisdictions, often receiving
    resources for such efforts from the federal government. For example, DHS's
    Homeland Security Grant Program provides funding to state and local
    governments to prevent, deter, respond to, and recover from acts of
    terrorism. Funding from this grant program can be used for, among other
    things, critical infrastructure protection activities. The protection of
    critical infrastructures and key assets has also gained momentum in
    foreign governments, particularly in countries like the United Kingdom
    that have recently faced terrorist attacks. Furthermore, because many U.S.
    critical infrastructures are owned and operated by the private sector, and
    because some of these infrastructures have been targeted by terrorists in
    the past, many private-sector entities have increased their investments in
    security efforts.

    Due in part to the growing attention to facility protection, we designated
    federal real property as a high-risk area in January 2003 and have since
    published a number of reports on this issue.15 In a November 2004 report,
    we identified six key practices in protecting federal facilities, one of
    which was measuring performance to help achieve broad program goals and to
    improve security at individual facilities. We reported that, for broader
    program goals, performance measures could indicate whether organizations
    establish timelines and adhere to budgets. And, at the individual facility
    level, on-site security assessments and other active testing could provide
    data on the effectiveness of efforts to reduce a facility's vulnerability
    to attack. Training exercises and drills are also useful in assessing
    preparedness.16

    The need for agencies to measure performance stemmed from the Government
    Performance and Results Act of 1993 (GPRA),17 which was intended to
    improve federal program effectiveness, accountability, and service
    delivery. This act required federal agencies to develop strategic plans,
    link them with outcome-oriented goals, and measure agency performance in
    achieving these goals. Likewise, in the security context, a number of
    national strategies called for federal agencies to use performance
    measures to, among other things, assist in the planning and budgeting of
    protection activities for critical infrastructures and key assets.

    We have previously reported that successful performance measures should
    (1) be linked to an agency's mission and goals; (2) be clearly stated; (3)
    have quantifiable targets or other measurable values; (4) be reasonably
    free of significant bias or manipulation that would distort the accurate
    assessment of performance; (5) provide a reliable way to assess progress;
    (6) sufficiently cover a program's core activities; (7) have limited
    overlap with other measures; (8) have balance, or not emphasize one or two
    priorities at the expense of others; and (9) address governmentwide
    priorities.18

    15For example, see GAO, High-Risk Series: An Update, GAO-05-207
    (Washington, D.C.: January 2005); GAO-05-790 ; and GAO-05-49 .

    16 GAO-05-49 .

    17Pub.L. No. 103-62, 107 Stat. 285 (1993).

    Managers can use performance measures in a number of ways to improve
    programs and allocate resources more efficiently and effectively. Decision
    makers can use results from performance measurement to identify problems
    or weaknesses in programs, identify factors causing the problems, and
    modify services or processes to try to address problems. Conversely,
    results from performance measurement can be used to identify and increase
    the use of program approaches that are working well and to consider
    alternative processes in areas where goals are not met. Separately,
    performance measures can also be used to identify priorities and allocate
    resources. Decision makers can compare performance measure results with
    program goals and subsequently determine where to target resources to
    improve performance. Furthermore, in a risk management process, agencies
    can use performance measurement to assess progress towards meeting
    homeland security goals. The intended effect of assessing such progress,
    when coupled with other aspects of the risk management process, is the
    reduction of risk.19 Finally, when performance information is used to
    reward individuals, these measures can hold individuals accountable for
    certain work activities and related goals and, as a result, create an
    incentive for achieving results. A greater focus on performance results
    can be achieved by creating a cascade from an organization's goals and
    objectives down to the individual performance level. Such alignment
    facilitates the linking of individual performance to organizational
    performance.20

    18See GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing
    Season Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 22, 2002),
    pp. 2-3, 46-53.

    19GAO, Risk Management: Further Refinements Needed to Assess Risks and
    Prioritize Protective Measures at Ports and Other Critical Infrastructure,
    GAO-06-91 (Washington, D.C.: Dec. 15, 2005), pp. 24, 105.

    20See GAO, Managing For Results: Enhancing Agency Use of Performance
    Information for Management Decision Making, GAO-05-927 (Washington, D.C.:
    Sept. 9, 2005), pp. 7-17 and 21.

     Organizations outside of the U.S. Government Use Security Performance Measures
               to Enhance Decision Making and Help Ensure Accountability

    We found a range of examples of performance measures that organizations
    outside the U.S. government-including private-sector firms, state and
    local governments, and foreign government agencies-used to track the
    number and types of security activities conducted, the quantity of
    security equipment and services delivered, and the outcomes of these
    security efforts.21 Security officials within these organizations
    recognized that performance measures helped them better assess how
    effective they were in protecting against threats to and vulnerabilities
    of their facilities. Organizations then used the results of these
    performance measures to improve security, inform the risk management
    process, make resource allocation decisions, and hold security officials
    and others in the organization accountable for security performance.
    Despite efforts by some organizations to use performance measures as an
    additional decision-making tool, some security officials told us that they
    faced some challenges in developing and implementing performance measures.
    The challenges include limited guidance and expertise in the performance
    measurement area.

    Selected Organizations Use a Range of Output, Outcome, and Process/Input
    Measures to Assess the Effectiveness of Facility Protection Efforts

    Security officials recognized that performance measurement is important
    for improving facility protection and ensuring accountability. They also
    acknowledged that performance measures would allow them to take a more
    strategic, outcome-based approach to managing their security programs and
    to better prepare their facilities against terrorism and other threats.
    However, less than a quarter of the organizations we interviewed told us
    that they have developed and used various performance measures for their
    security programs, and several of those that did have performance measures
    said that the measures are still a work in progress. Table 1 provides
    examples of the output, outcome, and process/input measures these
    organizations have developed. Appendix II provides additional examples of
    performance measures.

    21For this report, we categorized the District of Columbia as a local
    government.

    Table 1: Examples of Performance Measures for Facility Protection

    Type of measure Example
    Output             o  Number of risk assessments performed
                          o  Average time to process background screenings
                          o  Compliance with security policies
                          o  Client/customer satisfaction with security services
    Outcome            o  Evidence of damage to buildings and facilities
                          o  Change in risk rating resulting from countermeasures
                          deployed
                          o  Change in the total number of security-related
                          incidents
    Process/Input      o  Number of security clearances undertaken
                          o  Number of training courses and drills conducted
                          o  Number of security guards

    Source: GAO.

    Note: GAO analysis of data from selected state, local, and foreign
    government agencies and private-sector organizations.

    In some of the organizations we interviewed, some security officials use
    output measures to monitor the direct products and services delivered by a
    program and the characteristics of those outputs, including efficiency,
    cost-effectiveness, timeliness, quality, and customer service. Some
    security officials use outcome measures to compare the results of those
    products and services with the goals security officials are trying to
    achieve, such as reducing the total number of security incidents relating
    to thefts, vandalism, and acts of terrorism. In addition, some security
    officials use outcome measures to assess whether their security program is
    operating efficiently and to determine the quality of the services and
    products they are trying to provide. Separately, security officials use
    various process/input measures to provide a descriptive overview of the
    program activities and the resources of their security program, including
    the types and numbers of facilities they manage and the level of
    countermeasures,22 such as entry control security systems, they have
    installed. Input measures are used for resource allocation and monitoring
    and do little to reflect the effectiveness of the security program.

    As an additional output measure, some of the organizations we interviewed
    determine whether their security efforts comply with their security
    policies, standards, and guidance. For example, some of the government
    agencies in the three foreign countries we visited use performance
    measures to evaluate whether their security activities are compliant with
    their government's protective security policies. Several security
    officials in these agencies told us that they use this measure to
    demonstrate compliance with established government standards. Some of
    these foreign government agencies indicated that they measure compliance
    based on the results of security audits completed internally-by the
    security department or other departments within the organization-or
    externally. Some of these security officials then use the results of the
    audits to identify security weaknesses and make corrections to improve
    security. Other foreign government agencies use surveys to measure the
    degree of security policy compliance. For example, Australian government
    agencies are required to adhere to the minimum protective security
    standards contained in the Australian government's Protective Security
    Manual.23 Ministers and agency heads are accountable for their agency's
    compliance with these standards. Agencies are surveyed annually for
    compliance with the security manual standards. The survey results are
    assessed and reported to the central government.

    22A countermeasure is any action taken or physical equipment used
    principally to reduce or eliminate one or more vulnerabilities.

    Some of the nonfederal organizations we interviewed also measure the
    effectiveness of their countermeasures by determining whether the services
    and security equipment they provide are adequate under both real and
    simulated conditions. Some of the organizations we interviewed stated that
    they test security equipment, such as perimeter alarms and x-ray machines,
    and conduct simulated attacks and penetration exercises on a periodic
    basis. One official from the gaming industry said that it is important to
    test equipment to ensure it is being used properly, because the technology
    itself is not as important as how it is used. For example, a facility
    could have a sophisticated card access system but still be vulnerable if
    someone props the door open. To help government agencies select effective
    security equipment, a central agency in the United Kingdom tests security
    equipment and provides those in the security community with information to
    help the user match the appropriate equipment to the requirement.
    Similarly, an agency in Australia conducts tests on security equipment and
    provides agencies with a catalog of approved products. Security officials
    from the gaming industry also told us that they are members of an external
    group that tests security equipment and shares the results of the testing
    with security officials in other industries, such as the chemical,
    petrochemical, and pharmaceutical industries.

    23The Australian government's Protective Security Manual contains
    governmentwide policies and guidelines that establish the minimum
    standards for the protection of Australian government resources (including
    information, personnel, and assets) that all agencies governed by the
    country's Financial Management and Accountability Act of 1997 must meet.

    In some organizations, the selection of useful performance measures has
    evolved through a trial-and-error process. For example, one financial
    services organization went through several iterations of its security
    performance measures over a 1-1/2 year period in order to determine which
    performance measures were important to monitor and would provide them the
    right information needed to achieve the organization's security
    objectives. For example, they initially reported on the number of security
    alarms, and then changed the measure to a more useful measure-the number
    of alarms with unique responses (i.e., alarms that required a guard to
    respond in person)-so that they could better understand how security staff
    were interacting with the security equipment. One security official
    acknowledged that, although they were satisfied with their current
    performance measures, it would still be helpful to measure performance in
    other areas, such as employee satisfaction with security services.

      Case Example: A Financial Service Organization's Performance Measures

    Security officials at a large, well-known financial services organization
    use a number of output and outcome measures to regularly monitor the
    performance of their security program. In addition, they use process/input
    measures to assist them with resource allocation decisions. The security
    officials emphasized that there is a constant need to measure and evaluate
    what their security program does in order to educate business
    professionals on the importance of a security investment. While the
    organization assesses all of its facilities using a baseline set of
    security standards and risk assessments, performance measures provide
    security officials with information to understand whether these standards
    and risk assessments are actually improving their security situation. The
    security officials told us that they use the following performance
    measures:

               o  Outputs-Security officials use output measures relating to
               their operational readiness (i.e., how prepared the security
               program is against potential security threats), which includes the
               number of risk assessments performed. They also measure the number
               of non-security related incidents such as false alarms or broken
               security cameras. In addition, security officials monitor the
               number of policy exceptions that exist when a business line or
               facility cannot comply with the standards set forth in their
               security policy manual. If many exceptions to a particular section
               of the policy manual occur in a given month, a policy working
               group reviews the issue and determines whether additional
               assistance will be required to bring the facilities into
               compliance.
               o  Outcomes-One outcome measure is the monetary savings resulting
               from less costly, more efficient security processes and new
               technologies. Security officials use this outcome measure to
               demonstrate savings from the security program's budget as well as
               from the budgets external to the security division, such as
               operations. Officials are also able to prorate contract-related
               savings over the lifetime of the contract to better understand how
               the savings affect the organization over time. To understand the
               effectiveness of their security efforts, security officials use
               data on the responses to security incidents, which are classified
               by type (e.g., assault, burglary, terrorism). Security officials
               then analyze the data to help them make recommendations for
               additional security improvements.
               o  Process/Input-The financial organization tracks guard levels,
               security expenditures, and security activities across all its
               facilities. Security officials use these measures to compare the
               different levels of service required, given the risk associated at
               each facility or region. In a given month, they also measure the
               number of training sessions and drills conducted. The performance
               measure for training identifies the specialized fields in which
               the security staff are being trained and the type of training the
               security staff are providing to others.

               Security officials at this financial services organization told us
               that they monitor their performance measures on a monthly basis,
               and that the data are aggregated for the entire organization and
               also broken out by region. They developed, and have continued to
               modify, their performance measures based on the analysis of
               incidents and other activities in a particular region as well as
               trends across regional facilities. They also obtained feedback
               from regional offices and from their own security staff. Security
               officials noted that they tried to select performance measures
               that represented common threads and were not biased in favor of
               one particular region. They also continuously evaluate the
               usefulness of their performance measures, adding a measure if they
               determine that information is needed on a particular subject or
               dropping a measure if it does not seem to be informative.

               We have previously reported that organizations can use the results
               of performance measures to make various types of management
               decisions to improve programs.24 Security professionals also
               recognize the benefits of using performance measurement within the
               security industry. At a major security industry conference in
               2005, a conference presenter indicated that the ability to compare
               past performance and the performance of others contributes to the
               goal of continuous improvement, the result of which is a stronger,
               more mature security program with security processes that can
               better protect facilities and staff from harm. Performance
               measures also provide management with the tools to verify that the
               organization's resources are used responsibly and security risks
               are managed appropriately.

               In some of the organizations we interviewed, security officials
               and other decision makers use performance measures to manage risk,
               allocate resources, and improve the quality of the security
               services they provide for their facilities. For example, at one
               financial services organization, security officials installed
               protective security equipment at some of their facilities and then
               compared the number of security incidents and the level of
               customer satisfaction before and after the equipment was
               installed. In this particular case, security officials used this
               performance measurement data to demonstrate the value of that
               security investment to their corporate management and the business
               lines they supported. The performance measures also allowed
               security officials to compare individual facility performance to
               the average within the industry, which they use to demonstrate the
               risk level of a particular facility and take appropriate action to
               address the risk.

               Where security goals and objectives were not achieved, some
               security officials also used performance measurement results to
               identify problem areas and take corrective action. Several
               organizations mentioned that they measure the quality of their
               security efforts through an output measure by soliciting feedback
               from employees and clients through customer satisfaction surveys.
               For instance, one Canadian organization periodically surveys
               clients about their satisfaction with the security services the
               organization provides to government agencies. The survey questions
               range from how often the client saw security managers to how
               satisfied they were with the services they received. The responses
               to the surveys provide feedback that allows security officials to
               improve their provision of security services to both private and
               public sector clients.

               Performance measures helped security officials in one government
               agency in Australia become better risk managers and allocate
               resources more efficiently across facilities. The agency uses a
               security plan that includes security objectives that are linked to
               its strategic goals. The plan also lists strategies and actions
               for achieving these objectives, along with performance measures
               that assess the extent to which objectives are being achieved. For
               example, the performance measures monitor the extent to which
               security practices are in accordance with the agency's security
               policies, any evidence of harm to agency staff or facilities, and
               the extent to which agency stakeholders view the agency's
               facilities as safe for their resources and assets. To monitor
               performance, security officials use two different review
               processes. First, security officials can access the audit function
               of a computer-based risk assessment model to monitor the outcomes
               of the performance measures contained in their security plan and
               to understand how well their security efforts are performing
               within individual facilities. For example, the risk-assessment
               model allows security officials to monitor the impact of
               additional security expenditures on a facility's risk rating while
               controlling for existing security enhancements that mitigate the
               risk, such as the number of guard patrols and the adequacy of
               access control systems (e.g., electronic locks). Security
               officials can then use the results to justify spending decisions
               and prioritize security investments. For example, one facility
               requested a perimeter fence, and security officials were able to
               use the risk-assessment model to demonstrate that the facility's
               risk was adequately managed without the fence since there were no
               known risks in that location and since the facility already had
               guards and an alarm system. Second, the agency's audit unit also
               conducts its own independent measurement of the security
               activities so that security officials can compare across
               facilities to guide them in determining where they need to make
               adjustments. Together, these two security reviews provide the
               security program with enough information to assess their security
               position, according to one agency security official.

               Security officials recognized the value of performance measures to
               help ensure the accountability of security officials, management,
               and other employees throughout the organization. Many of the
               organizations we interviewed had security policies and procedures
               in place, and some of these organizations were able to link these
               plans directly to performance measures that demonstrated
               achievement of both the security-related strategic goals and the
               organization's broader strategic goals. We have previously
               reported that aligning the goals at the executive level with the
               goals and objectives at each operational level reinforces the
               connection between strategic goals and the day-to-day activities
               of managers and staff.25 For example, an annual goal that is
               linked to a program and also to a long-term goal can be used to
               hold agencies and program offices accountable for achieving those
               goals.26 Furthermore, we reported that such alignment increases
               the usefulness of performance information to decision makers at
               each level.27

               One agency within the District of Columbia (D.C.) government uses
               performance measures and targets to hold agency management and
               security officials responsible for its security-related
               activities. D.C.'s Office of Property Management is responsible
               for D.C. government buildings, and the Protective Services
               Division, which falls under Property Management, is responsible
               for security at these buildings. Protective Services faces a
               unique environment in protecting the facilities that it is
               responsible for because of the proximity of these assets to
               federal facilities, which are considered to be attractive targets
               for terrorist attacks. To help ensure that their security concerns
               are addressed, security officials in Protective Services noted
               that they have linked their security goals and related performance
               measures with the Property Management's goals and citywide
               strategic goals (see fig. 2). Specifically, Protective Services'
               goals, performance measures, and related targets support the goal
               of Property Management to provide a high-quality work environment
               and user-friendly facilities, and also support the broader
               citywide strategic goal of making government work. The security
               officials pointed out that this alignment is very deliberate and
               can help hold officials accountable for a security-related
               activity. For example, the Director of Property Management can use
               security-related performance measures and corresponding targets to
               hold the Protective Services Division accountable for its
               activity. If Protective Services does not meet the targets, it is
               required to submit justifications to senior management as to why
               they were not met. The officials explained, however, that in
               situations where there are unforeseen circumstances, their targets
               can be realigned, with the consent of senior management. For
               example, following Hurricane Katrina, Protective Services was
               required to provide security services for Katrina victims housed
               at a D.C. arena. The human resources required for this task made
               it impossible for Protective Services to meet all the targets, and
               the D.C. mayor's office allowed for adjustments to the target for
               that time. Separately, the mayor's office can also use the
               security-related performance measures and targets in conjunction
               with other Property Management performance measures and targets to
               monitor the work of the entire agency and hold the Director of
               Property Management accountable for agencywide activity.

               Figure 2: Linkages between District of Columbia Strategic Goals
               and Performance Measures for Facility Protection

               We also recognized in a previous report that the establishment of
               a chief security officer position is essential in organizations
               that own and operate large numbers of mission-critical
               facilities.28 Such a high-level position is important for
               coordinating security responsibilities across facilities and
               ensuring accountability for security results, including
               establishing linkages between security performance and outcomes.
               We found that government agencies in all three countries we
               visited are required to designate a departmental security officer
               (DSO) or an agency security executive to oversee security matters
               across all agency facilities and implement government security
               policies. For example, in the United Kingdom, security officials
               told us that the DSOs are sufficiently senior within each agency
               department to have an effective voice and to put security issues
               on the management agenda. These security officials also told us
               that the DSOs are playing a greater role in coordinating with
               other agency departments to enhance their security. The financial
               services and gaming organizations we interviewed also have
               directors or vice-presidents of security who have a direct line of
               communication to their corporate management. They said that this
               arrangement promotes a good working relationship with management
               and allows them to identify and fix security problems efficiently.

               Some of the organizations we interviewed also used performance
               measures to hold security officials accountable for program
               performance. For example, some organizations hold their security
               officials accountable for results through the use of customer
               satisfaction surveys. Security officials at one financial services
               organization indicated that they conduct quality surveys with
               their business-line clients, which allows clients to provide input
               to security officials on whether the security program is effective
               and whether the security program met the client's expectations.

               Two major financial services organizations we interviewed use
               performance measures to help ensure accountability for investments
               in security improvements and compliance with security policies and
               regulations. Security officials in one financial services
               organization told us that they work in a security culture that is
               very performance driven. While their security budget is fully
               separate from other corporate expenditures, regional security
               directors are responsible for determining how to spend security
               funds. Regional security directors use performance measures to
               justify security expenditures to all of the individual business
               lines they support and to demonstrate a return on investment for
               their security expenditures. For example, the organization uses
               output and outcome performance measures to monitor monetary
               savings, the number of security incidents, and the impact of new
               technologies and processes. When security officials want to invest
               in a new security technology, they use these performance measures
               to demonstrate to the business lines that they have investigated
               all of the alternatives and determined the cost and potential
               savings of the purchase. For example, they used past data on the
               cost and performance of security equipment and guards to calculate
               the cost of installing some security equipment versus hiring a
               security guard to protect one of its facilities. They were able to
               demonstrate that the security equipment would be more
               cost-efficient over time and be more effective in deterring
               certain crimes.

               Another financial services organization uses performance measures
               to help ensure that all facilities are complying with its security
               policies and regulations. The security policies for each of the
               organization's business lines differ based on their level of risk.
               As a form of quality control for its security operations, each
               business line is expected to conduct compliance reviews of all of
               its facilities, including confirming the presence of required key
               security equipment and determining whether staff members are
               following security policies. Each business manager is held
               accountable for the results of these reviews: senior security
               officials receive and review monthly compliance reports, and the
               financial services organization's central audit department ensures
               that the reviews were properly conducted. According to security
               officials, the data in the monthly reports are used to determine
               where problems exist and look for emerging security trends.

               One Australian government agency uses performance measures to hold
               its security executives accountable for identifying and addressing
               security risks. Officials from the agency noted that they have
               historically had a strong security and risk management culture
               that emphasizes executive accountability for performance. The
               agency holds its security executives accountable by requiring them
               to produce a certificate of assurance that includes physical and
               personal security. The purpose of the certificate, which is signed
               by a senior agency executive, is to assure the chief executive
               that the agency is meeting its security obligations, and that
               action plans are in place to address any problems. It covers
               compliance with external requirements, including government
               regulations, and internal conformance with corporate security
               policies. The assurances given must be underpinned by evidence,
               which includes the results of physical security reviews that are
               conducted periodically at each facility. These reviews measure and
               report on the standard of physical security, including perimeter
               security, access control, alarm systems, and surveillance and
               communication systems. The certificate uses a color code to
               indicate the overall status of the security function-red, amber,
               or green. Certificates rated red or amber are reviewed and
               resubmitted every 6 months. Green certificates are reviewed
               annually. If the certificate identifies a security problem, it
               must be accompanied with an action plan for addressing the risks
               involved.

               Although performance measurement is seen as an important tool for
               assessing the effectiveness of security programs, developing and
               using performance measures can be challenging, according to
               security officials we interviewed at selected organizations. A
               difficulty with developing performance measures is determining
               whether the measures that are used are suitable, given a
               constantly changing threat environment. Some security officials
               said that it was difficult to know what to measure because
               security is intangible or difficult to quantify. Others also
               acknowledged that it is difficult to determine whether a potential
               security threat or incident has been prevented, even after
               additional countermeasures or security staff are introduced to
               address perceived security weaknesses, because deterrence is
               immeasurable. Several security officials cited the difficulty in
               determining a causal relationship between security efforts and
               changes in the number of security incidents. For example, a
               security official from an Australian government agency indicated
               that an increase in the number of breaches in a particular
               facility may result because an organization is being targeted at
               that particular point in time rather than because it lacks
               adequate security measures. Organizations also find it hard to
               measure the impact of some security actions, such as the potential
               financial savings resulting from attacks that have been
               discouraged. Organizations told us that they recognize the need to
               draw linkages between security incidents and security investments,
               but some organizations find it difficult to measure the benefit of
               a particular security process or piece of equipment in the absence
               of a security breach.

               A number of organizations also told us that other priorities and
               considerations might hinder their ability to effectively use
               performance measures for making security decisions. Some security
               officials pointed out that the ultimate decision on how to
               allocate security resources can be based on priorities other than
               performance. For example, several private sector and foreign
               government agencies we interviewed noted that they have to balance
               their security needs with their goals of maintaining sufficient
               public access to their facilities. Some security officials are
               also reluctant to use performance measures because they do not
               want to be held accountable for not meeting their performance
               targets. Several organizations mentioned that potential liability
               could be seen as a disincentive for using performance measurement
               data, because an organization may be seen as negligent if the
               performance data were to show that an organization could have done
               something to prevent an incident but chose not to. One security
               official told us that having established performance targets could
               also discourage organizations from accurately collecting data
               because security officials may be reluctant to report an incident
               if a decline in the number of incidents is one of the performance
               goals.

               Some organizations we interviewed cited the lack of knowledge and
               expertise available to collect and analyze security data as a
               limitation to overcoming some of the challenges of using
               performance measures. One financial services organization
               indicated that some of its security officials did not see the
               benefits of using performance measures until after they saw that
               their business line managers responded favorably to the use of
               performance measures to demonstrate a return on investment for
               security expenditures. Several state, local, and foreign
               government agency officials noted that they had limited management
               staff available to develop and monitor performance measures for
               physical security. According to one state government agency
               official, without staff expertise in this area, security staff
               tend to approach security initiatives like a project-they monitor
               the initiative's progress to make sure that it is delivered on
               time and on budget, but they do not necessarily measure the
               effectiveness of the security equipment once it is installed.

               Many organizations we interviewed said that they face the
               aforementioned challenges, and we noted that some of the entities
               outside the U.S. government rely on U.S. agencies for support and
               leadership in developing security standards and performance
               measures. One state government agency we interviewed expressed an
               interest in developing performance measures in the future and
               mentioned that it often looks to the federal government for
               guidance on security efforts. DHS officials told us that their
               agency was providing assistance to several foreign government
               agencies in the United Kingdom in measuring performance and
               allocating security resources. One foreign government agency said
               that it was interested in developing governmentwide guidance for
               measuring security performance but was looking to U.S. agencies
               for assistance in this area.

               Responding to the requirements in 2002 by the National Homeland
               Security Strategy and subsequent federal policies, agencies have
               paid greater attention to facility protection and have begun using
               key practices-such as performance measurement-to varying degrees.
               Agency officials noted that developing performance measures for
               facility protection was a difficult undertaking, since the results
               are not always readily observable. We found that some bureaus and
               services within three of the agencies we reviewed-DHS, USPS, and
               Interior-are using output measures and, to a lesser extent,
               outcome measures, while the VA and some bureaus and services
               within the other three agencies are not. Despite the lack of
               security performance measures, we found that ongoing protection
               activities within these bureaus and services and the VA, such as
               monitoring the numbers and types of security breaches at a given
               facility, generate a wealth of data that could provide useful
               feedback about the agency's effectiveness in mitigating building
               security risks, and therefore could be used as measures of
               performance. While the agencies have demonstrated some progress in
               applying performance measurement to facility protection, with
               limited precedent for how to do this, more work remains to
               identify measures-particularly outcome measures-that assess the
               impact of facility protection efforts. Output measures do not
               provide an indication of what security activities are
               accomplishing, while outcome measures that are clearly tied to
               results indicate the extent of progress made and help identify the
               security gaps that still remain. Officials expressed concerns
               about the lack of resources and the limitations of existing
               guidance in providing direction about how to measure progress and
               evaluate the effectiveness of physical security programs.

               In general, the agencies we reviewed have made some progress in
               collecting and using performance-related data for their facility
               protection program activities, but many of the measures are of
               program outputs rather than outcomes. While output measures are an
               important part of performance measurement, outcome measures could
               provide information to evaluate whether the benefits of security
               investments outweigh their costs and to determine the
               effectiveness of security activities. The agencies we reviewed use
               output measures, such as the timely completion of risk assessments
               and whether countermeasures work as intended once deployed, to
               inform risk management decisions and to help ensure adequate
               protection at the individual facility. Additionally, several
               bureaus and services within DHS, USPS, and Interior have developed
               outcome measures to rank and monitor the relative risk ratings of
               their respective facilities over time or to otherwise assess the
               effectiveness of their facility protection efforts.

               The effectiveness of security programs at GSA facilities is
               evaluated using performance measures developed by the Federal
               Protective Service (FPS) and a physical security testing program
               developed by GSA. FPS has identified four performance
               measures-both output and outcome measures-to assess its efforts to
               reduce or mitigate building security risks. These four performance
               measures, detailed in table 2, are at varying stages of
               implementation and are still evolving. Under the Homeland Security
               Act of 2002, DHS, through FPS, is directly responsible for law
               enforcement and security-related functions at facilities under
               GSA's control or custody. FPS delivers security and law
               enforcement services for approximately 8,000 facilities that fall
               under GSA's jurisdiction.

    Security Officials Use Performance Measure Results for Risk Management and
    Resource Allocation

    24See GAO-05-927 .

      Case Example: An Australian Agency's Risk Model

    Performance Measures Can Be Used to Hold Security Officials Accountable for
    Achieving Goals and Results

      Case Example: The District of Columbia's Alignment of Security Goals and
      Measures

    25 GAO-05-927 .

    26 GAO-03-143 .

    27 GAO-05-927 .

      Departmental Security Officers and Individual Accountability

      Case Example: Individual Accountability in Two Financial Services
      Organizations

    28 GAO-05-790 .

      Case Example: An Australian Agency's Security Certification Process

    Organizations Cited Challenges in Developing and Using Performance Measures

    U.S. Agencies Have Made Some Progress in Developing and Using Performance
    Measures for Facility Protection Programs, but Lack Guidance and Standards

    Agencies Use Output Measures and Some Outcome Measures to Inform Risk
    Management, Help Ensure Adequate Protection, and Assess Effectiveness of
    Facility Protection Efforts

      Case Example: DHS's Federal Protective Service in GSA Facilities

    Table 2: FPS's Performance Measures for Facility Protection

    Type of
    measure Performance measure         Purpose
    Output  Timely deployment of        To compare actual deployment dates
               countermeasures             with planned deployment dates
    Output  Countermeasure              To gauge whether those security
               functionality (e.g.,        countermeasures for which FPS is
               surveillance cameras, x-ray contractually responsible are working
               machines)                   as intended, once deployed
    Output  Patrol and response time    To assess FPS's ability to respond to
                                           calls for service within certain time
                                           limit goals
    Outcome Facility security index     To calculate FPS's average success
                                           rate for the above three performance
                                           measures

    Source: GAO.

    Note: GAO analysis of FPS data.

    The first measure-monitoring the deployment of countermeasures-focuses on
    the timeliness of implementation and serves as a measure of program
    output. Once approval and funding have been obtained to implement a
    recommended countermeasure, FPS personnel record planned deployment dates
    so that they can compare them with actual implementation dates. An FPS
    working group decided that the initial baseline for this measure,
    developed in fiscal year 2005, would be 90-percent success, which is
    calculated as the number of countermeasures actually deployed by the
    scheduled due date, divided by the number planned. FPS officials noted
    that they will not know how well they are progressing on this measure
    until the end of fiscal year 2006 because they are still automating the
    process and training regional staff. For fiscal year 2007 and subsequent
    years, FPS expects the annual goal to be some increment above the
    preceding year's results until the long-term goal of 98 percent is
    achieved and maintained.

    Countermeasure functionality, FPS's second measure, gauges whether a
    countermeasure works as intended once it is deployed. Specifically, it
    assesses the operational capability of five major groups of
    countermeasures for which FPS is contractually responsible: closed circuit
    television surveillance, contract security guards, intrusion detection
    systems, magnetometers, and x-ray machines. In some instances, contract
    guards are routinely evaluated to determine whether they are performing
    effectively. Performance includes the guards' knowledge of and compliance
    with relevant operations for their security post. Based on FPS testing
    results in fiscal year 2005, the baseline for this measure is 90-percent
    success, which is calculated as the number of countermeasures working and
    performing as intended divided by the number tested. According to FPS
    officials, FPS currently has about a 92-percent success rate for this
    measure. The long-term goal for this measure is 100-percent effectiveness.
    Related to facility protection, this output measure reflects the
    functionality of a program element, but not its effect.

    Patrol and response, the third measure, assesses FPS's ability to respond
    to calls for service within certain time limit goals. The initial baseline
    for this measure was established in October 2005 and was about 17.5
    minutes. This baseline represents an average response time for all of
    FPS's 11 regions, and is calculated using dispatch and arrival time
    information from FPS's online incident reporting system. The time
    parameters for data collection fell between FPS's core duty hours of 6:00
    a.m. and 6:00 p.m. The goal for this measure is to reduce response times
    by 10 percent, although FPS noted that this goal could increase or
    decrease depending on staffing levels or deployments. At the time of this
    report, FPS noted that they have collected statistics on response times
    for this measure and are in the process of evaluating whether they have
    achieved their goal.

    Finally, the facility security index-an outcome measure29-calculates the
    overall effectiveness of FPS operations in meeting the performance goals
    of the three output measures described above (timely deployment of
    countermeasures, countermeasure functionality, and patrol and response
    time). An index score of 1 indicates that FPS has met its performance
    goals, a score of greater than 1 indicates that FPS has exceeded the
    goals, and a score of less than 1 indicates that it has not met the goals.

    29Although FPS considers this an outcome measure, it is intended to
    reflect the composite level of performance of its three output measures.

    Taken together, these four FPS performance measures provide insight into
    activities designed to support FPS's efforts to prevent and respond to
    security and criminal incidents, including terrorist threats. In addition
    to assessing FPS's performance in fulfilling its facility protection
    responsibilities, the measures also serve as a baseline for making
    decisions about deploying existing resources or requesting additional
    resources. FPS officials told us that these measures are derived from
    strategic objectives established by DHS's Immigration and Customs
    Enforcement (ICE), of which FPS is a component. These objectives include
    implementing appropriate countermeasures to reduce vulnerabilities facing
    buildings under GSA's jurisdiction (see fig. 3). Aligning facility
    protection performance measures and targets with broader DHS and ICE
    mission, goals, and objectives helps hold employees accountable for
    security activity and allows them to observe how day-to-day security
    activities contribute to the broader mission, goals, and objectives.
    Similar to organizations outside the federal government, FPS provides its
    financial management staff with quarterly and annual reports that document
    the accomplishments for each measure in order to support planning and
    budgeting efforts included in DHS's Future Years Homeland Security Program
    document.30

    30The Homeland Security Act requires that, beginning in fiscal year 2005,
    DHS prepare the Future Years Homeland Security Program document-a 5-year
    resource plan that outlines departmental priorities and the ramifications
    of program and budget decisions. See GAO, Results Oriented Government:
    Improvements to DHS's Planning Process Would Enhance Usefulness and
    Accountability, GAO-05-300 (Washington, D.C.: Mar. 31, 2005).

    Figure 3: Linkages between DHS Mission and FPS Performance Measures for
    Facility Protection

    It is important to note that when FPS was a part of GSA, we reported on
    GSA's lack of performance goals and measures for its building security
    program. In June 1998, we testified that GSA had not established key
    program evaluation mechanisms for its building security program that could
    help determine how effective its security program has been in reducing or
    mitigating building security risks or in shaping new security programs.31
    At the time, we reported on features that would support program
    evaluation, including: (1) developing specific goals, outcomes, and
    performance indicators for the security program, such as reducing the
    number of unauthorized entries; (2) establishing and implementing
    systematic security program evaluations that provide feedback on how well
    the security program is achieving its objectives and contributing to GSA's
    strategic goals; and (3) ensuring that a reliable performance data
    information system is in place. While we found that GSA had established
    goals and measures for its security program both apart from and in
    connection with GPRA, we noted that these goals and measures were output
    oriented and did not address the outcomes or results the building security
    program was expected to achieve. Consequently, we recommended that GSA
    develop outcome-oriented goals and measures for its building security
    program. As previously noted, FPS has demonstrated some progress in moving
    beyond the use of output measures that monitor program activity in
    carrying out its responsibilities within the Department of Homeland
    Security (DHS).

    In addition to FPS's performance measures for assessing the security of
    properties under GSA's control, GSA's Office of the Chief Architect also
    has a program for testing the physical security of GSA buildings. Under
    this program, GSA performs explosive testing of various window systems;
    identifies gaps in protective design and security technologies; and
    provides criteria and tools for blast resistant design, progressive
    collapse in new and existing facilities, and upgrading walls to reduce
    fragmentation and hazards resulting from an explosion, among other things.
    The program team is also developing a tool to identify gaps in security
    planning, ensure consistency with GSA policies and ISC's security design
    criteria, and provide a consistent foundation and methodology for making
    security design decisions.

      Case Example: Interior's Bureau of Reclamation and National Park Service

    One bureau within Interior-the Bureau of Reclamation (BOR)-has identified
    performance measures for its facility protection programs, while the
    National Park Service (Park Service) generates information that could be
    used to monitor the effectiveness of its physical security efforts. Each
    of Interior's eight bureaus independently manages the protection program
    for the facilities that fall under its respective purview, and each bureau
    has developed broad security goals derived from the agency's overall
    mission.32 In general, Interior's program evaluation methods are based on
    GPRA and the Office of Management and Budget's (OMB) Program Assessment
    Rating Tool (PART).33 Several of the bureaus have had their programs
    reviewed under the PART system, and some security performance measures
    were identified as part of this effort. Over time, Interior intends to
    have all of its law enforcement programs assessed under the PART system.
    However, an agency official from the Park Service reported difficulty in
    developing formal performance measures because GPRA is directed toward
    evaluating federal programs and does not provide guidance on developing
    goals and measures specifically for security activities.

    31GAO, General Services Administration: Many Building Security Upgrades
    Made But Problems Have Hindered Program Information, GAO/T-GGD-98-141
    (Washington, D.C.: June 4, 1998).

    Within Interior, BOR has an important role in protecting critical
    infrastructures because of its responsibilities related to dams. BOR is
    responsible for managing and protecting well-known assets such as Hoover
    Dam in Arizona and Nevada, which receives approximately 1 million paying
    visitors each year. In 2005, the security program administered by BOR was
    selected for review under the PART system. To demonstrate its progress in
    meeting the long-term goal of reducing security-related risks at its
    critical facilities, BOR developed several output and outcome performance
    measures, including (1) timely completion of risk assessments, (2) the
    cost per active background investigation, (3) the percentage of
    recommendations that have been implemented based on the results of risk
    assessments, (4) the number of updated regional threat assessments, and
    (5) changes in the risk ratings as countermeasures are implemented for an
    individual asset (see table 3). Although these measures were developed for
    the protection of dams and related facilities, they could be applied to
    building security because there is some similarity in the protection
    activities. In all but one instance, BOR had achieved or exceeded its
    performance target for each measure established for fiscal year 2005.
    According to OMB's PART assessment, BOR's facility protection program was
    rated moderately effective and its performance measures were described as
    creative and useful measures that will help monitor program
    accomplishments and efficiency.34

    32However, to centrally manage Interior's security initiatives, the
    department established in 2002 a central coordination and oversight office
    for activities related to homeland security. This office-the Office of Law
    Enforcement and Security-has worked within Interior to identify assets
    that are likely targets, conduct risk assessments, and coordinate efforts
    by Interior's bureaus to enhance security at individual locations. See
    GAO-05-790 .

    33OMB developed PART to support the integration of performance information
    and budgeting. OMB describes it as a diagnostic tool meant to provide a
    consistent approach to evaluating federal programs as part of the
    executive budget formulation process.

    Table 3: BOR's Performance Measures for Facility Protection

    Type of
    measure Performance measure      Purpose
    Output  Timely completion of     To compare actual completion dates with
               risk assessments         planned completion dates
    Output  Cost per active          To monitor the cost efficiency of the
               background investigation personnel security program, including
               file                     processing of background investigations,
                                        issuance and verification of clearances,
                                        and case file maintenance
    Output  Status of                To indicate the percentage of recommended
               recommendations designed security enhancements that have been
               to mitigate risk         funded and implemented, and are
                                        operational
    Output  Number of updated        To assess the frequency with which
               regional threat          assessments are conducted and help ensure
               assessments              that current threat intelligence is
                                        incorporated as part of risk assessments
                                        and risk-reduction strategies
    Outcome Change in risk ratings   To assess the risk-reduction benefits
                                        associated with implementing
                                        countermeasures at an individual asset

    Source: GAO.

    Note: GAO analysis of BOR data.

    The Park Service is responsible for managing and protecting some of the
    nation's most treasured icons, including the Washington Monument, the
    Lincoln and Jefferson Memorials, and the Statue of Liberty. The Park
    Service manages more than 375 park units, covering more than 84 million
    acres, which provide recreational and educational opportunities and
    numerous other benefits to millions of visitors each year. From 2001 to
    2005, park units averaged a total of about 274 million recreation visits
    per year. While a Park Service official stated that they did not have any
    formal performance measures for facility protection, we found that their
    risk management methodology provides useful feedback about the bureau's
    effectiveness in reducing or mitigating security risks for facilities
    under its jurisdiction. In June 2005, we reported that Interior had made
    significant progress in the risk assessment area, in large part due to its
    new National Monuments and Icons Assessment Methodology (NM&I).35 NM&I-a
    uniform risk assessment and ranking methodology-is specifically designed
    to quantify risk, identify needed countermeasures, and measure
    risk-reduction benefits at icon and monument assets. According to an
    Interior official, Interior's Office of Law Enforcement and Security
    (OLES) developed NM&I to assist bureaus in quantifying risk levels and
    identifying needed security enhancements, initially at critical
    infrastructures and key assets, but eventually at all departmental
    facilities. The NM&I methodology has a consequence assessment phase and a
    risk assessment phase. First, during the consequence assessment phase,
    senior officials from the Park Service and OLES determine which icons are
    considered nationally significant.36 Specific attack scenarios-such as
    chemical/biological, aircraft, or improvised explosive device-are used to
    evaluate security at each asset and score attack consequences.37 During
    the risk assessment phase, a group of security professionals from the Park
    Service and OLES, assisted by the site security supervisor and the site
    manager, collectively determine the effectiveness of existing security
    systems using DHS guidelines. Using risk values calculated from this
    evaluation, OLES assigns asset risk ratings of high, medium, or low, and
    specific mitigation recommendations are formulated. As part of its annual
    review, OLES routinely monitors the security enhancements that have been
    implemented to reduce the risk rating designations. OLES has not had
    formal performance measures and targets for reducing risk ratings in the
    past. However, in April 2006, according to Interior officials, OLES
    developed and submitted for inclusion in the departmental strategic plan
    performance measures related to the reduction in the percentage of
    physical security vulnerabilities identified at departmental facilities.
    If adopted, such outcome measures could provide valuable feedback about
    the Park Service's progress and overall effectiveness in protecting its
    physical assets.

    34According to OMB, a moderately effective rating means that a program is
    well managed and has established ambitious goals. Programs with this
    rating likely need to improve their efficiency or address other problems
    in design or management to achieve better results. See www.expectmore.gov,
    which is a Web site that was developed by OMB and federal agencies to
    provide information on PART ratings.

    35See GAO-05-790 . Before the development of this approach, Interior did
    not have a uniform comprehensive risk management approach for national
    icons and monuments-most of which are highly visible and tend to have
    public access. It relied instead on the judgment of senior officials in
    determining where resources should be directed, and the risk assessments
    completed at individual sites were done by a number of external experts
    using different methodologies. In our June 2005 report, we recognized that
    Interior had made progress in addressing this concern but recommended that
    the agency link the results of its risk assessments and related risk
    rankings to its funding priorities and develop guiding principles for
    balancing security initiatives with its core mission. Regarding the
    recommendation to develop guiding principles, Interior officials told us
    that they have not made any progress on this effort, in large part because
    resources have been dedicated to meeting the requirements of a
    presidential directive that calls for governmentwide identification
    standards and processes for federal employees and contractors.

    36Interior officials said that they consider the following characteristics
    in determining which monuments and icons are nationally significant: (1)
    asset is widely recognized to represent the nation's heritage, tradition,
    or values or is widely recognized to represent important national
    cultural, religious, historical or political significance; (2) asset's
    primary purpose is to memorialize or represent some significant aspect of
    the nation's heritage, tradition, or values, and to serve as a point of
    interest for visitors and educational activities; (3) if asset were
    successfully attacked, it would damage the American psyche and/or
    international confidence in the United States; and (4) asset is a
    monument, physical structure, or geographic site.

    37Consequence categories include casualties, economic impact, and length
    of disruption.

      Case Example: USPS Inspection Service

    The USPS Inspection Service utilizes an outcome-oriented performance
    measure to help ensure that it is progressing towards its strategic goal.
    USPS has over 38,000 facilities nationwide that collectively handle about
    700 million pieces of mail every day, and the agency serves over 7.5
    million customers daily in its post offices. Postal facilities are a
    compelling target for criminal and terrorist attacks, as evidenced by the
    anthrax attacks in 2001, which put at risk the integrity of the mail and
    the safety of USPS's employees, customers, and assets. Within USPS, the
    Inspection Service-an investigative branch whose mission is to protect the
    nation's mail system and its critical assets (i.e., employees, customers,
    and facilities)-established its first performance measure related to
    facility protection: the percentage of facilities that have high-risk
    ratings (see table 4).38 This outcome measure allows the Inspection
    Service to monitor progress toward achieving its strategic goal of
    ensuring a safe, secure, and drug-free environment.

    Table 4: Inspection Service's Performance Measure for Facility Protection

    Type of measure Performance measure Purpose
    Outcome         Percentage of USPS  To monitor the effectiveness of
                       facilities with     countermeasures through the percentage
                       high-risk ratings   of USPS facilities that score more
                                           than 800 points

    Source: GAO.

    Note: GAO analysis of USPS data.

    Specifically, this effort involves annual security surveys of facilities
    conducted by facility protection control officers, as well as periodic
    comprehensive reviews of larger core postal facilities performed by the
    Inspection Service. The data from these surveys and reviews are maintained
    in a database and used by the Inspection Service to tabulate a risk score
    based on USPS's Facility Risk Rating Model. Several data elements are
    considered to compute the composite risk score for a given facility,
    including:

    38In addition to the Inspection Service, USPS also has an Emergency
    Preparedness group that works in close conjunction with the Inspection
    Service to integrate emergency preparedness training and awareness from an
    operational perspective.

               o  crime statistics;
               o  building characteristics (e.g., the absence or presence of
               customer parking, whether the facility is attached to an adjoining
               structure);
               o  location information (e.g., the number of federal buildings
               within a 1-mile radius of the post office);
               o  operational policies and procedures (e.g., the absence or
               presence of policies related to visitors, the timely completion of
               the facility security survey within the last 12 months); and
               o  countermeasures (e.g., the absence or presence of closed
               circuit television surveillance cameras).

               Using these data elements, the maximum risk score that can be
               computed for a facility is 2,854 points. After each element at a
               particular facility is assigned a risk score, the system ranks the
               facilities according to the designated composite risk score. The
               scoring and ranking system is national and is applied to all USPS
               facilities, which allows officials to compare facilities across
               the country using standardized data to identify which buildings
               are at the highest risk level. Facilities with scores at or above
               the threshold score of 800 are considered to be high-risk.39 The
               Inspection Service reassesses its facilities every 3 years or when
               a facility undergoes any major renovations or expansions. However,
               if a facility receives a high-risk score, the facility can be
               reassessed more often to help ensure that countermeasures are
               effective and that USPS has lowered the security risks. For
               example, if a facility received a high-risk score in fiscal year
               2005, the Inspection Service will revisit that facility again in
               fiscal year 2006 to try to lower the risk score. The target is to
               reduce facility risk scores for 90 percent of the facilities that
               have a high-risk designation. At the time of our review, USPS was
               successful in meeting its performance target, according to
               Inspection Service officials.

               The Inspection Service's outcome performance measure, outlined
               above, is closely aligned with its strategic goal-to ensure a
               safe, secure, and drug-free environment-and with its strategic
               objective-to enhance the security of USPS facilities. Linking
               their performance measures and targets with their strategic goals
               and objectives in this way provides managers and staff in the
               Inspection Service with a roadmap that shows how their day-to-day
               activities contribute to achieving broader Inspection Service
               goals (see fig. 4). Inspection Service officials told us that they
               designed their security-related strategic goal and objective to
               support USPS's broader strategic goal of improving services, which
               includes activities that protect mail, employees, and customers in
               order to improve services.40

               Figure 4: Linkages between USPS Inspection Service Strategic Goals
               and Performance Measure for Facility Protection

               Although it does not use security performance measures, VA
               collects data that could be used to assess the effectiveness of
               the agency's facility protection program. VA manages a large
               health system for veterans that now includes 154 medical centers,
               875 ambulatory care and community-based outpatient clinics, and
               136 nursing homes. In 2005, more than 5 million people received
               care in VA health care facilities, and VA's outpatient clinics
               registered nearly 58 million visits. VA also operates 61 major
               veterans' benefits facilities, including 57 regional offices, 3
               records centers, and headquarters.41 While VA officials noted the
               absence of performance measures for facility protection, we found
               that the Veterans Health Administration and the Veterans Benefit
               Administration rely on physical security assessments to inform
               risk-management and resource-allocation decisions, just as other
               federal agencies and nonfederal entities do. The phases of the
               physical security assessment include defining the criticality of
               VA facilities, identifying and analyzing the vulnerabilities of
               VA's critical facilities, and identifying appropriate
               countermeasures. VA determines vulnerability based on factors such
               as facility population, building characteristics (e.g., the number
               of floors in the facility), and the presence or absence of armed
               officers and surveillance cameras. VA's assessment includes a
               procedure for scoring and prioritizing identified vulnerabilities
               at each assessed site. The objective of the security assessment is
               to identify shortcomings in the physical security of functional
               areas within critical facilities and to estimate the cost of
               mitigating the risk of disruption or termination of the facility's
               ability to provide services to veterans. For example, they assess
               the vulnerability of a facility's air system to a criminal attack.
               For each assessed functional area, a composite score and
               corresponding risk rating is assigned. The risk-rating system is
               based on a color-coded "traffic light" scheme to designate low-,
               medium-, and high-risk functional areas. The results from the
               security assessment-in particular, the risk-rating designation-are
               used to develop recommendations to mitigate the security risk and
               to prioritize and justify resource-allocation decisions. VA
               officials said that they had conducted full assessments at 18
               critical facilities and revisited these facilities a year later to
               determine progress since the assessment. At the time,
               approximately 16 percent of recommended mitigation items had been
               completed, were in progress, or had been planned for. VA officials
               said they are finalizing a database and software that would
               facilitate the tracking of facilities' responses to assessment
               recommendations. The officials said that they expect to roll out
               the database and software within a few months.

               Besides conducting security assessments, organizations can
               mitigate risk by testing their facility protection
               countermeasures. Like FPS, VA conducts inspections and tests to
               evaluate compliance with security policies and procedures and to
               help ensure that adequate levels of protection are employed. In
               some instances, such as in the VA headquarters building,
               inspections can include simulated attempts to gain unauthorized
               access to a building or to smuggle fake weapons into a building.42
               For example, within VA, scenario-based tests that are derived from
               emerging security threats are commonly used to assess police
               officers' knowledge of, and compliance with, policies and
               procedures and to evaluate preparedness in the event of an attack.
               Earlier in this report, we noted that FPS has developed a
               performance measure using similar tests in order to assess the
               effectiveness of security countermeasures, such as contract
               security guards, in mitigating risk. In addition, both VA and FPS
               conduct biannual inspections of compliance with standards and
               policies, including for physical security.

               Such measurable activity could enable the measurement of program
               outcomes, including changes in the number of unauthorized building
               entries or the number of weapons and other prohibited items
               detected as part of facility intrusion tests. Although VA
               officials told us they had not developed performance measures, we
               believe they have valuable data that can be used to measure the
               overall effectiveness of the agency's facility protection program.
               For VA, security assessments and testing activity provide useful
               feedback on how well security measures have operated and whether
               they continue to be appropriate for the future. Further, these
               evaluations could form the basis for overall evaluations of VA's
               building security program and could provide data for performance
               measurement initiatives.

               While performance measures have been used to monitor many federal
               programs, little has been done to apply performance measurement to
               physical security programs-a complex and challenging undertaking,
               since outcomes may not be quickly achieved or readily observable.
               Although we found that physical security performance measurement
               is a challenge for many organizations in the public and private
               sector, we found that the information technology (IT) security
               area has performance measurement initiatives under way. Similar to
               facility protection, IT security has been a considerable concern
               in large part because computer systems are vital to many of our
               nation's critical operations and infrastructure. The dependency on
               these systems prompted a number of congressional actions,
               including various mandates for agencies to implement security
               controls to protect information systems within the federal
               government. In compliance with these federal requirements,
               agencies must demonstrate their progress in meeting requisite
               information security requirements and report on their actual level
               of performance based on the results of annual program reviews.

               In its role as a leader on technology issues, the National
               Institute of Standards and Technology (NIST), a subagency within
               the Department of Commerce, issued a report in 2003-Security
               Metrics Guide for Information Technology Systems-to provide
               guidance on how an organization can use performance measures to
               determine the adequacy of in-place security controls, policies,
               and procedures intended to mitigate security risks.43 More
               specifically, the report provides an approach that helps managers
               decide where to invest additional security protection resources or
               how to identify and evaluate controls that are not effective. The
               guidance is the culmination of several efforts to identify a
               suitable method for measuring security and supplemented ongoing
               initiatives by OMB to help agencies develop workable measures of
               job and program performance that would hold federal employees
               accountable for their IT security responsibilities. In addition to
               providing practical examples of security performance measures that
               can be readily used or modified to meet agency-specific needs, the
               report provides a detailed description of how performance
               measurement is being approached in the IT security area and
               addresses the following areas: (1) the roles and responsibilities
               of agency staff at all levels, (2) the benefits of using
               performance measures, and (3) an overview of the performance
               measures development and implementation process.

               The NIST report advocates the use of measurable performance
               measures based on IT security performance goals and objectives. In
               turn, the report describes performance measures as tools designed
               to facilitate decision making and improve performance and
               accountability through the collection, analysis, and reporting of
               relevant performance-related data. NIST describes three types of
               performance measures-implementation, efficiency and effectiveness,
               and impact-that can be used to measure progress (see table 5).
               Although NIST uses different terminology to describe the three
               types of performance measures, they are similar to the output and
               outcome measures that we have advocated for use in monitoring and
               reporting program accomplishments. The NIST report cautions that
               the type of performance measures that can realistically be
               obtained and used for performance improvement depends on the
               maturity of the security program. According to NIST, in the early
               stages of establishing a security program, the focus tends to be
               on developing security policies and procedures, and beginning to
               ensure that security controls are implemented. In such an
               environment, an appropriate performance measure would be one that
               focuses on implementation, such as the percentage of information
               systems with approved security plans. In contrast, a more mature
               security program may evolve to measure the efficiency and
               effectiveness of security controls and the impact of these
               controls on the organization's mission. In such cases, the
               performance measures may concentrate on the evidence and results
               of testing.

    39Inspection Service officials told us that they chose 800 as the
    threshold score because they wanted to further review the security of the
    top 10 percent of the most vulnerable facilities. When this performance
    measure was implemented, the top 10 percent of most vulnerable facilities
    scored above 800. While this threshold remains the same today, the
    threshold score may decrease or increase over time due to implementation
    of countermeasures and changes in risk elements. To date, the Inspection
    Service has decided not to change the threshold score in order to keep the
    scoring methodology consistent.

    40In its Strategic Transformation Plan 2006-2010, USPS has identified four
    strategic goals: (1) generate revenue; (2) reduce costs; (3) achieve
    results with a customer-focused, performance-based culture; and (4)
    improve service.

      Case Example: Department of Veterans Affairs

    41VA officials noted that the majority of the space occupied by VA's
    Veterans Benefit Administration is in GSA-held buildings. As such, FPS is
    responsible for security at these facilities.

    Federal Guidance for Developing and Using Performance Measures Exists for IT
    Security, but Not for Physical Security

    42VA officials noted that most Veterans Health Administration buildings
    are designed for maximum public access and therefore do not have
    magnetometers or metal detectors, so such tests are not conducted in those
    facilities. In addition, many Veterans Benefit Administration facilities
    are in GSA buildings, so FPS is responsible for providing security and
    conducting related tests.

    43National Institute of Standards and Technology, Security Metrics Guide
    for Information Technology Systems, NIST Special Publication 800-55 (July
    2003).

    Table 5: Types of Information Technology Security Performance Measures
    Described by NIST

    Type of measure Performance measure             Purpose
    Implementation  Percentage of systems with      Assess the extent to which
                       approved security plans and the security plans and
                       percentage of systems with      password policies have
                       password policies configured as been documented and
                       required                        implemented to support the
                                                       security program
    Efficiency and  Percentage of crackable         Evaluate the results of
    effectiveness   passwords within a predefined   security controls that
                       time threshold                  have been implemented;
                                                       validate whether security
                                                       controls, as described in
                                                       the security plan, are
                                                       effective in protecting
                                                       the organization's assets
    Impact          Quantify incidents by type      Measure the impact of
                       (e.g., root compromise,         training on security
                       password compromise, malicious
                       code, denial of service) and
                       correlate incident data with
                       the percentage of trained users
                       and system administrators

    Source: NIST.

    The guidance goes beyond extolling the virtues of using performance
    measures and illustrates the place of IT security within a larger
    organizational context, provides a roadmap for how to develop and
    implement a performance measurement program, and includes practical
    examples of performance measures. According to NIST, the performance
    measures that are ultimately selected can be useful not only for measuring
    performance, identifying causes of unsatisfactory measurements, and
    pinpointing improvement areas, but also for facilitating continuous policy
    implementation, effecting security policy changes, and redefining goals
    and objectives. NIST notes that successful implementation of a security
    performance measurement program can also assist agencies in meeting OMB's
    annual requirements to report the status of agency IT security programs.
    In addition to providing examples of performance measures, some of which
    are required by OMB, the report also includes a standardized template that
    describes the various data elements that should be documented (see fig.
    5). The data elements include:

               o  Performance goal: States the desired results of implementing
               security control objectives that are measured by the metric.
               o  Performance objective: States the actions that are required to
               accomplish the performance goal.
               o  Metric: Defines the metric by describing the quantitative
               measurements it provides.
               o  Purpose: Describes the overall functionality obtained by
               collecting the metric; includes whether a metric will be used for
               internal performance measurement or for external reporting, what
               insights are hoped to be gained from the metric, and whether
               regulatory or legal lessons exist for collecting a specific metric
               if applicable.
               o  Implementation evidence: Includes indirect indicators that
               validate that the activity is being performed and causation
               factors that may point to the causes of unsatisfactory results for
               a specific metric.
               o  Frequency: Establishes time periods for collecting data that is
               used for measuring changes over time.
               o  Formula: Describes the calculation to be performed that results
               in a numeric expression of the metric.
               o  Data source: Identifies the location of the data to be used in
               calculating the metric (e.g., databases, tracking tools,
               organizations, or specific roles within the organization that can
               provide required information).
               o  Indicators: Provide information about the meaning of the metric
               and its performance trend; state the performance target and
               indicate what trends would be considered positive in relation to
               the performance target.

               The NIST report notes that the universe of possible performance
               measures, based on policies and procedures in place in the
               organization, will be quite substantial and that the final
               performance measurement set selected for initial implementation
               should relate to high-priority areas, use data that can be
               realistically obtained, and measure processes that already exist
               and are relatively stable. The guidance further states that
               performance measures can be developed and selected using a phased
               approach. This approach identifies short-, mid-, and long-term
               measures where the time frame in which these measures are
               implemented depends on a combination of system-level
               effectiveness, performance measure priority, data availability,
               and process stability. The NIST report also notes that, once
               applicable performance measures have been identified, they should
               be documented using a standardized template (see figure 5).
               Standardizing the reporting process is particularly useful in
               cases where the reporting process within an organization is
               inconsistent. Such practices, among others, can help ensure the
               success of a performance measurement program.

    Figure 5: Sample Standardized Performance Measurement Data Form

      Federal Agencies Have Received Minimal Guidance on Using Performance
      Measurement for Facility Protection Programs

    We have previously reported that, at the agencywide level, agencies face
    obstacles in developing meaningful, outcome-oriented performance goals and
    in collecting data that can be used to assess the true impact of facility
    protection efforts. GPRA emphasizes measuring the results of products and
    services delivered by a federal program (i.e., outcomes). For programs
    that have readily observable results or outcomes, performance measurement
    may provide sufficient information to evaluate the effectiveness of
    facility protection efforts. Yet in some programs, such as facility
    protection, outcomes are not quickly achieved or readily observable, or
    their relationship to the program is often not clearly defined. In such
    cases, more in-depth program evaluations, in addition to performance
    measurement, may be needed to examine the extent to which a program is
    achieving its objectives.

    While federal agencies have made some progress developing performance
    measures for facility protection, we noted that the emphasis is on using
    output measures that monitor program activity rather than outcome measures
    that assess the overall impact of program activity. This lack of outcome
    measures leaves agencies with insufficient information to determine
    whether security activities are effective and to evaluate whether the
    benefits of security investments justify their costs. We have previously
    reported that various security program outputs-such as conducting
    patrols-may have contributed to improved security, but that using them as
    performance measures may not systematically target areas of higher risk
    and may not result in the most effective use of resources, because these
    measures are not pointed toward outcomes. Such output measures do not
    provide an indication of what these activities are accomplishing. By
    contrast, outcome measures that are clearly tied to results would indicate
    the extent of progress made and help identify the security gaps that still
    remain.44 Without more information on security program outcomes, agencies
    do not know the extent to which security enhancements have improved
    security or reduced federal facilities' vulnerability to acts of terrorism
    or other forms of violence. In addition, there is some inconsistency in
    the types of activities that are being monitored and used as indicators of
    an agency's progress in fulfilling its facility protection
    responsibilities. If agencies use inconsistent approaches to performance
    measurement, decision makers could be at risk of having incomparable
    performance information to determine funding priorities within and across
    agencies.

    44 GAO-06-91 .

    Echoing what organizations outside the U.S. federal government told us,
    some agency security officials said it was challenging to measure the
    impact that various approaches have on actually improving security. Some
    agency officials also noted that resources for performance measurement
    initiatives were scarce. Additionally, the availability of information
    needed for applying performance measurement to facility protection is
    somewhat limited. More generally, with the exception of DHS, the agencies
    that we reviewed do not view security as their primary mission, and some
    agencies are faced with competing demands for limited resources to
    accomplish broader agency goals. In such an environment, security must be
    integrated using scarce resources.

    In spite of the inherent difficulty in measuring facility protection
    performance, and the considerable emphasis on doing so, agencies have
    minimal guidance on how to accomplish this. There is, however, broad
    guidance for the protection of critical infrastructures, which includes
    government facilities. Using a risk-based approach, the Draft National
    Infrastructure Protection Plan (NIPP) was developed to provide an
    integrated, comprehensive approach to addressing physical, cyber, and
    human threats and vulnerabilities.45 As part of the NIPP, DHS officials
    have provided guidance and collected information on core performance
    measures-which are common measures that can be broadly applied to all
    protection programs for critical infrastructures and key assets. These
    measures are mostly process/input and output oriented, and DHS officials
    noted that they hope to develop outcome measures as the program matures.
    The NIPP, however, does not provide or collect information on specific
    performance measures related to the protection of federal facilities.
    Rather, it notes that FPS-the agency assigned responsibility for
    implementing the NIPP framework and guidance in the government facilities
    sector-will develop such performance measures. Separately, OMB issued a
    memorandum in June 2004 that reported it was working with agencies on
    initiatives related to physical security reporting requirements noted in
    Homeland Security Presidential Directive Number 7 (HSPD-7).46 The
    memorandum instructed each agency to disclose the performance measures it
    had designed and implemented to measure outputs and outcomes. However, OMB
    did not provide specific guidance or standards and instead directed
    agencies to use DHS guidance-related to the NIPP-that does not specify
    measures for facility protection.

    45DHS released the first Draft NIPP for public comment in November 2005.
    In January 2006, DHS released a revised Draft NIPP that incorporated some
    of the comments it had already received.

    46As mentioned earlier, HSPD-7 establishes a national policy for federal
    departments and agencies to identify and prioritize U.S. critical
    infrastructures and key assets so that they can be protected from
    terrorist attacks.

    By contrast, the IT security performance measurement guidance issued by
    NIST includes information on: (1) clearly defining roles and
    responsibilities for relevant stakeholders; (2) establishing security
    goals and objectives; (3) identifying and implementing performance
    measures and performance targets; and (4) using measures that are unique
    to IT security to assess the impact of IT security efforts. One security
    official from the gaming industry said that IT security performance was
    somewhat easier to evaluate than physical security performance because it
    is possible to directly monitor the number of attempted IT security
    breaches. A foreign government agency we interviewed is farther along in
    developing standards and performance measures for IT security than for
    physical security. In general, IT security approaches are slightly more
    standardized than physical security because the field is newer than
    physical security and because organizations had to work together to
    prepare for possible complications in the year 2000 (Y2K). Despite such
    differences between IT and physical security performance measurement, some
    of the performance measurement guidance could be applicable to physical
    security situations.

    ISC is a body that addresses governmentwide security policy issues and,
    like NIST, is well positioned to develop guidance and promote performance
    measurement. Executive Order 12977 calls for ISC to play an oversight role
    in implementing appropriate security measures in federal facilities and
    taking actions that would enhance the quality and effectiveness of
    security in federal facilities. As we reported in November 2004, ISC has
    already made progress in coordinating the federal government's facility
    protection efforts through activities such as developing security policies
    and standards for leased space, improving information sharing, and
    coordinating the development of a security database of all federal
    facilities.47 The ISC Chair told us that he supports the use of
    performance measurement as a means of strengthening federal facility
    protection efforts.

    47See GAO-05-49 .

                                      Conclusions

    Given their competing priorities and limited security resources, U.S.
    federal agencies could benefit from specific performance measurement
    guidance and standards for facility protection to help them address the
    challenges they face and help ensure that their physical security efforts
    are achieving the desired results. While some of these agencies have
    implemented performance measures to monitor their security programs'
    outputs, fewer have developed outcome measures to assess the extent to
    which security enhancements have improved security or reduced their
    facilities' vulnerability to acts of terrorism or other forms of violence.
    Without a means of comparing security effectiveness across facilities,
    particularly program outcomes, the U.S. government is open to the risk of
    either spending more money for less effective physical security or
    investing in the wrong areas. The output measures that federal agencies
    have developed provide an indication of what their security activities are
    accomplishing but do not indicate the extent of progress made or help
    identify the security gaps that still remain, as outcome measures would.
    Fundamentally, performance measurement helps ensure accountability, since
    it enables decision makers to isolate certain activities that are
    hindering an agency's ability to achieve its strategic goals. Performance
    measurement can also be used to prioritize security needs and justify
    investment decisions so that an agency can maximize available resources.
    Over time, a thorough performance measurement approach could allow the
    federal government to manage the risks to federal facilities both within
    and across agencies. Recognizing the unique nature of U.S. federal
    agencies' missions, some uniformity in measuring performance in facility
    protection efforts could facilitate comparisons across agencies.

    Organizations outside of the U.S. government-including private-sector
    entities as well as state, local, and foreign government agencies-have
    developed and are using performance measures for facility protection, and
    their knowledge and experience could be helpful to U.S. federal agencies
    in developing and refining their own performance measures. Likewise,
    because the application of performance measures to facility protection can
    be challenging, many nonfederal organizations are looking to U.S.
    government agencies for assistance and leadership. Some U.S. federal
    agencies are already collecting data that could be used for measuring
    security performance, and they currently have guidance for measuring
    information technology security, but not physical security. The U.S.
    federal government has provided guidance and collected information on a
    set of common measures that can be broadly applied to all protection
    programs for critical infrastructures and key assets, and agencies will be
    required to report on additional security performance measures that are
    sector-specific. With regard to federal facilities, the ISC, in serving as
    the central coordinator for U.S. agencies' federal facility protection
    efforts, is well positioned to develop and promote performance measurement
    guidance and standards for physical security, and could look to
    information technology security as a model to follow. In turn, it could
    draw from examples of performance measurement we identified in the private
    sector and foreign government agencies. Federal agencies could
    subsequently follow the guidance and standards to evaluate their actions,
    identify lessons learned, and develop strategies for overcoming any
    challenges in developing and using performance measures for facility
    protection. Because of the ever-changing nature of security threats and
    new security technologies and countermeasures, such guidance and standards
    would need to be periodically reviewed and updated. The development of
    guidance and standards for facility protection could help ensure uniform
    application of performance measurement so that the U.S. federal
    government, particularly its largest real-property-holding agencies, would
    be accountable for its facility protection programs and would be able to
    demonstrate that security investments are producing a return, both within
    and across agencies, in terms of better-protected facilities.

                          Recommendations for Executive Action

    To ensure that useful information is available for making decisions about
    the allocation of resources for, and the effectiveness of investments in,
    the protection of federal facilities, we recommend that the Secretary of
    Homeland Security direct the Chair of ISC to do the following:

               o  as part of ISC's efforts to support DHS in developing
               sector-specific performance measures for the security of federal
               government facilities, establish guidance and standards, with
               input from ISC member agencies, for measuring performance in
               facility protection-with a particular focus on developing outcome
               measures;
               o  communicate the established guidance and standards to the
               relevant federal agencies; and
               o  ensure that the guidance and standards are regularly reviewed
               and updated.

               We provided a draft of this report to DHS, GSA, USPS, VA, and
               Interior for their official review and comment. DHS concurred with
               the report's overall findings and recommendations. DHS comments
               are contained in appendix III. USPS and VA concurred with the
               report's findings. In addition, DHS and USPS provided separate
               technical comments, which we incorporated into the final report
               where appropriate. GSA notified us that they had no comments on
               this report.

               Interior, while generally agreeing with the report's findings,
               suggested that an agency-by-agency assessment of each federal
               agency's facility vulnerabilities would be more effective than a
               cross-agency facility protection performance measure. We agree
               that identifying and monitoring vulnerabilities is important, but
               believe that it is also important for decision makers to have
               comparable information about the relative security performance of
               facilities within an agency as well as across the federal
               government. Interior also expressed concern that a more public
               viewing of agency facility protection performance could reveal
               weaknesses or vulnerabilities that could be exploited. We agree
               that this could be a concern but leave the development of
               guidelines for using and protecting this information to ISC and
               its member agencies. Interior also provided technical comments,
               which we incorporated. Comments from Interior and our evaluation
               can be found in appendix IV.

               As agreed with your office, unless you publicly announce the
               contents of this report earlier, we plan no further distribution
               until 30 days from the report date. At that time, we will send
               copies of this report to other interested congressional committees
               and the Secretaries of the Interior, Homeland Security, and
               Veterans Affairs; the Administrator of GSA; and the Postmaster
               General of the U.S. Postal Service. We will also make copies
               available to others upon request. In addition, the report will be
               available at no charge on the GAO Web site at http://www.gao.gov.

               If you have any questions regarding this report, please contact me
               on (202) 512-2834 or at goldsteinm@gao.gov . Contact points for
               our Offices of Congressional Relations and Public Affairs may be
               found on the last page of this report. GAO staff who made major
               contributions to this report are listed in appendix V.

               Sincerely yours,

               Mark L. Goldstein Director, Physical Infrastructure Issues

               Appendix I: Objectives, Scope, and Methodology

    			  The objectives of our report were (1) to identify examples of
               performance measures for facility protection being used by
               selected organizations outside of the federal government-including
               private-sector entities, state and local governments, and foreign
               governments; and (2) to determine the status of U.S. federal
               agencies' efforts to develop and use performance measures as part
               of their facility protection programs.

               To identify examples of performance measures for facility
               protection being used by selected organizations outside the
               federal government, we interviewed representatives from the
               private sector, U.S. state and local governments, and foreign
               governments. With respect to the private sector, we asked a number
               of umbrella organizations to identify industries that are likely
               to utilize performance measures for facility protection and known
               leaders in the security performance measurement area. These
               umbrella organizations included ASIS International, Real Estate
               Roundtable, Financial Services Roundtable, Financial Services
               Information Sharing and Analysis Committee, International Facility
               Management Association, and National Association of Industrial and
               Office Properties. GAO staff also attended the annual ASIS
               International Conference in 2005. Some of these entities stated
               that the gaming and finance industries would be the most
               appropriate to review, since these industries have invested
               significantly in the quality of their security efforts. As a
               result, we interviewed officials from four gaming entities and
               five major financial services organizations. To maintain the
               organizations' security and the confidentiality of proprietary
               information, we do not identify specific organizations in this
               report.

               For the gaming industry, a member of the Real Estate Roundtable
               provided a contact who was known to be active in physical security
               testing and performance measurement. This individual then arranged
               a joint interview for us with a number of gaming entities. Some of
               the representatives present at the interview were also members of
               the Las Vegas Security Chiefs Association or ASIS International
               Gaming and Wagering Council. The five financial services
               organizations we interviewed were selected because they (1) were
               considered to be leaders in their industry; (2) were recommended
               by others within the industry; (3) were members of ASIS
               International, the largest organization supporting security
               professionals; or (4) have had prior security concerns related to
               threats of terrorism.

               To determine if U.S. state and local governments have developed
               performance measures for facility protection, we attempted to
               contact 10 state and 10 local governments. For state governments,
               we selected the 10 states receiving the most funding from the
               Department of Homeland Security's (DHS) State Homeland Security
               Program grant in fiscal year 2005. For local governments, we
               selected the 10 local governments/urban areas receiving the most
               funding from DHS's Urban Areas Security Initiative grant in fiscal
               year 2005.1 Of the 20 state and local governments we attempted to
               contact, we were able to obtain information from officials from 17
               of them. While all 17 of these state and local governments were
               engaged in facility protection efforts, only a few had developed
               performance measures to evaluate the effectiveness of these
               efforts. Table 6 shows a listing of these state and local
               governments. The agencies we approached within each of the state
               and local governments were often, but not always, the agencies
               responsible for real property or policing/security. Some of the
               state and local governments we attempted to contact were also
               identified by the Government Accounting Standards Board as having
               performance measurement initiatives on a variety of their
               organizations, departments, and projects.

               Table 6: U.S. State and Local Governments Contacted

               Source: GAO.

               aFor the purposes of this report, Washington, D.C., was treated as
               a local government.

               For our work with foreign governments, we conducted international
               site visits in three foreign countries-Australia, Canada, and the
               United Kingdom-where we interviewed a number of government
               agencies and organizations about their use of performance measures
               for facility protection. (Table 7 shows a listing of each of these
               agencies.) We selected these three countries for site visits
               because they are known to have experience with threats of
               terrorism and because they have been identified by the Government
               Accounting Standards Board as having performance measurement
               initiatives, not necessarily for facility protection but for
               government initiatives in general. We also spoke with
               representatives from a number of other foreign governments. While
               these other governments have facility protection efforts in place,
               they said they did not use performance measures to assess the
               effectiveness of these efforts. Furthermore, officials from some
               of these countries told us that they look to the United States for
               guidance on a number of issues relating to facility protection,
               including how to measure effectiveness. For such reasons, these
               countries were not highlighted in this report.

               Table 7: Foreign Government Agencies and Organizations Visited

               Source: GAO.

               In addition to interviewing officials from the nonfederal entities
               identified above, we reviewed relevant documentation obtained from
               these organizations, previous GAO reports, and performance
               measurement and facility protection literature from ASIS
               International and other sources.

               For the second objective-to determine the status of U.S. federal
               agencies' efforts to develop and use performance measures as part
               of their facility protection programs-we interviewed selected
               officials from the major civilian real property holding agencies.
               These agencies include the General Services Administration (GSA),
               the United States Postal Service (USPS), the Department of
               Veterans Affairs (VA), and the Department of Interior (Interior).
               GSA acknowledged the need to measure the performance of facility
               protection efforts; however, for most facility protection issues,
               they defer to the Federal Protective Service (FPS) within DHS.
               Because FPS is responsible for protecting all GSA buildings, we
               also interviewed officials from FPS. For each of the selected
               federal agencies, we reviewed agency strategic and performance
               plans, security goals, performance reports, and other relevant
               documentation provided to us. We also interviewed the Executive
               Director of the Interagency Security Committee (ISC)-a DHS-led
               committee that is tasked with coordinating federal agencies'
               facility protection efforts. Finally, we reviewed a number of
               national strategies and presidential directives; previous GAO
               reports; and relevant reports by the Office of Management and
               Budget, the Congressional Budget Office, the Congressional
               Research Service, and other government entities. We also reviewed
               laws and authorities related to facility protection.

               It is important to note that the private-sector entities, U.S.
               state and local governments, and foreign governments selected for
               our review are not representative of the universe of such
               organizations. Furthermore, GAO has not evaluated the robustness
               and quality of the performance measures cited in this report.
               Rather, these measures are simply a compilation of what we have
               gathered from the nonfederal and federal entities we have
               interviewed. Additionally, the performance measures identified in
               this report may not include all performance measures relating to
               the protection of federal facilities. We used our judgment to
               classify the performance measures into process/input, output, and
               outcome measures according to our definitions, but these
               performance measures could be classified differently depending on
               the performance measurement goals or objectives used by an
               organization.

               Also, ISC has identified GAO as an associate member, which
               includes the ability to serve on ISC subcommittees. No GAO staff
               member, however, serves on any subcommittee. Furthermore, no GAO
               staff member actively participates in ISC meetings or contributes
               to decisions. Rather, GAO's role on ISC is only to observe
               proceedings and obtain ISC information distributed to the other
               ISC members. Because of GAO's observational role, our independence
               in making recommendations involving ISC and in completing this
               engagement was maintained.

               Officials from nonfederal and federal entities provided much of
               the information used in this report. In most cases where officials
               provided their views as representatives of their organizations, we
               corroborated the information with other officials or with
               documentation provided to us. We requested official comments on
               this report from DHS, GSA, USPS, VA, and Interior. Furthermore,
               when we used examples from the private sector, state and local
               governments, foreign governments, and the National Institute of
               Standards and Technology (NIST), we provided the respective entity
               an opportunity to review relevant portions of the report and offer
               comments, thus ensuring the validity of our reporting. We
               conducted site visits and interviews from July 2005 through
               January 2006. We conducted our work from May 2005 through April
               2006 in accordance with generally accepted government auditing
               standards.

               Appendix II: Examples of Performance Measures Used by Selected
    			  Organizations outside of the Federal Government

    			  The performance measures below were provided by the selected
               organizations we interviewed outside of the federal government. We
               did not evaluate the quality of the performance measures, and we
               used our judgment to classify them according to the following
               definitions of performance measures:

               o  Output measures focus on the quantity of direct products and
               services a program delivers and the characteristics of those
               outputs, including efficiency, cost-effectiveness, timeliness,
               quality, and customer service.
               o  Outcome measures provide information on the results of the
               direct products and services a program has delivered.
               o  Process/input measures address the type or level of program
               activities an organization conducts and the resources used by the
               program.

               The performance measures could be classified differently depending
               on the performance measurement goals or objectives used by an
               organization.

               Source: GAO.

               Note: GAO analysis of data from state, local, and foreign
               government agencies and private-sector organizations.

               .

               The following are GAO's comments on Interior's letter dated May
               15, 2006.

               Appendix V: GAO Contact and Staff Acknowledgments

    			  GAO Contact

    			  Staff Acknowledgments

    			  Mark Goldstein (202) 512-2834 or goldsteinm@gao.gov

               Other key contributors to this report were Roshni Dave, Tamera
               Dorland, Brandon Haller, Anne Izod, Jessica Lucas-Judy, Susan
               Michal-Smith, David Sausville, Scott Tessier, Candice Wright, and
               Dorothy Yee

               The Government Accountability Office, the audit, evaluation and
               investigative arm of Congress, exists to support Congress in
               meeting its constitutional responsibilities and to help improve
               the performance and accountability of the federal government for
               the American people. GAO examines the use of public funds;
               evaluates federal programs and policies; and provides analyses,
               recommendations, and other assistance to help Congress make
               informed oversight, policy, and funding decisions. GAO's
               commitment to good government is reflected in its core values of
               accountability, integrity, and reliability.

               The fastest and easiest way to obtain copies of GAO documents at
               no cost is through GAO's Web site ( www.gao.gov ). Each weekday,
               GAO posts newly released reports, testimony, and correspondence on
               its Web site. To have GAO e-mail you a list of newly posted
               products every afternoon, go to www.gao.gov and select "Subscribe
               to Updates."

               The first copy of each printed report is free. Additional copies
               are $2 each. A check or money order should be made out to the
               Superintendent of Documents. GAO also accepts VISA and Mastercard.
               Orders for 100 or more copies mailed to a single address are
               discounted 25 percent. Orders should be sent to:

               U.S. Government Accountability Office 441 G Street NW, Room LM
               Washington, D.C. 20548

               To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
               (202) 512-6061

               Contact:

               Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
               fraudnet@gao.gov Automated answering system: (800) 424-5454 or
               (202) 512-7470

               Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400
               U.S. Government Accountability Office, 441 G Street NW, Room 7125
               Washington, D.C. 20548

               Paul Anderson, Managing Director, AndersonP1@gao.gov (202)
               512-4800 U.S. Government Accountability Office, 441 G Street NW,
               Room 7149 Washington, D.C. 20548

    			  GAO Comments

                            1. Interior suggested that an agency-by-agency
                            assessment of each federal agency's facility
                            vulnerabilities would be more effective than a
                            cross-agency facility protection performance measure.
                            We agree that identifying vulnerabilities and
                            monitoring efforts to address those vulnerabilities
                            is a useful part of an agency's comprehensive
                            facility protection program. For example, the
                            Department of Veterans Affairs conducts vulnerability
                            assessments, and one Australian government agency we
                            interviewed monitors the effect of different security
                            investments on its facilities' risk ratings (which
                            typically involve threat and vulnerability factors).
                            However, we believe it is also important for decision
                            makers to have comparable information about the
                            relative security performance of facilities within an
                            agency, rather than just in one bureau or service, as
                            well as across the federal government. Such
                            information could help reduce the risk of spending
                            more money for less effective physical security or
                            investing in the wrong areas.
                            2. Interior expressed concern that a more public
                            viewing of agency facility protection performance
                            could reveal weaknesses or vulnerabilities that could
                            be exploited. We agree that this could be a concern,
                            but choose to leave the development of guidelines for
                            using and protecting such information to the
                            Interagency Security Committee and its member
                            agencies.

                           Agency Comments and Our Evaluation

    1The State Homeland Security Program and Urban Areas Security Initiative
    grants can be applied to a number of homeland security efforts, including
    facility protection. See U.S. Department of Homeland Security, Fiscal Year
    2005 Homeland Security Grant Program, Program Guidelines and Application
    Kit.

    Organization           Location
    U.S. state governments California

                              Florida

                              Georgia

                              Illinois

                              Michigan

                              New Jersey

                              New York

                              Ohio

                              Pennsylvania

                              Texas
    U.S. local governments Boston, Mass.

                              Detroit, Mich.

                              Washington, D.C.a

                              Los Angeles, Calif.

                              New York, N.Y.

                              Philadelphia, Pa.

                              San Francisco, Calif.

    Location       Organization
    Australia      Airservices Australia

                      Attorney-General's Department

                      Commonwealth Scientific and Industrial Research
                      Organization

                      Customs Service

                      Department of Defence

                      Department of Foreign Affairs and Trade

                      Federal Police

                      National Audit Office

                      Taxation Office
    Canada         Bank of Canada

                      Corps of Commissionaires

                      Department of National Defence

                      National Gallery

                      Office of Auditor General

                      Public Works and Government Services Canada

                      Royal Canadian Mounted Police

                      Treasury Board
    United Kingdom Cabinet Office

                      Department for Transport

                      Foreign and Commonwealth Office

                      Home Office

                      National Infrastructure Security Coordination Centre,
                      Security Service

                      National Security Advice Centre, Security Service

                      Office for Civil Nuclear Security

    Appendix II: Examples of Performance Measures Used by Selected
    Organizations outside of the Federal Government Appendix II: Examples of
    Performance Measures Used by Selected Organizations outside of the Federal
    Government

    Output
    Number of risk assessments performed

    New security projects

          o  Security checklist completed during planning stages
          o  Security officials consulted

    Number of security requests received

          o  Security report requests
          o  New access badge requests
          o  Requests for changes to existing badges

    Security clearance

          o  Number of background screenings completed
          o  Average time to process background screenings
          o  Average number of days to process security clearances
          o  Number of overdue security clearances by more than 4 weeks
          o  Cost per security clearance
          o  Percentage of officers/contractors who hold sufficient level of
          security clearance when compared to their duties

    Alarm systems

          o  Responded to and cleared
          o  Alarms with unique responses (i.e., alarms requiring guards to
          respond in person)
          o  Failed to arm Number of police incidents/reports filed Number of
          threats
          o  Against employees
          o  Against facilities

    Security incident reaction/response

          o  Number of avoidable incidents detected
          o  All significant investigations completed within 45 days

    Compliance with security policies and standards

          o  Number of exceptions reviewed
          o  Number of significant policy breaches
          o  Surveillance and communication systems are compliant with standards
          o  Entry/access control systems are compliant with standards
          o  Security staff are fulfilling their contract obligations

    Customer/client satisfaction

          o  Staffing-training, professional appearance, professional behavior,
          turnover rate, supervision
          o  Security reporting-accuracy, timeliness, use of correct forms
          o  Management-responsiveness, understanding of issues, availability,
          number of personal contacts

    Timely delivery of security alerts and briefings

    Percentage of alarms responded to within 20 minutes during nonpublic
    service hours

    Increased attendance at training courses for security officers

    Number of new employees, contractors, and consultants who have not
    attended a security awareness session within 4 weeks of receiving their
    identification pass

    Percentage of security guards in compliance with licensing standards
    within a 7-day period

    All scheduled audit and compliance reports completed in 14 days
    Outcome
    Change in the total number of security-related incidents

          o  Accident
          o  Assault
          o  Burglary
          o  Organization assets
          o  Personal assets
          o  Drugs/Alcohol
          o  Extortion
          o  Fire
          o  Fraud Referral
          o  Harassment
          o  Larceny/Theft
          o  Malicious damage
          o  Public disorder
          o  Robbery
          o  Suspicious activity
          o  Terrorism
          o  Vandalism
          o  Workplace violence

    Evidence of damage to building and facilities

    Evidence of harm to staff or tenants

    Change in risk rating resulting from countermeasures deployed

    Security policies and practices receive favorable comment from security
    audit program

    Agency stakeholders view agency as a safe custodian of allocated resources
    and assets
    Process/Input
    Number of facilities being protected (including types and locations)

    Number of security staff

    Number of security guards/security escorts

    Personal security arrangements for after-hours entry/access

    Perimeter security

          o  Assessment of entry/exit points
          o  Serviceability of perimeter security equipment (locks, door frames,
          security signs)
          o  Sufficiency of perimeter lighting
          o  Presence of obstructions, waste containers/material, combustibles,
          other risk factors
          o  Evidence of vandalism, malicious damage, or other criminal activity
          o  Maintenance schedules

    Number of security clearances undertaken Number of training courses and
    drills conducted

    Security threats and general risks discussed at management forum and
    disseminated to all levels of agency staff Security spending per square
    foot

    Appendix III: Comments from the Department of Homeland Security Appendix
    III: Comments from the Department of Homeland Security

    Note: GAO comments supplementing those in the report text appear at the
    end of this appendix.

    Appendix IV: Comments from the Department of the Interior Appendix IV:
    Comments from the Department of the Interior

    See comment 1.

    See p. 34.

    References to BLM have been deleted.

    See comment 2.

    (543129)

    GAO's Mission

    Obtaining Copies of GAO Reports and Testimony

    Order by Mail or Phone

    To Report Fraud, Waste, and Abuse in Federal Programs

    Congressional Relations

    Public Affairs

    www.gao.gov/cgi-bin/getrpt? GAO-06-612 .

    To view the full product, including the scope

    and methodology, click on the link above.

    For more information, contact Mark Goldstein at (202) 512-2834 or
    goldsteinm@gao.gov.

    Highlights of GAO-06-612 , a report to the Chairman, Committee on
    Government Reform, House of Representatives

    May 2006

    HOMELAND SECURITY

    Guidance and Standards Are Needed for Measuring the Effectiveness of
    Agencies' Facility Protection Efforts

    The protection of U.S. federal facilities has become an important concern
    due to the ongoing threat of terrorism. The General Services
    Administration (GSA), U.S. Postal Service (USPS), and the Departments of
    Veterans Affairs (VA) and Interior (Interior) hold the most domestic,
    nonmilitary property. Additionally, the Department of Homeland Security
    (DHS) is responsible for the protection of GSA facilities. DHS chairs the
    Interagency Security Committee (ISC), which is tasked with coordinating
    federal agencies' facility protection efforts. The need to better protect
    federal facilities, as well as federal budget constraints, have prompted
    the need for these agencies to measure the performance of their facility
    protection efforts. GAO's objectives were (1) to identify examples of
    performance measures for facility protection being used by selected
    organizations outside of the federal government; and (2) to determine the
    status of U.S. federal agencies' efforts to develop and use performance
    measures as a part of their facility protection programs.

    What GAO Recommends

    GAO is recommending that the Secretary of DHS direct ISC to establish
    guidance and standards for measuring performance in federal government
    facility protection. DHS agreed with the findings and recommendations in
    this report.

    GAO found a range of examples of performance measures that organizations
    outside the U.S. government-including private-sector entities, state and
    local governments, and foreign government agencies-have developed that,
    collectively, indicate whether facility protection efforts are achieving
    results (see figure below). These organizations use security-related
    performance measures to help improve security, make decisions about risk
    management and resource allocation, and hold employees accountable for
    whether a program meets its security goals and objectives. However, many
    of the organizations said that developing and using these measures can be
    challenging and that they look to the U.S. government for assistance and
    leadership in developing standards and guidance for facility protection.

    Performance Measurement Types, Examples, Uses, and Results

    Note: Output measures focus on the direct product/services delivered by a
    program. Outcome measures provide information on the results of
    products/services.

    We found that some bureaus and services within DHS (for GSA properties),
    USPS, and Interior are using security performance measures, while VA and
    other bureaus and services within the three agencies collect data that
    could be used to measure security performance. Agencies that have
    performance measures use them to ensure adequate protection at individual
    facilities, make risk management decisions, and evaluate program
    effectiveness. However, agencies face challenges-similar to those cited by
    nonfederal entities-in further developing and using security performance
    measures. Currently, there is no governmentwide guidance or standards on
    measuring facility protection performance to help federal agencies address
    these challenges. This differs from information technology security, where
    agencies have detailed, governmentwide guidance for developing and using
    performance measures. Without effective performance measurement data,
    decision makers may have insufficient information to evaluate whether
    their investments have improved security or reduced federal facilities'
    vulnerability to acts of terrorism or other forms of violence. ISC is
    uniquely positioned to develop and disseminate guidance and standards for
    measuring the performance of federal government facility protection
    efforts.
    *** End of document. ***