    Agency Performance Plans: Examples of Practices That Can Improve
    Usefulness to Decisionmakers (Letter Report, 02/26/99,
    GAO/GGD/AIMD-99-69).

    Pursuant to a congressional request, GAO: (1) identified and described
    practices that were applied in some federal agencies' 1999 performance
    plans that might, if consistently applied, improve the usefulness of all
    agencies' annual performance plans; and (2) provided examples from
    agencies' fiscal year 1999 performance plans that illustrate each
    practice.

    GAO noted that: (1) opportunities to improve the usefulness of annual
    plans include: (a) creating a set of performance goals and measures that
    addresses important dimensions of program performance and balances
    competing priorities; (b) using intermediate goals and measure to show
    progress or contribution to intended results; (c) including explanatory
    information on the goals and measures; (d) developing performance goals
    to address mission-critical management problems; (e) showing baseline
    and trend data for past performance; (f) identifying projected target
    levels of performance for multiyear goals; and (g) linking the goals of
    component organizations to departmental strategic goals; (2) to
    coordinate crosscutting programs, agencies should: (a) identify programs
    that contribute to the same or similar results; (b) set complementary
    performance goals to show how differing program strategies are mutually
    reinforcing and establishing common or complementary performance
    measures, as appropriate; and (c) describe--briefly or refer to a
    separate document--planned coordination strategies; (3) to show how
    strategies will be used to achieve goals, agencies should: (a) link
    strategies and programs to specific performance goals and describe how
    they will contribute to the achievement of those goals; (b) describe
    strategies to leverage or mitigate the effects of external factors on
    the accomplishment of performance goals; (c) discuss strategies to
    resolve mission-critical management problems; and (d) discuss--briefly
    or refer to a separate document--plans to ensure that mission-critical
    processes and information systems function properly and are secures; (4)
    to show performance consequences of budget and other resources
    decisions, agencies should: (a) show how budgetary resources relate to
    the achievement of performance goals; (b) discuss--briefly and refer to
    the agency capital plan--how proposed capital assets (specifically
    information technology investments will contribute to achieving
    performance goals; and (c) discuss--briefly or refer to a separate
    plan--how the agency will use its human capital to help achieve
    performance goals; and (5) to build the capacity to gather and use
    performance information, agencies should identify internal and external
    sources for data.

    --------------------------- Indexing Terms -----------------------------

     REPORTNUM:  GGD/AIMD-99-69
         TITLE:  Agency Performance Plans: Examples of Practices That Can
                 Improve Usefulness to Decisionmakers
          DATE:  02/26/99
       SUBJECT:  Reporting requirements
                 Program evaluation
                 Performance measures
                 Strategic planning
                 Private sector practices
                 Interagency relations
                 Data integrity
    IDENTIFIER:  GPRA
                 Government Performance and Results Act
                 Pell Grant
                 Federal Family Education Loan Program
                 William D. Ford Federal Direct Loan Program

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO report.  Delineations within the text indicating chapter **
    ** titles, headings, and bullets are preserved.  Major          **
    ** divisions and subdivisions of the text, such as Chapters,    **
    ** Sections, and Appendixes, are identified by double and       **
    ** single lines.  The numbers on the right end of these lines   **
    ** indicate the position of each of the subsections in the      **
    ** document outline.  These numbers do NOT correspond with the  **
    ** page numbers of the printed product.                         **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ** A printed copy of this report may be obtained from the GAO   **
    ** Document Distribution Center.  For further details, please   **
    ** send an e-mail message to:                                   **
    **                                                              **
    **                                            **
    **                                                              **
    ** with the message 'info' in the body.                         **
    ******************************************************************

    Cover
    ================================================================ COVER

    Report to the Chairman, Committee on Governmental Affairs, U.S.
    Senate

    February 1999

    AGENCY PERFORMANCE PLANS -
    EXAMPLES OF PRACTICES THAT CAN
    IMPROVE USEFULNESS TO
    DECISIONMAKERS

    GAO/GGD/AIMD-99-69

    GAO/GGD-99-69

    Agency Performance Plans

    (410311)

    Abbreviations
    =============================================================== ABBREV

      CFOAct - Chief Financial Officers Act
      GPRA - Government Performance and Results Act
      OMB - Office of Management and Budget
      ONDCP - Office of National Drug Control Policy
      OPM - Office of Personnel Management

    Letter
    =============================================================== LETTER

    B-279902

    February 26, 1999

    The Honorable Fred Thompson
    Chairman, Committee on Governmental Affairs
    United States Senate

    Dear Mr.  Chairman:

    The Government Performance and Results Act of 1993 seeks to improve
    the effectiveness, efficiency, and accountability of federal programs
    by establishing a system for agencies to set goals for program
    performance and to measure results.  Under the Act, agencies are to
    prepare annual performance plans to systematically provide
    decisionmakers with information on the results to be achieved for a
    proposed level of resources.  These plans are to reinforce the
    connection between the long-term strategic goals outlined in
    agencies' strategic plans and the day-to-day activities of their
    program managers and staff.  In so doing, the annual performance
    plans provide a basis for establishing accountability for results.

    Fiscal year 1999 marks the first time agencies across the federal
    government were to prepare annual performance plans covering each
    program activity in their budgets.\1 As requested by several members
    of the congressional leadership, we reviewed the performance plans of
    the 24 agencies covered by the Chief Financial Officers (CFO) Act.
    After briefing congressional staff, we issued separate reports on
    each of the 24 agencies' fiscal year 1999 performance plans.\2 In
    September 1998, we issued a report that summarized our observations
    on the agencies' plans and noted that substantial further development
    is needed for these plans to be useful in a significant way to
    congressional and other decisionmakers.\3 That report identified
    opportunities to improve the usefulness of future performance plans.
    These opportunities include (1) better articulating a results
    orientation, (2) coordinating crosscutting programs, (3) clearly
    showing how strategies will be used to achieve goals, (4) showing
    performance consequences of budget decisions, and (5) building
    capacity within agencies to gather and use performance information.

    As you requested, this report builds on those improvement
    opportunities by (1) identifying and describing practices that were
    applied in some plans that might, if consistently applied, improve
    the usefulness of all agencies' annual performance plans; and (2)
    providing examples from agencies' fiscal year 1999 performance plans
    that illustrate each practice.  A forthcoming report that you also
    requested will provide additional information and examples related to
    one of those practices--showing the relationship of budgetary
    resources to performance goals.

    --------------------
    \1 The term "program activity" refers to the listing of projects and
    activities in the appendix portion of the Budget of the United States
    Government.  Program activity structures are intended to provide a
    meaningful representation of the operations financed by a specific
    budget account.

    \2 See appendix I for the list of reports on fiscal year 1999
    performance plans.

    \3 Managing for Results:  An Agenda to Improve the Usefulness of
    Agencies' Annual Performance Plans (GAO/GGD/AIMD-98-228, Sept.  8,
    1998).

       RESULTS
    ------------------------------------------------------------ Letter :1

    Taken as a whole, some of the fiscal year 1999 annual performance
    plans contain practices that, if consistently applied, should help
    improve the usefulness of subsequent agency performance plans to
    congressional and other decisionmakers.  These practices are shown in
    table 1.

                                         Table 1

                         Summary of Opportunities to Improve the
                          Usefulness of Annual Plans and Applied
                                        Practices

    Opportunities to improve
    the usefulness of annual
    plans                      Applied practices
    -------------------------  -----------------------------------------------------
    Better articulate a        1. Create a set of performance goals and measures
    results orientation.       that addresses important dimensions of program
                               performance and balances competing priorities.

                               2. Use intermediate goals and measures to show
                               progress or contribution to intended results.

                               3. Include explanatory information on the goals and
                               measures.

                               4. Develop performance goals to address mission-
                               critical management problems.

                               5. Show baseline and trend data for past
                               performance.

                               6. Identify projected target levels of performance
                               for multiyear goals.

                               7. Link the goals of component organizations to
                               departmental strategic goals.

    Coordinate crosscutting    8. Identify programs that contribute to the same or
    programs.                  similar results.

                               9. Set complementary performance goals to show how
                               differing program strategies are mutually reinforcing
                               and establish common or complementary performance
                               measures, as appropriate.

                               10. Describe--briefly or refer to a separate
                               document--planned coordination strategies.

    Show how strategies will   11. Link strategies and programs to specific
    be used to achieve goals.  performance goals and describe how they will
                               contribute to the achievement of those goals.

                               12. Describe strategies to leverage or mitigate the
                               effects of external factors on the accomplishment of
                               performance goals.

                               13. Discuss strategies to resolve mission-critical
                               management problems.

                               14. Discuss--briefly or refer to a separate document-
                               -plans to ensure that mission-critical processes and
                               information systems function properly and are secure.

    Show performance           15. Show how budgetary resources relate to the
    consequences of budget     achievement of performance goals.
    and other resource
    decisions.                 16. Discuss--briefly and refer to the agency capital
                               plan--how proposed capital assets (specifically
                               information technology investments) will contribute
                               to achieving performance goals.

                               17. Discuss--briefly or refer to a separate plan--
                               how the agency will use its human capital to help
                               achieve performance goals.

    Build the capacity to      18. Identify internal and external sources for data.
    gather and use
    performance information.   19. Describe efforts to verify and validate
                               performance data.

                               20. Identify actions to compensate for unavailable or
                               low-quality data.

                               21. Discuss implications of data limitations for
                               assessing performance.
    --------------------------------------------------------------------------------
    Source:  GAO analysis of agency performance plans.

    The applied practices and examples illustrating how these practices
    were incorporated into certain agencies' fiscal year 1999 performance
    plans are discussed in the individual sections accompanying this
    report.

       OBJECTIVES, SCOPE, AND
       METHODOLOGY
    ------------------------------------------------------------ Letter :2

    At your request, the objectives of this review were to (1) identify
    and describe practices to improve the usefulness of agencies' annual
    performance plans and (2) provide examples from agencies' fiscal year
    1999 performance plans that illustrate each practice.

    To meet both of our objectives, we contacted selected House and
    Senate congressional staff--representing appropriation,
    authorization, budget, and oversight committees that reviewed the 24
    CFO Act agencies' annual plans--to solicit their views on the
    features they found to be particularly useful in the plans.  We also
    reviewed our individual and summary reports on agencies' fiscal year
    1999 plans; the Results Act requirements for agency annual
    performance plans; guidelines contained in the Office of Management
    and Budget (OMB) Circular No.  A-11, Part 2; fiscal year 1999
    performance plans of the 24 CFO Act agencies; our guides to assist
    agencies and Congress to effectively implement the Results Act;\4 and
    our work over the last several years examining agencies' efforts to
    implement the Results Act.  Appendix II provides references to the
    Results Act; OMB Circular No.  A-11, Part 2; and our recently issued
    related products and is arranged by the sections discussed in this
    report.

    We did our work from March 1998 to January 1999 in Washington, D.C.,
    in accordance with generally accepted government auditing standards.
    We did not seek agency comments on this report because much of the
    analysis was drawn from our review of the fiscal year 1999
    performance plans on which the agencies had already commented.
    However, we asked officials in each of the agencies profiled to
    verify the accuracy of the information presented on their respective
    agencies' fiscal year 1999 performance plans, and we incorporated
    their comments where applicable.  We did not independently verify the
    accuracy of the information contained in the agencies' fiscal year
    1999 performance plans.

    --------------------
    \4 The Results Act:  An Evaluator's Guide to Assessing Agency Annual
    Performance Plans, Version 1 (GAO/GGD-10.1.20, Apr.  1998); Agencies'
    Annual Performance Plans Under the Results Act:  An Assessment Guide
    to Facilitate Congressional Decisionmaking, Version 1
    (GAO/GGD/AIMD-10.1.18, Feb.  1998); Agencies' Strategic Plans Under
    GPRA:  Key Questions to Facilitate Congressional Review, Version 1
    (GAO/GGD-10.1.16, May 1997); and Executive Guide:  Effectively
    Implementing the Government Performance and Results Act
    (GAO/GGD-96-118, June 1996).  See also GPRA Performance Reports
    (GAO/GGD-96-66R, Feb.  14, 1996).

    ---------------------------------------------------------- Letter :2.1

    We are sending copies of this report to the Ranking Minority Member
    of the Senate Committee on Governmental Affairs; the House Majority
    Leader; the Chair and Ranking Minority Member of the House Committee
    on Government Reform; other interested congressional committees; and
    to the Director, Office of Management and Budget.  We will also make
    copies available to others on request.

    The major contributors to this report are listed in appendix III.  If
    you have any questions concerning this report, please contact me on
    (202) 512-8676.

    Sincerely yours,

    J.  Christopher Mihm
    Associate Director, Federal Management and Workforce Issues

    BETTER ARTICULATE A RESULTS
    ORIENTATION
    ============================================================ Chapter 1

    Annual performance plans could be more useful documents in helping to
    guide decisions and, subsequently, assess actual performance if the
    plans consistently showed how the agency was adopting a results
    orientation.  Table 1.1 identifies practices that can help agencies
    articulate a results orientation.

                                   Table 1.1

                     Applied Practices to Address a Results
                                  Orientation

    ----------------------------------  ----------------------------------
    A performance plan that better      1. Create a set of performance
    articulates a results orientation   goals and measures that addresses
    includes the following practices:   important dimensions of program
                                        performance and balances competing
                                        priorities.

                                        2. Use intermediate goals and
                                        measures to show progress or
                                        contribution to intended results.

                                        3. Include explanatory information
                                        on the goals and measures.

                                        4. Develop performance goals to
                                        address mission-critical
                                        management problems.

                                        5. Show baseline and trend data
                                        for past performance.

                                        6. Identify projected target
                                        levels of performance for
                                        multiyear goals.

                                        7. Link the goals of component
                                        organizations to departmental
                                        strategic goals.
    ----------------------------------------------------------------------
    1.  Create a set of performance goals and measures that addresses
    important dimensions of program performance and balances competing
    priorities.

    Performance goals and measures that successfully address important
    and varied aspects of program performance are key aspects of a
    results orientation.  To illustrate, the Department of
    Transportation's general goal to advance highway safety was included
    in its performance plan and was accompanied by various goals and
    measures to depict the complex performance they are intended to
    assess.  For example, although the department's performance goals and
    measures on the highway-related fatality rate and highway-related
    injury rate are the primary gauges of safety, they are complemented
    by other goals and measures that contribute to those results.
    Specifically, the plan includes a goal to increase the percentage
    rate of seat belt usage by front seat occupants, a key element in
    reducing overall highway-related fatalities and injuries.

    Federal programs are designed and implemented in dynamic environments
    where competing program priorities and stakeholders' needs must be
    balanced continuously and new needs must be addressed.\1 As a result,
    programs are often forced to strike difficult balances among
    priorities that reflect competing demands, such as timeliness,
    service quality, customer satisfaction, program cost, and other
    stakeholder concerns.  Sets of performance goals and measures could
    provide a balanced perspective of the intended performance of a
    program's multiple priorities.  In its performance plan, for example,
    the Department of Commerce's National Oceanic and Atmospheric
    Administration defined its expected performance to improve the
    timeliness and accuracy of short-range environmental predictions that
    have immediate impact on individuals and many sectors of the economy.
    The relationship between "timeliness" and "accuracy" can represent
    competing demands because the accuracy of predictions could be
    improved as the weather event comes closer to a vicinity.  However,
    the closer the weather event to a vicinity, the shorter and less
    timely the warning or lead time to the public in that vicinity.  For
    fiscal year 1999, the agency set a lead time of 11 minutes for
    tornado warnings with 70-percent accuracy.  The agency also reported
    that it set a lead time of 10 minutes with 59-percent accuracy in
    fiscal year 1997 and 10 minutes with 65-percent accuracy in fiscal
    year 1998.  (See fig.  1.1.)

       Figure 1.1:  Excerpt From the
       Department of Commerce's Fiscal
       Year 1999 Annual Performance
       Plan Featuring the National
       Oceanic and Atmospheric
       Administration

       (See figure in printed
       edition.)

    Program cost is another important factor to be considered.  In
    crafting the Results Act, Congress expected that agencies, whenever
    possible, would develop performance measures that correlate the level
    of program activity with program costs, such as costs per unit of
    result, costs per unit of service, or costs per unit of output.
    Agencies were expected to assign a high priority to developing these
    types of unit cost measures.  The successful implementation of the
    managerial cost accounting standards recommended by the Federal
    Accounting Standards Advisory Board and issued by OMB and GAO are
    vital to providing agencies the program cost information needed to
    develop such performance measures.  Effective for fiscal year 1998,
    these standards are to provide decisionmakers with information on the
    costs of all resources used and the costs of services provided by
    others to support activities or programs, thus allowing
    decisionmakers to compare the costs of various programs and
    activities with their performance outputs and results.

    2.  Use intermediate goals and measures to show progress or
    contribution to intended results.

    Intermediate goals and measures, such as outputs or intermediate
    outcomes, can be used to show progress or contribution to intended
    results.  For instance, when it may take years before an agency sees
    the results of its programs, intermediate goals and measures can
    provide information on interim results.  Also, when program results
    could be influenced by external factors, agencies can use
    intermediate goals and measures to identify the programs' discrete
    contribution to a specific result.

    As an example, the Department of Health and Human Services' Centers
    for Disease Control and Prevention reported in its performance plan
    that heart disease is the nation's number one killer among men and
    women of all racial and ethnic groups.  However, because of the long
    latency period of chronic diseases, such as heart disease, health
    outcome measures do not provide a complete picture of the agency's
    annual performance.  To address the burden of heart disease, the
    Centers for Disease Control and Prevention intends, among other
    things, to focus on the prevention of risk factors, such as tobacco
    use.  The plan explained that comprehensive state programs have been
    shown to be effective in reducing the prevalence of tobacco use.  To
    show a nationwide reduction in morbidity and mortality attributable
    to behavioral risk factors in tobacco use, the plan presented an
    output measure to increase the number of state programs with tobacco
    prevention capacities from 17 in 1996 to 30 in 1999.  The Centers for
    Disease Control and Prevention intends to fund programs in all 50
    states.

    3.  Include explanatory information on the goals and measures.

    Explanatory information in a performance plan can help to show the
    relationship among results-oriented goals, measures, and program
    outputs and services.  Such information is particularly important
    when an agency uses intermediate goals or has several goals that need
    to be balanced.  In its performance plan, the Centers for Disease
    Control and Prevention linked its program objectives to its
    performance goals and measures and to the Department of Health and
    Human Services' strategic goals.  Also included was a rationale for
    the use of process and output measures to inform the user of the plan
    about limitations of the data and associated program constraints.  As
    previously illustrated, the section in its performance plan on
    chronic disease prevention noted that health outcome measures for
    these programs have been difficult to define for a number of reasons,
    including the long latency period of chronic diseases like heart
    disease.

    4.  Develop performance goals to address mission-critical management
    problems.

    Weaknesses in internal management processes and systems undermine the
    achievement of program results.  Therefore, the value of performance
    plans could be increased if they more fully included performance
    goals to address mission-critical management problems that may exist.
    Discussing the most critical management problems--such as those
    identified by the administration as a priority management objective
    in the governmentwide performance plan, the agency's financial audit,
    and other agency assessments made by external organizations,
    including our recent performance and accountability and high-risk
    series\2 --ensures that those management problems that would have the
    greatest impact on results receive the most attention.  A specific
    reference to those priority management objectives identified in the
    governmentwide performance plan can help show an integrated approach
    to their resolution.

    The Department of Education's annual performance plan referred to the
    priority management objectives identified in the fiscal year 1999
    governmentwide performance plan.  Specifically, the plan acknowledged
    that Education's strategic objective on postsecondary student aid
    delivery and program management relates to the governmentwide
    performance plan's priority management objective on student aid
    programs.  Education is responsible for the collection of more than
    $150 billion in outstanding loans, and its data systems track 93
    million student loans and 15 million Pell Grants.  One of Education's
    measures for the Federal Family Education Loan and Direct Loan
    programs was a decline in the percentage of borrowers leaving school
    who default on their student loans within 2 years.  For the Federal
    Family Education Loan program, the annual plan provided baseline and
    trend data for the default rate and indicated that the rate declined
    from 22.4 to 10.4 percent from fiscal years 1990 to 1995.  For fiscal
    year 1999, Education's goal for this program is to reduce the default
    rate to 10.1 percent of borrowers.  According to the plan, future
    declines are likely to be steady but smaller because of the large
    number of high-default schools that have already been eliminated from
    the program.

    5.  Show baseline and trend data for past performance.

    With baseline and trend data, the more useful performance plans
    provided a context for drawing conclusions about whether performance
    goals are reasonable and appropriate.  Decisionmakers can use such
    information to gauge how a program's anticipated performance level
    compares with improvements or declines in past performance.

    In addition to baseline and trend data on past performance, the
    Department of the Treasury's U.S.  Customs Service performance plan
    showed both actual and planned levels of performance for fiscal year
    1997.  For example, in presenting its goal to maximize trade
    compliance, Customs reported the extent of actual and proposed
    compliance with trade laws in key industries.  Specifically, in the
    bearings industry, Customs reported that in fiscal year 1997, it
    achieved an 85.6 percent level of compliance compared to a planned
    level of 83 percent; in the automobile and truck parts industry, it
    achieved an 82.5 percent compliance level compared to a planned level
    of 86 percent.  The proposed performance goals for fiscal year 1999
    in both industries are 89 percent.  (See fig.  1.2.)

       Figure 1.2:  Excerpt From the
       Department of the Treasury's
       Fiscal Year 1999 Annual
       Performance Plan Featuring the
       U.S.  Customs Service

       (See figure in printed
       edition.)

    6.  Identify projected target levels of performance for multiyear
    goals.

    Where appropriate, an agency can convey what it expects to achieve in
    the long term by including multiyear performance goals in its
    performance plan.  Such information can provide congressional and
    other decisionmakers with an indication of the incremental progress
    the agency expects to make in achieving results.  As a start, for
    example, the Department of Veterans Affairs' performance plan has an
    objective for fiscal year 2002 to increase by 20 percent the number
    of patients (as opposed to visits) of its healthcare system over its
    fiscal year 1997 baseline.  The plan shows that the department
    projects achieving a 4.4 percent increase in patients treated in
    fiscal year 1998 and an 8.6 percent increase in fiscal year 1999.
    (See fig.  1.3.)

       Figure 1.3:  Excerpt From the
       Department of Veterans Affairs'
       Fiscal Year 1999 Annual
       Performance Plan

       (See figure in printed
       edition.)

    7.  Link the goals of component organizations to departmental
    strategic goals.

    Linking component performance to departmental goals can provide a
    clear, direct understanding of how the achievement of the components'
    annual goals will lead to the achievement of the agency's strategic
    goals.  The Centers for Disease Control and Prevention, a component
    of the Department of Health and Human Services, linked its annual
    performance goals to the departmentwide strategic goals.  For
    example, the plan set an objective to reduce the case rate of
    tuberculosis--an infectious disease that was nearly eliminated in the
    United States in the mid-1980s, only to reemerge strongly in the
    1990s with drug-resistant strains.  The plan linked that objective to
    Health and Human Services' departmentwide goals to reduce major
    threats to the health and productivity of all Americans.  Specific
    performance goals and measures were included in the plan to reduce
    the tuberculosis case rate, such as increase to 85 percent the
    patients who will complete a course of curative treatment within 12
    months of initiation of treatment.

    --------------------
    \1 Performance and Accountability Series:  Major Management
    Challenges and Program Risks:  A Governmentwide Perspective
    (GAO/OCG-99-1, Jan.  1999).

    \2 Performance and Accountability Series:  Major Management
    Challenges and Program Risks (GAO/OCG-99-22SET, Jan.  1999) and
    High-Risk Series:  An Update (GAO/HR-99-1, Jan.  1999).

    COORDINATE CROSSCUTTING PROGRAMS
    ============================================================ Chapter 2

    A focus on results implies that agencies will coordinate efforts in
    significant program areas where responsibility for achieving results
    is shared among agencies.  Agencies' annual performance plans can be
    useful tools for identifying such crosscutting programs, setting
    complementary performance goals, and describing planned coordination
    strategies.  Table 2.1 identifies applied practices that can help
    agencies address coordination of crosscutting programs.

                                   Table 2.1

                          Applied Practices to Address
                             Crosscutting Programs

    ----------------------------------  ----------------------------------
    A performance plan that addresses   8. Identify programs that
    coordination of crosscutting        contribute to the same or similar
    programs includes the following     results.
    practices:
                                        9. Set complementary performance
                                        goals to show how differing
                                        program strategies are mutually
                                        reinforcing and establish common
                                        or complementary performance
                                        measures, as appropriate.

                                        10. Describe--briefly or refer to
                                        a separate document--planned
                                        coordination strategies.
    ----------------------------------------------------------------------
    8.  Identify programs that contribute to the same or similar results.

    Annual performance plans that include listings of programs and
    agencies that share common results are useful in that they begin to
    document the range and degree to which those federal programs have
    shared responsibilities for achieving results.  These listings can
    serve as the prelude to the substantive work and fuller discussion of
    how agencies' contributions to the same or similar results are being
    coordinated.  As a first step, the Department of Commerce included in
    its performance plan a table identifying interagency activities
    within which the department participates, such as formulating policy
    proposals.  The three themes drawn from Commerce's mission statement
    provide the organizational framework for the listing that includes
    crosscutting activities by bureaus between Commerce and other federal
    agencies.  (See fig.  2.1.)

       Figure 2.1:  Excerpt From the
       Department of Commerce's Fiscal
       Year 1999 Annual Performance
       Plan

       (See figure in printed
       edition.)

    9.  Set complementary performance goals to show how differing program
    strategies are mutually reinforcing and establish common or
    complementary performance measures, as appropriate.

    After agencies have identified other agencies with which they have a
    shared responsibility, the next step is to include in their
    performance plans complementary performance goals to show how
    differing program strategies are mutually reinforcing and establish
    common or complementary performance measures, as appropriate.  Taking
    this further, the usefulness of a performance plan to congressional
    and other decisionmakers could be enhanced if an agency identifies
    the results-oriented performance goals that involve other agencies
    and sets intermediate goals that clarify the specific contribution
    the agency makes to the common result.

    The Department of Health and Human Services' Indian Health Service
    associated its annual performance goals to the governmentwide goals
    of Healthy People.  Started in 1979, Healthy People is a series of
    outcome-based public health objectives and measures developed and
    updated each decade by the U.S.  Public Health Service in
    consultation with other federal agencies, state governments, and
    national organizations.  The objectives for 2000 succeeded the 1990
    health objectives that were set in 1980.  The Public Health Service
    has started planning for the 2010 health objectives, which are
    scheduled to be released in January 2000.

    In developing its performance plan, the Indian Health Service
    reviewed the appropriateness of the Healthy People objectives that
    focused on the most significant health problems affecting American
    Indians and Alaska Natives and the essential services that address
    them.  The plan reported that virtually all of the Service's
    performance goals support the Healthy People objectives for 2000.
    For example, the Indian Health Service's fiscal year 1999 goal to
    ensure that the proportion of the American Indian and Alaska Native
    female population over 40 years of age who have had screening
    mammography is no lower than the fiscal year 1996 level relates to a
    Healthy People objective on reducing the number of women who die from
    breast cancer.  The plan also described the approach for meeting this
    goal, as well as the data sources and baseline data.

    The performance plans for both the Department of Transportation and
    the Department of Justice referenced the Office of National Drug
    Control Policy (ONDCP) goal framework.  Established in 1988, the
    principal purpose of ONDCP is to establish policies, priorities, and
    objectives for the nation's drug control program.  In conjunction
    with stakeholders from both the public and private sectors, ONDCP has
    proposed a system to assess the effectiveness of the national drug
    control strategy to reduce the use of drugs, their availability, and
    the damaging consequences associated with drug use and trafficking.
    The strategic goals and performance measures form a framework that
    shows how the efforts of more than 50 federal drug control agencies'
    programs--along with state, local, and foreign governments and the
    private sector--can contribute to the drug control strategy.

    Transportation's performance plan stated that the Coast Guard is
    working with ONDCP to develop a performance goal and indicator
    measuring the rate of illegal drug smuggling in noncommercial
    maritime routes.  Justice's performance plan stated it will continue
    to work with ONDCP and other agencies in developing and implementing
    an interagency performance measurement system to help gauge the
    effectiveness of the national drug control strategy.

    10.  Describe--briefly or refer to a separate document--planned
    coordination strategies.

    The more useful performance plans not only identified crosscutting
    efforts, they also described how agencies expected to coordinate
    efforts with other agencies that have similar responsibilities.  When
    interagency coordination is difficult because of disparate or even
    conflicting missions across agencies, performance plans offer
    agencies an opportunity to identify statutory or regulatory options
    for better aligning disparate missions and for identifying
    initiatives to avoid or mitigate conflicts over strategic and
    performance goals.

    Because progress in coordinating crosscutting programs is still in
    its infancy, discussions in performance plans of what is being done
    can contribute to the development of a common base for understanding
    the strengths and weaknesses of various approaches to coordination.
    The discussion in the annual performance plans can describe
    opportunities for interagency coordination that are identified in
    other agencies' strategic and performance plans, describe the status
    of coordination efforts, and develop interagency strategies to
    achieve shared outcome goals in the most efficient and effective
    manner.  As a next step, annual performance plans could discuss how
    agencies can cooperate in data collection and program evaluation.

    For example, the Department of Education's performance plan discussed
    how it plans to coordinate its programs that share common results and
    its efforts to cooperate with others to collect data and conduct
    program evaluations.  For its strategic objective that schools are
    strong, safe, disciplined, and drug-free, the performance plan
    indicated that the agency will continue to pursue joint prevention
    activities, such as truancy and hate crimes initiatives, with the
    Department of Justice.  Education intends to develop a database for
    the President's Report Card on School Violence with the Department of
    Justice and make maximum use of data from other federal agencies.
    Education also plans to continue to cooperate on evaluation projects
    with Justice and Health and Human Services.  (See fig.  2.2.)

       Figure 2.2:  Excerpt From the
       Department of Education's
       Fiscal Year 1999 Annual
       Performance Plan

       (See figure in printed
       edition.)

    SHOW HOW STRATEGIES WILL BE USED
    TO ACHIEVE GOALS
    ============================================================ Chapter 3

    The more useful annual performance plans discussed how specific
    strategies and programs will contribute to achieving performance
    goals.  Agencies can also use their performance plans to articulate
    their strategies to leverage or mitigate key external factors that
    will affect program performance and strategies to address
    mission-critical management problems that threaten the achievement of
    program results.  Table 3.1 identifies applied practices that can
    help agencies show how strategies will be used to achieve goals.

                                   Table 3.1

                       Applied Practices to Address Goal
                             Achievement Strategies

    ----------------------------------  ----------------------------------
    A performance plan that shows how   11. Link strategies and programs
    strategies will be used to achieve  to specific performance goals and
    goals includes the following        describe how they will contribute
    practices:                          to the achievement of those
                                        goals.

                                        12. Describe strategies to
                                        leverage or mitigate the effects
                                        of external factors on the
                                        accomplishment of performance
                                        goals.

                                        13. Discuss strategies to resolve
                                        mission-critical management
                                        problems.

                                        14. Discuss--briefly or refer to a
                                        separate document--plans to ensure
                                        that mission-critical processes
                                        and information systems function
                                        properly and are secure.
    ----------------------------------------------------------------------
    11.  Link strategies and programs to specific performance goals and
    describe how they will contribute to the achievement of those goals.

    The more useful annual performance plans linked the agencies'
    strategies and programs to specific performance goals and described
    how they will contribute to the achievement of those goals.  The
    listings of current programs and initiatives that often were included
    in agencies' fiscal year 1999 plans were useful for providing an
    understanding of what agencies do.  However, presentations that more
    directly explain how programs and initiatives achieve goals will be
    most helpful to congressional and other decisionmakers in assessing
    the degree to which strategies are appropriate and reasonable.  For
    example, an agency might state in its performance plan that it will
    administer grants to states.  In order to link this strategy to its
    intended results, the agency should explain how administering grants
    to states will help it achieve specific performance goals.  Over time
    and as agencies gain experience in linking strategies to results,
    agencies will be in a better position to understand and discuss
    alternative strategies and program design and the relative
    contributions these alternatives could make to results.  A subsequent
    discussion of how different governing tools (for example,
    intergovernmental partnerships, performance-based contracts,
    financial grants and credits, and direct service provision by the
    federal government) can be used in achieving goals could further
    enhance the plans.

    The Department of Agriculture's Food Safety and Inspection Service
    provided a detailed discussion of its strategy to achieve its
    performance goal of reducing pathogens on raw meat and poultry
    products.  The performance plan proposed to expand in-plant
    inspections and laboratory testing of meat and poultry samples taken
    from slaughtering plants.  Laboratory personnel will conduct expanded
    testing of pathogens of samples taken from plants slaughtering
    cattle, swine, chicken, and turkeys and use upgraded technology to
    conduct inspections including stethoscopes, thermometers, and
    computer software programs that are not currently available.

    The Federal Emergency Management Agency identified in its performance
    plan a number of strategies relating to its Project Impact program,
    which is designed to promote predisaster mitigation.  One performance
    goal for this program is to increase the number of Project Impact
    communities in each state by at least one.  The strategies identified
    for achieving this goal included working with states and federal
    agencies to identify candidate communities, providing grants as seed
    funding, providing technical information, and monitoring progress.

    12.  Describe strategies to leverage or mitigate the effects of
    external factors on the accomplishment of performance goals.

    The extent to which results are achieved is influenced by a number of
    external factors, such as economic, social, and technological
    developments; and the actions of numerous outside entities, including
    state and local governments.  The Results Act requires agencies to
    identify key external factors in their strategic plans that would
    affect the degree to which they achieve their strategic goals.  The
    value of performance plans can be augmented by including discussions
    on the strategies agencies plan to take to leverage or mitigate the
    influence these factors can have on achieving results.  Such
    discussions can help congressional and other decisionmakers determine
    if the agency has the best mix of program strategies in place to
    achieve its goals, or if additional agency or congressional actions
    are needed to better meet those goals.

    The Department of Agriculture's Food and Nutrition Service discussed
    external factors as well as strategies to mitigate conditions.  The
    annual performance plan included a performance goal to reduce food
    stamp trafficking, which is the illegal sale of food stamps for cash.
    The plan cited the agency's dependence on grocery stores to properly
    handle food stamps as an external factor that affects the agency's
    ability to achieve this goal.  To mitigate this factor, the plan
    noted that it has promulgated rules that participating stores must
    follow and instituted a system of sanctions that may be applied to
    stores that violate the rules.

    13.  Discuss strategies to resolve mission-critical management
    problems.

    Performance plans that contain specific strategies to resolve
    mission-critical management problems provide congressional and other
    decisionmakers with an understanding of how the agency plans to
    improve its management.  By discussing such activities, plans also
    provide a basis for accountability.  As mentioned earlier, agencies
    should set goals to address these management problems, such as those
    identified by the administration as a priority management objective
    in the governmentwide performance plan, the agency's financial audit,
    and other agency assessments by external organizations, including our
    performance and accountability and high-risk series.  Annual
    performance plans can lay out concrete actions the agency plans to
    resolve mission-critical management problems that threaten the
    achievement of major program goals.

    As a first step, Commerce's performance plan identified major
    management challenges raised by the Inspector General, such as the
    department's financial management system, which is also included in
    our performance and accountability series.  The plan stated that the
    system, the Commerce Administrative Management System, is
    substantially over its original estimated cost and continues to
    experience performance shortfalls and schedule delays.  The plan
    identified actions the department will take to reduce expenditures
    and impose greater control over the project's direction, such as
    implementing the system as a pilot at the Census Bureau.

    14.  Discuss--briefly or refer to a separate document--plans to
    ensure that mission-critical processes and information systems
    function properly and are secure.

    By including information technology issues in annual performance
    plans, agencies underscore the critical role that technology plays in
    achieving results.  Key information technology issues include the
    ability to ensure that mission-critical information systems will
    function properly after the year 1999 and are secure from intrusion
    and misuse--two of the issues we have identified to be high risk for
    the federal government.\1 Year 2000-induced failures of one or more
    mission-critical systems may have a severe impact on agencies'
    abilities to deliver critical services.  Agencies can use their
    fiscal year 2000 performance plans, which become effective in October
    1999, to discuss their final business continuity and contingency
    plans to reduce the risk of Year 2000 business failures.

    As a start, the Social Security Administration reported in its
    performance plan that it expects that by January 1999 all of its
    mission-critical systems will have the ability to operate properly in
    2000.  It reported that it is also preparing a Year 2000 contingency
    plan that addresses how core business processes will be supported if
    planned conversion activities experience unforeseen disruptions.

    Despite the sensitivity and criticality of federal information
    systems, they are not being adequately protected from unauthorized
    access.\2 Although advances in the use of electronic data promise to
    streamline federal operations and improve the delivery of federal
    services, security weaknesses can create serious pervasive risks for
    the federal government, such as potential disclosure of sensitive
    data, loss of assets worth billions of dollars due to fraud, and
    disruption of critical operations.  The annual performance plan could
    discuss the agency's ability to ensure the integrity and availability
    of information resources supporting mission-critical operations and
    the confidentiality of sensitive information.

    As a first step, the Department of Transportation's fiscal year 1999
    performance plan stated that a key component of continued security of
    its information system is the education of the workforce.  By the end
    of fiscal year 1998, the plan stated that the agency will develop
    information systems security training for various elements of the
    workforce.  A fiscal year 1999 milestone is that 100 percent of
    senior management, 75 percent of system administrators, and 60
    percent of end users will be trained.

    In its performance plan, the Office of Personnel Management discussed
    plans to enhance its information security program by conducting
    internal and external evaluations of its systems, such as engaging
    the assistance of the National Security Agency to review its security
    capabilities and implementing appropriate recommendations to improve
    its security.  In addition to training staff, the agency will have in
    place a tested disaster recovery capability for general support and
    major financial, benefits, and workforce information application
    systems.

    --------------------
    \1 High-Risk Series:  An Update (GAO/HR-99-1, Jan.  1999).

    \2 Information Security:  Serious Weaknesses Place Critical Federal
    Operations and Assets at Risk (GAO/AIMD-98-92, Sept.  23, 1998).

    SHOW PERFORMANCE CONSEQUENCES OF
    BUDGET AND OTHER RESOURCE
    DECISIONS
    ============================================================ Chapter 4

    The more useful annual performance plans related the performance
    consequences of budget and other resource decisions by indicating how
    the funding from agencies' program activities will be allocated to a
    discrete set of performance goals.  Table 4.1 identifies applied
    practices that can help agencies show performance consequences of
    budget and other resource decisions.

                                   Table 4.1

                    Applied Practices to Address Budget and
                               Resource Decisions

    ----------------------------------  ----------------------------------
    A performance plan that shows       15. Show how budgetary resources
    performance consequences of budget  relate to the achievement of
    and other resource decisions        performance goals.
    includes the following practices:
                                        16. Discuss--briefly and refer to
                                        the agency capital plan--how
                                        proposed capital assets
                                        (specifically information
                                        technology investments) will
                                        contribute to achieving
                                        performance goals.

                                        17. Discuss--briefly or refer to a
                                        separate plan--how the agency will
                                        use its human capital to help
                                        achieve performance goals.
    ----------------------------------------------------------------------
    15.  Show how budgetary resources relate to the achievement of
    performance goals.

    A key expectation of the Results Act is that Congress will gain a
    clearer understanding of what is being achieved in relation to what
    is being spent.  Therefore, plans are more useful when they convey
    the requested funding level associated with achieving a discrete set
    of performance goals and identify where that funding was included in
    the structure of agencies' budget requests.  With such information,
    decisionmakers can better compare planned levels of accomplishments
    with the resources requested.  Decisionmakers can then consider how
    significant changes from the requested amounts for these goals could
    affect performance.  Our forthcoming report on this practice provides
    additional information on the variety of approaches agencies used to
    connect budget resources with results.

    The Results Act gives agencies the flexibility to aggregate,
    disaggregate, or consolidate the budget's program activities so that
    they align with the performance goals.  If an agency takes one of
    these approaches, its performance plan will be more useful if it
    includes a crosswalk that shows how the program activities in its
    budget request relate to the program activities in its annual
    performance plan.  In addition to this flexibility, an agency may
    propose a change to its budget account or program activity structures
    to facilitate an understanding of performance.

    The Department of Health and Human Services' Indian Health Service
    has taken the first step in its performance plan to aggregate program
    activities for the purposes of performance planning.  The plan
    includes a crosswalk that aligns its budget activities with its
    performance in four areas identified as Results Act (GPRA) aggregated
    program activities:  Treatment; Prevention; Capital
    Programming/Infrastructure; and Partnerships, Consultation, Core
    Functions, and Advocacy.  The crosswalk indicates that from two
    separate budget accounts--Services and Facilities--the Indian Health
    Service consolidated program activities (6, 7, 8, and 9 from the
    Services budget account and 21 and 22 from the Facilities budget
    account) in the area of Prevention.  The crosswalk also shows that
    budget program activities 21 and 22 support GPRA program activities
    2, 3, and 4 (Prevention; Capital Programming/Infrastructure; and
    Partnerships, Consultation, Core Functions, and Advocacy).  (See fig.
    4.1.) The Indian Health Service's intended performance in the
    Prevention area includes such things as increasing childhood
    immunization rates, reducing deaths by unintentional injuries, and
    reducing childhood obesity.

       Figure 4.1:  Excerpt From the
       Department of Health and Human
       Services' Fiscal Year 1999
       Annual Performance Plan
       Featuring the Indian Health
       Service

       (See figure in printed
       edition.)

    The Environmental Protection Agency proposed changes to its program
    activity structure to better align its budget resources with
    performance goals.  The agency proposed a uniform program activity
    structure across all of its accounts in which each program activity
    represents one of the agency's strategic goals.  Using this realigned
    program activity structure in its annual plan, the agency showed, by
    account, the funding it requested in order to achieve each strategic
    objective and the supporting annual performance goals.  For example,
    it requested almost $22 million in fiscal year 1999 to meet its
    strategic objective related to the reduction of acid rain.  This
    request is associated with performance goals relating to emissions of
    sulfur dioxide (SO2) and nitrogen oxides (NOx) that react in the
    atmosphere and fall to earth as acid rain.  The request for the acid
    rain strategic objective was related to three accounts from the
    "clean air" program activity:  Environmental Program and Management,
    Science and Technology, and State and Tribal Assistance Grants.  (See
    fig.  4.2.)

       Figure 4.2:  Excerpt From the
       Environmental Protection
       Agency's Fiscal Year 1999
       Annual Performance Plan

       (See figure in printed
       edition.)

    In addition to aligning budget resources with performance goals and
    including trend data and target levels of performance, the Department
    of the Treasury's Internal Revenue Service displayed in its
    performance plan the budget amounts that corresponded with the
    performance actually achieved in a given fiscal year.  Specifically,
    the agency showed that for fiscal year 1997, it received budget
    authority of almost $795 million in its Submission Processing
    Activity to achieve a set of performance goals:  Improve Customer
    Service, Increase Compliance, and Increased Productivity.  The
    performance level that it achieved in fiscal year 1997 to Improve
    Customer Service included, among other things, a timeliness refund
    rate of 14.5 days for electronic filings, an improvement over the
    rates in previous years of 15.5 days in fiscal year 1996 and 21 days
    in fiscal year 1995.  (See fig.  4.3.)

       Figure 4.3:  Excerpt From the
       Department of the Treasury's
       Fiscal Year 1999 Annual
       Performance Plan Featuring the
       Internal Revenue Service

       (See figure in printed
       edition.)

    16.  Discuss--briefly and refer to the agency capital plan--how
    proposed capital assets (specifically information technology
    investments) will contribute to achieving performance goals.

    The federal government's spending on major physical capital
    investment is projected to total over $68 billion in fiscal year
    1999.  Capital assets include information technology; equipment;
    land; structures; and intellectual property, such as software.  With
    federal agencies facing increasing demands to improve performance,
    the importance of making the most effective capital acquisition
    choices--choices that are linked to results--will also intensify.  A
    number of laws are beginning to propel agencies toward improving
    their capital decisionmaking practices.  For example, Title V of the
    Federal Acquisition Streamlining Act of 1994 was designed to foster
    the development of measurable cost, schedule, and performance goals
    and incentives for acquisition personnel to reach these goals.  The
    Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996 put
    in place statutory requirements for agencies to better link their
    information technology plans and decisions to their missions and
    programmatic goals.

    OMB has taken an active leadership role in seeking to link capital
    decisionmaking to program results.  OMB developed a Capital
    Programming Guide that provides agencies with the key elements for
    producing effective plans and investments.\1 OMB's Guide drew on our
    work on best practices used by leading private sector and state and
    local governments, which was subsequently published.\2 Consistent
    with these best practices, OMB has required agencies to submit 5-year
    capital spending plans and justifications--thus encouraging a
    longer-term consideration of agency capital needs and alternatives
    for addressing them.  OMB's Guide provides a basic reference on
    principles and techniques, including appropriate strategies for
    analyzing benefits and costs, preparing budget justifications, and
    managing capital assets once they are in place.  In addition, OMB has
    worked closely with the President's Commission to Study Capital
    Budgeting, which is expected to issue its report and recommendations
    soon.

    In this regard, a more useful performance plan describes the
    processes, technologies, and types of resources, including capital,
    that are needed to achieve performance goals.  Further, performance
    plans provide agencies with the opportunity to show how a proposed
    capital asset will, for example, decrease costs, improve service
    quality, or increase productivity.  For example, the Department of
    the Treasury's U.S.  Customs Service performance plan explained that
    continuing the acquisition and installation of the Land Border
    automation equipment is needed to allow inspectors to perform more
    careful screening and questioning of vehicle occupants, which should
    help to achieve Customs' goal of improving its efficiency at
    targeting arriving vehicles for enforcement purposes.

    17.  Discuss--briefly or refer to a separate plan--how the agency
    will use its human capital to help achieve performance goals.

    In an era that demands improved performance at reduced costs, an
    agency's success depends upon its ability to assemble the human
    capital with the right blend of talents and skills.  Thus, human
    capital planning must be an integral part of an organization's
    strategic and program planning; human capital itself should be
    thought of not as a cost to be minimized but as an asset to be
    enhanced.  To show that the vital integration of human capital and
    program planning is in place, the more useful annual performance
    plans could discuss the size and composition--in terms of knowledge,
    skills, and abilities--of the human capital needed to support the
    achievement of performance goals and explain the workforce planning
    methods by which these needs were determined.  Also, the more useful
    plans could describe how strategies--in such areas as recruitment and
    hiring, retention and separation, training and career development,
    and employee incentives and accountability systems--could meet
    workforce needs and support the achievement of annual performance
    goals.

    The U.S.  Agency for International Development stated in its
    performance plan that its direct-hire workforce has been trimmed by
    hiring freezes, early retirements, and a reduction-in-force in 1996.
    In the wake of these events, the plan recognized that the agency must
    identify critical skills needed to achieve its goals.  Its Office of
    Human Resources is examining issues and changes in workforce planning
    and is focusing, among other things, on improving basic personnel
    operations, such as assignments and employee evaluations.

    --------------------
    \1 Capital Programming Guide, Version 1, July 1997 (Executive Office
    of the President, OMB).

    \2 Executive Guide:  Leading Practices in Capital Decision-Making
    (GAO/AIMD-99-32, Dec.  1998).

    BUILD THE CAPACITY TO GATHER AND
    USE PERFORMANCE INFORMATION
    ============================================================ Chapter 5

    In order to successfully measure progress toward intended results,
    agencies need to build the capacity to gather and use performance
    information.  Under the Results Act, agencies are to discuss in their
    annual performance plans how they will verify and validate the
    performance information that they plan to use to show whether goals
    are being met.  The usefulness of agencies' plans could increase if
    they identify data sources, describe efforts to verify and validate
    data, identify actions to compensate for unavailable data or
    low-quality data, and discuss implications of data limitations for
    assessing performance.  Table 5.1 identifies applied practices that
    can help agencies describe their capacity to gather and use
    performance information.

                                   Table 5.1

                          Applied Practices to Address
                        PerformanceInformation Capacity

    ----------------------------------  ----------------------------------
    A performance plan that describes   18. Identify internal and external
    an agency's capacity to gather and  sources for data.
    use performance information
    includes the following practices:   19. Describe efforts to verify and
                                        validate performance data.

                                        20. Identify actions to compensate
                                        for unavailable or low-quality
                                        data.

                                        21. Discuss implications of data
                                        limitations for assessing
                                        performance.
    ----------------------------------------------------------------------
    18.  Identify internal and external sources for data.

    The usefulness of performance data ultimately depends on the degree
    of confidence that users have in that data.  The more informative
    annual performance plans can provide decisionmakers with confidence
    in the agency's ability to report on its performance goals and
    measures by identifying existing internal and external data sources.
    In those instances when performance data do not exist, a description
    in the performance plan of efforts by the agency to identify relevant
    data sources could inform decisionmakers of actions the agency is
    taking in order to measure performance.

    The Department of Justice identified in its performance plan specific
    sources of performance data for many of its performance measures.
    According to its plan, data are already collected and reported
    through existing statistical series and internal Justice data
    systems, such as the department's Bureau of Justice Statistics--the
    primary source of justice statistics on crime, criminal offenders,
    victims of crime, and the operation of the justice system at all
    levels of government.

    As another example, the Small Business Administration recognized in
    its performance plan its need for data from external sources, such as
    the Federal Procurement Data System.  This system provides
    information relating to the agency's goal to help increase the share
    of federal procurement dollars awarded to small firms.  Specifically,
    the Federal Procurement Data System provides the agency with data on
    the percentage of federal procurement dollars awarded to small
    businesses, women-owned businesses, and small disadvantaged
    businesses.

    19.  Describe efforts to verify and validate performance data.

    Performance plans are to include descriptions of the procedures that
    will be used to verify and validate the measured values of actual
    performance.  The procedures should be credible and specific to
    ensure that performance information is sufficiently complete,
    accurate, and consistent to document performance and support
    decisions on how best to manage programs.  Information in the
    performance plan about the procedures to be used to verify and
    validate the data--such as scope, methodology, and timing--can
    reinforce an agency's ability to generate reliable and timely
    performance data.

    To ensure the accuracy and reliability of reported data, the
    Department of Justice stated in its performance plan that it will (1)
    examine and analyze data to identify anomalies or inconsistencies,
    (2) make comparisons to other data series measuring the same or
    similar variables, (3) require the submitting entities to certify the
    accuracy and completeness of data and to identify any data
    limitations, (4) institute appropriate quality control checks, (5)
    conduct periodic data audits, and (6) obtain independent audits of
    its financial statements.  Justice expects that its component
    organizations will independently pursue several of these initiatives,
    consistent with their individual data requirements.

    As part of its efforts to underscore that program managers are to use
    performance data to make decisions, the Department of Education has
    decided to hold program managers accountable for the quality of the
    performance data related to their programs.  In its performance plan,
    the Department of Education established accountability for data
    quality with its program managers as one of its strategies for
    ensuring high-quality information.  Specifically, program managers
    will attest that the data used for their program's performance
    measurement are reliable and valid, or they will submit plans for
    data improvement.  To validate program managers' assertions to the
    quality of their data, Inspector General audits and program
    evaluations will be used.  Education will support this strategy
    through guidance to managers on developing and monitoring quality
    data systems and using data to manage program performance.

    In addition to the procedures used to verify and validate performance
    data, the use of program evaluations is important for assessing how
    well a program is working, identifying the factors affecting
    performance, and pinpointing improvement opportunities.  In its
    performance plan, the Department of Transportation recognized that
    program evaluation will help it set meaningful performance goals by
    quantifying the relationship between program efforts and levels of
    outcomes.  The plan further identified specific program evaluations
    planned for fiscal years 1998 and 1999.  The department also noted
    that it is developing a method to both plan and track program
    evaluations, as well as to develop the skills and resources
    sufficient to conduct evaluations throughout its organization.

    20.  Identify actions to compensate for unavailable or low-quality
    data.

    When data are unavailable or existing data contain significant
    limitations, providing credible data may require new information
    systems or major changes to existing systems.  We recently reported,
    for example, that some agencies developed or adopted standardized
    data collection systems to obtain data not readily available to them
    or to fill information gaps.\1 However, an agency's ability to
    collect and generate data often needs to be weighed against several
    factors, such as cost or the burden on respondents.  The more
    informative annual performance plans provide a vehicle for discussing
    where the agency struck such a balance and the implications for the
    quality of the data--including accuracy, reliability, and
    timeliness--and the cost of collection.

    The Department of Education recognized that to effectively report the
    performance of the department in achieving its strategic plan goals
    and objectives requires developing some new data systems and fixing
    old ones.  For example, Education stated in its performance plan that
    its student aid delivery system has suffered from significant data
    quality problems.  The plan outlined several steps that Education
    plans to take to address these problems, including improving data
    accuracy by establishing industrywide standards for data exchanges,
    receiving individual student loan data directly from lenders,
    expanding efforts to verify data reported to the National Student
    Loan Data System, and preparing a systems architecture for the
    delivery of federal student aid.

    21.  Discuss implications of data limitations for assessing
    performance.

    Performance plans could be more useful if they recognize and identify
    significant data limitations and their implications for assessing
    performance.  Plans can identify known data limitations from internal
    and external sources, such as quality or timeliness of the data;
    indicate what actions will be taken to compensate for the
    limitations; and report when those actions will be taken.  The more
    informative performance plans also identify implications for
    decisionmaking arising from data limitations on reported results.

    The Department of Transportation stated in its performance plan that
    one of the most significant limitations of both internal and external
    data is timeliness.  One way the department will deal with this
    limitation is to compile preliminary estimates from the portion of
    data that is available in time to report on the performance measures.
    According to the plan, fatality data from the first 6 months of the
    year could be compared with data from the first 6 months of the
    previous year for an initial performance measurement.

    --------------------
    \1 Managing for Results:  Measuring Program Results That Are Under
    Limited Federal Control (GAO/GGD-99-16, Dec.  11, 1998).

    GAO PRODUCTS ON FISCAL YEAR 1999
    PERFORMANCE PLANS
    =========================================================== Appendix I

    Managing for Results:  An Agenda to Improve the Usefulness of
    Agencies' Annual Performance Plans (GAO/GGD/AIMD-98-228, Sept.  8,
    1998).

    The Results Act:  Assessment of the Governmentwide Performance Plan
    for Fiscal Year 1999 (GAO/AIMD/GGD-98-159, Sept.  8, 1998).

    Managing for Results:  Measuring Program Results That Are Under
    Limited Federal Control (GAO/GGD-99-16, Dec.  11, 1998).

       DEPARTMENTS
    --------------------------------------------------------- Appendix I:1

    Results Act:  Observations on the U.S.  Department of Agriculture's
    Annual Performance Plan for Fiscal Year 1999 (GAO/RCED-98-212R, June
    11, 1998).

    Results Act:  Observations on the Department of Commerce's Annual
    Performance Plan for Fiscal Year 1999 (GAO/GGD-98-135R, June 24,
    1998).

    Results Act:  DOD's Annual Performance Plan for Fiscal Year 1999
    (GAO/NSIAD-98-188R, June 5, 1998).

    The Results Act:  Observations on the Department of Education's
    Fiscal Year 1999 Annual Performance Plan (GAO/HEHS-98-172R, June 8,
    1998).

    Results Act:  Observations on DOE's Annual Performance Plan for
    Fiscal Year 1999 (GAO/RCED-98-194R, May 28, 1998).

    The Results Act:  Observations on the Department of Health and Human
    Services' Fiscal Year 1999 Annual Performance Plan (GAO/HEHS-98-180R,
    June 17, 1998).

    Results Act:  Observations on the Department of Housing and Urban
    Development's Fiscal Year 1999 Annual Performance Plan
    (GAO/RCED-98-159R, June 5, 1998).

    Results Act:  Department of the Interior's Annual Performance Plan
    for Fiscal Year 1999 (GAO/RCED-98-206R, May 28, 1998).

    Observations on the Department of Justice's Fiscal Year 1999
    Performance Plan (GAO/GGD-98-134R, May 29, 1998).

    Results Act:  Observations on Labor's Fiscal Year 1999 Performance
    Plan (GAO/HEHS-98-175R, June 4, 1998).

    The Results Act:  Observations on the Department of State's Fiscal
    Year 1999 Annual Performance Plan (GAO/NSIAD-98-210R, June 17, 1998).

    Results Act:  Observations on the Department of Transportation's
    Annual Performance Plan for Fiscal Year 1999 (GAO/RCED-98-180R, May
    12, 1998).

    Results Act:  Observations on Treasury's Fiscal Year 1999 Annual
    Performance Plan (GAO/GGD-98-149, June 30, 1998).

    Results Act:  Observations on VA's Fiscal Year 1999 Performance Plan
    (GAO/HEHS-98-181R, June 10, 1998).

       INDEPENDENT AGENCIES
    --------------------------------------------------------- Appendix I:2

    Results Act:  EPA's Annual Performance Plan for Fiscal Year 1999
    (GAO/RCED-98-166R, Apr.  28, 1998).

    Results Act:  Observations on the Federal Emergency Management
    Agency's Fiscal Year 1999 Annual Performance Plan (GAO/RCED-98-207R,
    June 1, 1998).

    Results Act:  Observations on the General Services Administration's
    Annual Performance Plan (GAO/GGD-98-110, May 11, 1998).

    Managing for Results:  Observations on NASA's Fiscal Year 1999
    Performance Plan (GAO/NSIAD-98-181, June 5, 1998).

    Results Act:  NSF's Annual Performance Plan for Fiscal Year 1999
    (GAO/RCED-98-192R, May 19, 1998).

    Results Act:  NRC's Annual Performance Plan for Fiscal Year 1999
    (GAO/RCED-98-195R, May 27, 1998).

    Results Act:  Observations on the Office of Personnel Management's
    Annual Performance Plan (GAO/GGD-98-130, July 28, 1998).

    Results Act:  Observations on the Small Business Administration's
    Fiscal Year 1999 Annual Performance Plan (GAO/RCED-98-200R, May 28,
    1998).

    The Results Act:  Observations on the Social Security
    Administration's Fiscal Year 1999 Performance Plan (GAO/HEHS-98-178R,
    June 9, 1998).

    The Results Act:  Observations on USAID's Fiscal Year 1999 Annual
    Performance Plan (GAO/NSIAD-98-194R, June 25, 1998).

    RELATED GUIDANCE ON ANNUAL
    PERFORMANCE PLANS
    ========================================================== Appendix II

    This is a compilation of related guidance, including the Results Act,
    OMB Circular A-11, and GAO reports and is arranged by the sections
    discussed in this report.

       SECTION 1:  BETTER ARTICULATE A
       RESULTS ORIENTATION
    -------------------------------------------------------- Appendix II:1

    Government Performance and Results Act (Results Act), 5 U.S.C.
    306(c), 31 U.S.C.  1115(a)(1), 1115(a)(2), 1115(a)(4), and
    1115(a)(5).

    Senate Committee on Governmental Affairs Report accompanying the
    Results Act (Senate Report 103-58, June 16, 1993), pp.  15-16,
    "Performance Plans"; pp.  27-28, "Federal Government Performance
    Plan"; p.  28, "Agency Performance Plans"; p.  29, "Performance
    Goals"; and pp.  29-30, "Performance Indicators."

    OMB Circular A-11(1998), secs.  210.4, 220.1, 220.3, 220.4, 220.5(a),
    220.5(b), 220.5(c), 220.5(f), 220.6(a), 220.6(b), 220.6(d), 220.7(a),
    220.7(b), 220.7(c), 220.7(d), 220.7(f), 220.9(a), 220.9(b), 220.9(c),
    220.9(d), 220.9(e), 220.13, 220.15, 220.16, 221.1(a), 221.1(b),
    221.1(c), 221.2(a), 221.2(b), 221.2(c), 221.2(d), 221.2(e), and
    221.3.

    Managing for Results:  Measuring Program Results That Are Under
    Limited Federal Control (GAO/GGD-99-16, Dec.  11, 1998).

    Managing for Results:  An Agenda to Improve the Usefulness of
    Agencies' Annual Performance Plans (GAO/GGD/AIMD-98-228, Sept.  8,
    1998).

    The Results Act:  Assessment of the Governmentwide Performance Plan
    for Fiscal Year 1999 (GAO/AIMD/GGD-98-159, Sept.  8, 1998).

    The Results Act:  An Evaluator's Guide to Assessing Agency Annual
    Performance Plans, Version 1 (GAO/GGD-10.1.20, Apr.  1998).

    Agencies' Annual Performance Plans Under the Results Act:  An
    Assessment Guide to Facilitate Congressional Decisionmaking, Version
    1 (GAO/GGD/AIMD-10.1.18, Feb.  1998).

    The Government Performance and Results Act:  1997 Governmentwide
    Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).

    Managing for Results:  Analytic Challenges in Measuring Performance
    (GAO/HEHS/GGD-97-138, May 30, 1997).

    Measuring Performance:  Strengths and Limitations of Research
    Indicators (GAO/RCED-97-91, Mar.  21, 1997).

    Executive Guide:  Effectively Implementing the Government Performance
    and Results Act (GAO/GGD-96-118, June 1996).

    GPRA Performance Reports (GAO/GGD-96-66R, Feb.  14, 1996).

    Implementation of the Government Performance and Results Act (GPRA),
    A Report on the Chief Financial Officer's Role and Other Issues
    Critical to the Governmentwide Success of GPRA, Chief Financial
    Officers Council, GPRA Implementation Committee, May 1995.

       SECTION 2:  COORDINATE
       CROSSCUTTING PROGRAMS
    -------------------------------------------------------- Appendix II:2

    OMB Circular A-11(1998), secs.  220.7(e) and 220.9(f).

    Managing for Results:  An Agenda to Improve the Usefulness of
    Agencies' Annual Performance Plans (GAO/GGD/AIMD-98-228, Sept.  8,
    1998).

    The Results Act:  Assessment of the Governmentwide Performance Plan
    for Fiscal Year 1999 (GAO/AIMD/GGD-98-159, Sept.  8, 1998).

    The Results Act:  An Evaluator's Guide to Assessing Agency Annual
    Performance Plans, Version 1 (GAO/GGD-10.1.20, Apr.  1998).

    Agencies' Annual Performance Plans Under the Results Act:  An
    Assessment Guide to Facilitate Congressional Decisionmaking, Version
    1 (GAO/GGD/AIMD-10.1.18, Feb.  1998).

    Managing for Results:  Using the Results Act to Address Mission
    Fragmentation and Program Overlap (GAO/AIMD-97-146, Aug.  29, 1997).

    The Government Performance and Results Act:  1997 Governmentwide
    Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).

       SECTION 3:  SHOW HOW STRATEGIES
       WILL BE USED TO ACHIEVE GOALS
    -------------------------------------------------------- Appendix II:3

    Results Act, 31 U.S.C.  9703.

    Senate Committee on Governmental Affairs Report accompanying Results
    Act (Senate Report 103-58, June 16, 1993), pp.  15-16, "Performance
    Plans"; pp.  17-18, "Managerial Flexibility Waivers"; and pp.  34-36,
    "Section 5.  Managerial Accountability and Flexibility."

    OMB Circular A-11 (1998), secs.  220.5(d), 220.5(e), 220.9(b),
    220.9(d), 220.11(a), 220.11(b), 220.11(c), 220.11(d), and 220.11(e).

    Year 2000 Computing Crisis:  A Testing Guide (GAO/AIMD-10.1.21, Nov.
    1998).

    Managing for Results:  An Agenda to Improve the Usefulness of
    Agencies' Annual Performance Plans (GAO/GGD/AIMD-98-228, Sept.  8,
    1998).

    The Results Act:  Assessment of the Governmentwide Performance Plan
    for Fiscal Year 1999 (GAO/AIMD/GGD-98-159, Sept.  8, 1998).

    Year 2000 Computing Crisis:  Business Continuity and Contingency
    Planning (GAO/AIMD-10.1.19, Aug.  1998).

    Executive Guide:  Information Security Management (GAO/AIMD-98-68,
    May 1998).

    The Results Act:  An Evaluator's Guide to Assessing Agency Annual
    Performance Plans, Version 1 (GAO/GGD-10.1.20, Apr.  1998).

    Executive Guide:  Measuring Performance and Demonstrating Results of
    Information Technology Investments (GAO/AIMD-98-89, Mar.  1998).

    Agencies' Annual Performance Plans Under the Results Act:  An
    Assessment Guide to Facilitate Congressional Decisionmaking, Version
    1 (GAO/GGD/AIMD-10.1.18, Feb.  1998).

    Year 2000 Computing Crisis:  An Assessment Guide (GAO/AIMD-10.1.14,
    Sept.  1997).

    The Government Performance and Results Act:  1997 Governmentwide
    Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).

    Managing for Results:  Analytic Challenges in Measuring Performance
    (GAO/HEHS/GGD-97-138, May 30, 1997).

    Business Process Reengineering Assessment Guide, Version 3
    (GAO/AIMD-10.1.15, Apr.  1997).

    Privatization:  Lessons Learned by State and Local Governments
    (GAO/GGD-97-48, Mar.  14, 1997).

    Executive Guide:  Effectively Implementing the Government Performance
    and Results Act (GAO/GGD-96-118, June 1996).

       SECTION 4:  SHOW PERFORMANCE
       CONSEQUENCES OF BUDGET AND
       OTHER RESOURCE DECISIONS
    -------------------------------------------------------- Appendix II:4

    Results Act, 31 U.S.C.  1115(a)(3) and 1115(c).

    Senate Committee on Governmental Affairs Report accompanying Results
    Act (Senate Report 103-58, June 16, 1993), pp.  15-16, "Performance
    Plans"; pp.  18-19, "Performance Budgeting"; pp.  29-30, "Performance
    Indicators"; and p.  31, "Coverage of Program Activities."

    OMB Circular A-11 (1998), secs.  220.1, 220.8(a), 220.8(b), 220.8(c),
    220.8(d), 220.8(e), 220.9(d), 220.10(a), 220.10(b), 220.10(c),
    220.11(a), 220.11(d), 221.1(a), 221.1(b), 221.1(c), and Part 3.

    OMB Capital Programming Guide, v.  1.0 (July 1997).

    Executive Guide:  Leading Practices in Capital Decision-Making
    (GAO/AIMD-99-32, Dec.  1998).

    Managing for Results:  An Agenda to Improve the Usefulness of
    Agencies' Annual Performance Plans (GAO/GGD/AIMD-98-228, Sept.  8,
    1998).

    Performance Management:  Aligning Employee Performance With Agency
    Goals at Six Results Act Pilots (GAO/GGD-98-162, Sept.  4, 1998).

    The Results Act:  An Evaluator's Guide to Assessing Agency Annual
    Performance Plans, Version 1 (GAO/GGD-10.1.20, Apr.  1998).

    Acquisition Reform:  Implementation of Key Aspects of the Federal
    Acquisition Streamlining Act of 1994 (GAO/NSIAD-98-81, Mar.  9,
    1998).

    Budget Issues:  Budgeting for Capital (GAO/T-AIMD-98-99, Mar.  6,
    1998).

    Executive Guide:  Measuring Performance and Demonstrating Results of
    Information Technology Investments (GAO/AIMD-98-89, Mar.  1998).

    Agencies' Annual Performance Plans Under the Results Act:  An
    Assessment Guide to Facilitate Congressional Decisionmaking, Version
    1 (GAO/GGD/AIMD-10.1.18, Feb.  1998).

    The Government Performance and Results Act:  1997 Governmentwide
    Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).

    Performance Budgeting:  Past Initiatives Offer Insights for GPRA
    Implementation (GAO/AIMD-97-46, Mar.  27, 1997).

    Assessing Risks and Returns:  A Guide for Evaluating Federal
    Agencies' IT Investment Decision-making, Version 1 (GAO/AIMD-10.1.13,
    Feb.  1997).

    Information Technology Investment:  Agencies Can Improve Performance,
    Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, Sept.  30, 1996).

    Executive Guide:  Effectively Implementing the Government Performance
    and Results Act (GAO/GGD-96-118, June 1996).

    GPRA Performance Reports (GAO/GGD-96-66R, Feb.  14, 1996).

    Transforming the Civil Service:  Building the Workforce of the
    Future--Results of a GAO-Sponsored Symposium (GAO/GGD-96-35, Dec.
    26, 1995).

    Integrating Performance Measurement into the Budget Process, Chief
    Financial Officers Council, GPRA Implementation Committee
    Subcommittee Project, September 22, 1997.

    Federal Accounting Standards Advisory Board (FASAB) Volume 1 Original
    Statements:  Statements of Federal Financial Accounting Concepts and
    Standards, Statement of Federal Financial Accounting Standards No.
    1, Objectives of Federal Financial Reporting (GAO/AIMD-21.1.1, Mar.
    1997).

    FASAB Volume 1 Original Statements:  Statements of Federal Financial
    Accounting Concepts and Standards, Statement of Federal Financial
    Accounting Standards No.  4, Managerial Cost Accounting Standards
    (GAO/AIMD-21.1.1, Mar.  1997).

    Budget and Financial Management:  Progress and Agenda for the Future
    (GAO/T-AIMD-96-80, Apr.  23, 1996).

       SECTION 5:  BUILD THE CAPACITY
       TO GATHER AND USE PERFORMANCE
       INFORMATION
    -------------------------------------------------------- Appendix II:5

    Results Act 31 U.S.C.  1115 (a)(6).

    Senate Committee on Governmental Affairs Report accompanying Results
    Act (Senate Report 103-58, June 16, 1993), p.  30, "Verification and
    Validation."

    OMB Circular A-11(1998), secs.  220.6(a), 220.12(a), and 220.12(b).

    Managing for Results:  Measuring Program Results That Are Under
    Limited Federal Control (GAO/GGD-99-16, Dec.  11, 1998).

    Managing for Results:  An Agenda to Improve the Usefulness of
    Agencies' Annual Performance Plans (GAO/GGD/AIMD-98-228, Sept.  8,
    1998).

    The Results Act:  An Evaluator's Guide to Assessing Agency Annual
    Performance Plans, Version 1 (GAO/GGD-10.1.20, Apr.  1998).

    Agencies' Annual Performance Plans Under the Results Act:  An
    Assessment Guide to Facilitate Congressional Decisionmaking, Version
    1 (GAO/GGD/AIMD-10.1.18, Feb.  1998).

    Managing for Results:  Regulatory Agencies Identified Significant
    Barriers to Focusing on Results (GAO/GGD-97-83, June 24, 1997).

    The Government Performance and Results Act:  1997 Governmentwide
    Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).

    Managing for Results:  Analytic Challenges in Measuring Performance
    (GAO/HEHS/GGD-97-138, May 30, 1997).

    Measuring Performance:  Strengths and Limitations of Research
    Indicators (GAO/RCED-97-91, Mar.  21, 1997).

    Executive Guide:  Effectively Implementing the Government Performance
    and Results Act (GAO/GGD-96-118, June 1996).

    GPRA Performance Reports (GAO/GGD-96-66R, Feb.  14, 1996).

    MAJOR CONTRIBUTORS
    ========================================================= Appendix III

    GENERAL GOVERNMENT DIVISION

    Alan M.  Stapleton, Assistant Director
    Lisa R.  Shames, Project Manager
    Dorothy L.  Self, Senior Evaluator

    ACKNOWLEDGMENTS

    The examples used in this report are drawn from the assessments of
    the individual agency annual performance plans that were done by
    staff across GAO.  Thus, in addition to the individuals noted above,
    the staff who worked on the individual agency plan assessments and
    summary assessment also made important contributions to this report.
    These individuals are identified in the separate reports on agency
    fiscal year 1999 plans.

    *** End of document. ***