The Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans (Guidance, 04/01/98, GAO/GGD-10.1.20).

GAO published an evaluator's guide to facilitate assessment of the
strengths and weaknesses of agencies' annual performance plans in
meeting both the requirements of the Government Performance and Results
Act and congressional expectations that the plans inform Congress and
the public about agencies' performance goals, including how they will
accomplish them and how they will measure their results.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  GGD-10.1.20
     TITLE:  The Results Act: An Evaluator's Guide to Assessing Agency 
             Annual Performance Plans
      DATE:  04/01/98
   SUBJECT:  Program evaluation
             Accountability
             Congressional oversight
             Strategic planning
             Cost control
             Agency missions
             Reporting requirements
             Evaluation methods
IDENTIFIER:  Government Performance and Results Act
             GPRA
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


General Government Division

April 1998
Version 1

THE RESULTS ACT - AN EVALUATOR'S
GUIDE TO ASSESSING AGENCY ANNUAL
PERFORMANCE PLANS

GAO/GGD-10.1.20

GAO/GGD-98-001p

Guide to Assessing Agency Annual Performance Plans

(966706)


Abbreviations
=============================================================== ABBREV

  CFO - x
  GAO - x
  OMB - x
  P&F - x

PREFACE
============================================================ Chapter 0

As Congress seeks to reduce the cost and improve the performance of
the federal government, holding agencies accountable for results is
believed to be key to better management of programs.  However,
Congress found that the lack of adequate information on federal
agencies' performance was handicapping congressional policymaking,
spending decisions, and oversight and diminishing federal
accountability for program results.  In response, Congress enacted
the Government Performance and Results Act of 1993 (the Results Act)
to address these concerns and thereby improve federal program
effectiveness and public accountability. 

The Results Act seeks to improve the management of federal programs
by shifting the focus of decisionmaking from staffing and activity
levels to the results of federal programs.  Under the Results Act,
executive agencies are to prepare 5-year strategic plans that set the
general direction for their efforts.  Agencies then are to prepare
annual performance plans that establish the connections between the
long-term strategic goals outlined in the strategic plans and the
day-to-day activities of program managers and staff.  Finally, the
Act requires that each agency report annually on the extent to which
it is meeting its annual performance goals and the actions needed to
achieve or modify those goals that have not been met. 

This guide is intended to facilitate assessment of the strengths and
weaknesses of agencies' annual performance plans in meeting both the
requirements of the Results Act and congressional expectations that
the plans inform Congress and the public about agencies' performance
goals, including how they will accomplish them and how they will
measure their results.  While the Results Act does not require a
specific format for the annual performance plan, it requires the plan
to (1) identify annual goals and measures covering each of its
program activities, (2) discuss the strategies and resources needed
to achieve annual goals, and (3) describe the means the agency will
use to verify and validate its performance data.  This guide is
organized around three core questions that can help in determining
how well the agency's plan meets these general requirements.  For
each core question, issues are identified that need to be addressed,
and criteria and guidance are provided to assist evaluators in
answering the questions raised by these issues.  A companion
assessment guide intended to facilitate congressional use of agency
performance plans contains the same core questions and issues. 

As agencies and Congress gain experience in developing and using
annual performance plans, additional issues and key questions will
likely emerge.  Accordingly, this guide is subject to refinement to
reflect agencies' and Congress' experiences. 

This guide was developed by the Advanced Studies and Evaluation
Methodology group in GAO's General Government Division.  If you have
questions or comments, please contact Susan Westin or Stephanie
Shipman, Advanced Studies and Evaluation Methodology, on (202)
512-7997.  Other contributors to this guide were Joyce D.  Corry,
Allen C.  Lomax, and Thomas M.  Beall, all of the General Government
Division, and Stan Divorski, on special assignment to GAO from the
Office of the Auditor General of Canada.  An electronic version of
this guide is available from GAO's World-Wide Web server at the
following address:  http://www.gao.gov. 

Susan S.  Westin
Associate Director, Advanced Studies and
 Evaluation Methodology

J.  Christopher Mihm
Associate Director, Federal Management
 and Workforce Issues


INTRODUCTION
============================================================ Chapter 1


   RESULTS ACT REQUIREMENTS
---------------------------------------------------------- Chapter 1:1

The Results Act seeks to improve the management of federal programs,
as well as their effectiveness and efficiency, by establishing a
system under which agencies set goals for program performance and
measure their results.  Specifically, the Results Act requires
executive agencies to prepare multiyear strategic plans, annual
performance plans, and annual performance reports. 

Under the Act, strategic plans are the starting point for setting
goals and measuring progress towards them.  The Act requires
virtually every executive agency to develop a strategic plan,
covering a period of at least 5 years forward from the fiscal year in
which the plan is submitted.  These strategic plans are to include an
agency's mission statement, long-term general goals, and the
strategies that the agency will use to achieve these goals.  The
plans should also explain the key external factors that could
significantly affect achievement of these goals, and describe how
long-term goals will be related to annual performance goals.\1

Next, the Results Act requires executive agencies to prepare an
annual performance plan that shall

"(1) establish performance goals to define the level of performance
to be achieved by a program activity;

"(2) express such goals in an objective, quantifiable, and measurable
form .  .  .  ;

"(3) briefly describe the operational processes, skills and
technology, and the human, capital, information, or other resources
required to meet the performance goals;

"(4) establish performance indicators to be used in measuring or
assessing the relevant outputs, service levels, and outcomes of each
program activity;

"(5) provide a basis for comparing actual program results with the
established performance goals; and

"(6) describe the means to be used to verify and validate measured
values."\2

The annual performance plans should systematically provide
congressional decisionmakers with information on the results to be
achieved for a proposed level of resources.  These plans also
reinforce the connections between the long-term strategic goals
outlined in the strategic plans and the day-to-day activities of
program managers and staff.  Moreover, through its requirements for
these annual plans, the Results Act established the first statutory
link between agencies' budget requests and their performance planning
efforts.  The Act requires that performance goals and measures be
linked to the program activities in agencies' budget requests.  The
term "program activity" refers to the listings of projects and
activities in the Appendix to the Budget of the United States
Government.  Program activity structures are intended to provide a
meaningful representation of the operations financed by a specific
budget account. 

In addition, agency performance plans are to be the basis for the
overall federal government performance plan that the Office of
Management and Budget (OMB) must submit to Congress.  OMB is to
submit this governmentwide plan each year to Congress with the
president's budget.  According to the Senate Committee on
Governmental Affairs report on the Act, the overall governmentwide
plan is to present to Congress a single, cohesive picture of the
federal government's annual performance goals for the fiscal year.\3

The agency annual performance plans are to be sent to Congress and
made available to the public soon after transmittal of the
president's budget.\4

Finally, the Results Act requires each agency to prepare annual
reports on program performance for the previous fiscal year.  In each
report, the agency is to compare its performance with the goals
established in its annual performance plan, summarize the findings of
program evaluations completed during the year, and describe the
actions needed to address or revise any unmet goals.  The performance
reports are to be issued by March 31 each year, with the first (for
fiscal year 1999) to be provided to Congress and the president by
March 31, 2000. 

Translating the use of agency resources into concrete, measurable
results will be a continual challenge requiring time and effort. 
Agencies' early efforts to pilot test the Act's requirements faced a
variety of significant challenges, some of which will not be easily
overcome.\5 One set of challenges arises from the complications of
government structure.  Others involve methodological difficulties in
identifying performance measures or a lack of the data needed to
establish goals and assess performance.\6 GAO's review of agencies'
strategic plans produced in September 1997 found that they provided a
workable foundation for Congress to use in pursuing its
responsibilities.\7 However, GAO also found that agencies need to
make continued progress in setting a strategic direction,
coordinating crosscutting programs, and ensuring their capacity to
gather and use reliable performance and cost data.  With sustained
attention from Congress and executive agencies, annual performance
plans can be an invaluable tool for informing policy decisions,
improving program management, enhancing accountability, and helping
to increase American citizens' confidence in their government. 


--------------------
\1 Agencies were to submit their first strategic plans to the Office
of Management and Budget (OMB) and Congress by September 30, 1997. 

\2 P.L.  103-62, sec.  4(b). 

\3 S.  Rep.  No.  58, 103d Cong.  1st Sess.  (1993). 

\4 The first agency performance plans are to cover fiscal year 1999. 

\5 The Government Performance and Results Act:  1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997). 

\6 Managing for Results:  Analytic Challenges in Measuring
Performance (GAO/HEHS/GGD-97-138, May 30, 1997). 

\7 Managing for Results:  Agencies' Annual Performance Plans Can Help
Address Strategic Planning Challenges (GAO/GGD-98-44, Jan.  30,
1998). 


   HOW THIS GUIDE WAS DEVELOPED
---------------------------------------------------------- Chapter 1:2

In May 1997, GAO developed a guide to facilitate congressional review
of agencies' strategic plans at the request of the Chairmen of the
Government Reform and Oversight Committee, Committee on
Appropriations, and Committee on the Budget in the House of
Representatives.\8 The guide GAO developed contained key questions to
help Congress determine how these plans could be improved to better
support congressional and agency decisionmaking.  Subsequently, GAO
employed that guide, the Results Act, and other guidance to assess
whether agencies' draft strategic plans met the requirements of the
law and congressional expectations. 

Building on that guide, those Chairmen, the Speaker of the House, the
Majority Leader of the House, the Chairman of the House Committee on
Science, and the Chairmen of the Committee on the Budget and the
Committee on Governmental Affairs of the Senate subsequently asked
GAO to develop a guide to help congressional decisionmakers both
elicit the information that Congress needs from agencies' annual
performance plans and assess the quality of those plans.\9 That guide
provides questions for assessing the strengths and weaknesses of
agencies' annual performance plans in meeting both the requirements
of the Results Act and congressional expectations that the plans will
inform Congress and the public about agencies' performance goals,
including how they will meet them and how they will measure their
results.  GAO developed this companion evaluator's guide to help GAO
and other evaluators make detailed assessments using the same core
questions and issues as are contained in the congressional guide.  In
particular, this evaluator's guide includes criteria, additional
discussion and guidance, and illustrations of what to look for in
assessing agencies' progress. 

While the issues in this guide make up a complete assessment of an
agency's plan when considered together, and address concerns likely
to be common across a variety of agencies and congressional users,
the significance of some issues may vary for specific agencies and
for appropriations, authorization, oversight, and budget committees. 
This guide, as well as the assessments derived from it, should help
identify additional questions that individual committees may wish to
pursue subsequently in more depth.  GAO hopes that this guide will
also be useful to agencies in preparing future annual performance
plans. 

This guide was developed from the Results Act requirements for agency
performance plans; guidelines contained in OMB Circular No.  A-11,
Part 2; and other relevant documents.  For the purposes of this
guide, the six requirements of the Results Act for the annual
performance plans (stated previously) were collapsed into three core
questions.  GAO also drew on its work over the last several years
examining agencies' pilot efforts to implement the Act, and in
particular our work identifying promising practices for the Act's
effective implementation.\10 The core questions and issues benefited
from the insights of a wide range of congressional staff representing
oversight, authorization, budget, and appropriation committees.  GAO
also received and incorporated valuable comments from senior agency
officials directly involved in the development of annual performance
plans, including officials from OMB and other agencies, as well as
outside experts.  To help identify the information needed to apply
this guide, the key questions, criteria and guidance were critiqued
by selected evaluators across GAO with experience in reviewing the
strategic plans of a variety of different agencies. 


--------------------
\8 Agencies' Strategic Plans Under GPRA:  Key Questions to Facilitate
Congressional Review (GAO/GGD-10.1.16, May 1997). 

\9 Agencies' Annual Performance Plans Under the Results Act:  An
Assessment Guide to Facilitate Congressional Decisionmaking
(GAO/GGD/AIMD-10.1.18, February 1998). 

\10 Executive Guide:  Effectively Implementing the Government
Performance and Results Act (GAO/GGD-96-118, June 1996). 


   GENERAL GUIDANCE
---------------------------------------------------------- Chapter 1:3


      OBJECTIVES AND SCOPE OF THE
      INTENDED REVIEW
-------------------------------------------------------- Chapter 1:3.1

This document outlines the issues that evaluators should address, in
reviewing an agency's annual performance plan prepared under the
Results Act, to assess the extent to which the plan meets the
requirements of the Act and related expectations.  The focus is on
expectations concerning how well the plan communicates what an agency
proposes to accomplish on an annual basis, how it will accomplish
those goals, and how it will assess whether those results were
achieved.  In order to encourage agencies to improve their
performance, as well as to depict what can be expected in the future,
GAO developed criteria geared to agencies with a mature performance
measurement system.  However, in the early stages of implementing the
law, performance plans are not expected to completely meet these
expectations, because overcoming the challenges involved may take
many years.  Agencies' early performance plans should be judged,
instead, on whether the agencies are making reasonable progress
towards meeting the expectations.  Therefore, this document is
intended to guide assessment of agency progress towards meeting those
expectations and identify opportunities for agencies to refine and
clarify their plans. 

While the Results Act does not require a specific format for the
annual performance plan, it does require the plan to (1) identify
annual goals and measures covering each of its program activities,
(2) discuss the strategies and resources needed to achieve annual
goals, and (3) describe the means the agency will use to verify and
validate its performance data.  This guide is organized around three
core questions that can help determine how well the agency's plan
meets these general requirements: 

1.  Annual Performance Goals and Measures:  To what extent does the
agency's performance plan provide a clear picture of intended
performance across the agency? 

2.  Strategies and Resources:  How well does the performance plan
discuss the strategies and resources the agency will use to achieve
its performance goals? 

3.  Validation and Verification:  To what extent does the agency's
performance plan provide confidence that its performance information
will be credible? 

To help evaluators answer each core question, the guide identifies
key issues under each core question that the plan should address. 
For each of these issues, the guide provides criteria--reflecting
expectations derived from the Act and other guidance--for evaluators
to use to assess how well the plan addresses that issue.  The
guidance also identifies what evaluators should look for in making
that assessment, and provides some illustrations of the concept and
how the information might be presented in the agency's plan.  It is
expected that, as the agencies and Congress gain experience with the
Act, additional issues will emerge.  Those issues will be
incorporated into subsequent versions of this document and the
congressional guide, as appropriate. 

This guide was developed to direct a timely, general assessment of an
agency's annual performance plan viewed as an informative document. 
It is not intended to direct a thorough review of an agency's
proposed performance reporting system, although it may identify
issues that deserve additional in-depth attention in a follow-up
review.  This intended review is expected to be based on an agency's
strategic and annual performance plans; budget justification
materials; the capital asset and technology plans; the Appendix to
the Budget of the United States Government for the relevant fiscal
year; previous reviews, if any, of the agency's strategic plan; and
the evaluators' knowledge of, and available materials on, the
agency's programs, operations, and databases.  In addition,
evaluators will want to consult the Results Act; the Senate Report on
S.  20,\11 OMB Circular No.  A-11, part 2; and OMB's checklist for
reviewing agencies' performance plans (dated November 24, 1997). 
Interviews with agency and Inspector General officials may also be
required to clarify issues that are incompletely described in the
performance plan. 


--------------------
\11 S.  Rep.  No.  58, 103d Cong.  1st Sess.  (1993). 


      ASSESSING THE PERFORMANCE
      PLAN AS A WHOLE
-------------------------------------------------------- Chapter 1:3.2

The review guide poses three core questions about how well the
performance plans meet both the requirements of the Act and
congressional expectations for the plans.  Under each of these core
questions are issues and criteria spelling out the key dimensions of
those requirements and the expectations about how they should be
addressed in the performance plan.  Evaluators are expected to
develop answers to each of the three core questions by reviewing the
plan against the individual criteria for each issue, to assess how
well the issues were addressed.  Answers to the three core questions
should then be combined to form an overall judgment about what level
of progress the plan as a whole represents--for example, whether the
plan "generally meets," "partially meets," or "falls well short of
meeting" these expectations.  A plan generally meets expectations if
it addresses all the key issues and contains no significant
deficiencies.  It is quite likely that the quality of an agency's
plan will be uneven in the way it addresses distinct components of
the agency, because an agency plan may represent a compilation of
plans developed by components of the agency.  If the parts of the
plan that address one or two components fall well short of
expectations, but the remainder of the plan meets them, evaluators
should characterize the agency's plan as having generally met
expectations, and then note those exceptions. 

Each of the three core questions should be answered by characterizing
the strengths and weaknesses of the performance plan and identifying
any areas needing improvement, in accordance with how well each of
the issues was addressed.  In addition, evaluators are encouraged to
provide suggestions for how the plan could be improved.  Since a
basis for setting expectations for a specific level of performance is
not yet available, evaluators must assess the plans against the
ideal, while recognizing that few initial plans are likely to reach
that level.  In assessing progress towards the ideal, evaluators
should also recognize that some deficiencies are more important than
others, and that an unsuccessful agency effort to address a
particularly challenging task shows greater progress towards
accomplishing the Act's goals than does not having attempted the task
at all.  Answers to the core questions should use the same three
categories recommended for assessing the plan overall:  "generally
meets," "partially meets," or "falls well short of meeting" these
expectations. 


   ASSESSING THE PLAN AGAINST THE
   QUESTIONS AND CRITERIA
---------------------------------------------------------- Chapter 1:4

"Answers" to each of the three core questions should strive to
capture whether or not the plan generally addressed each issue, while
also recognizing where the plan handled an issue very well.  In some
cases, while the criteria provide standards representing the Act's
requirements, the accompanying textual guidance also describes what a
particularly good plan would contain. 

Scoring the plans by tallying up the instances of compliance--for
example, by counting the number of performance measures that were
clearly linked to the strategic plan--is not recommended.  Rather, in
this example, the concern should be that, if there are no performance
goals specified for a given strategic goal, or if there are a large
number of performance goals with no visible link to the agency's
strategic plan, then there should be an explanation of those
exceptions.  In some cases, evaluators will need to make a judgment
as to whether the exceptions are limited enough, or whether the
presentation appears to be a reasonable and good-faith effort, to
warrant a "generally meets expectations" rating.  The criteria and
accompanying guidance for each core question and issue are provided
to help guide these judgments. 

All of the issues outlined in the guide are not equally important,
and some issues may be more important for some agencies than for
others.  It is not necessary, or even likely, for evaluators to draw
a conclusion against every criterion in the guide.  However, in those
cases where evaluators lack the information to assess the plan on an
issue that seems significant to forming a conclusion on one of the
core questions, they should so indicate in their report. 

Although criteria and guidance are provided to help evaluators judge
whether an issue was adequately addressed, the criteria are not
self-assessing.  The issues involved in setting goals for, and
measuring the accomplishments of, public programs are complex and
subject to interpretation, and can also vary from one agency context
to another.  Thus, evaluators must apply the criteria in
consideration of the specific circumstances of each agency. 


ANNUAL PERFORMANCE GOALS AND
MEASURES
============================================================ Chapter I


   CORE QUESTION 1
---------------------------------------------------------- Chapter I:1

To what extent does the agency's performance plan provide a clear
picture of intended performance across the agency? 

   Figure I.1:  Summary of the
   Issues and Criteria for Core
   Question 1

   (See figure in printed
   edition.)

Guidance.  As the Results Act requires, the annual performance plan
must provide a basis for an agency to compare actual results with
performance goals (Results Act, sec.  4(b)).  To do this, the agency
needs to set goals, develop measures, and show in its performance
plan how it will use these goals and measures to assess
performance.\12 By showing the relationship between the annual
performance goals and the agency's strategic goals and mission, an
agency's performance plan can demonstrate how the agency intends to
make progress toward achievement of its strategic goals. 

In addition, the Results Act requires that the performance plan
contain one or more performance goals covering the program activities
in the agency's budget, which can be done by aggregating,
disaggregating, or consolidating the program activities.  Thus, the
plan should define performance to be achieved throughout the agency. 
Performance plans do not have to include the complete array of goals
and measures used in managing programs, but they should reflect a
picture of intended performance without any significant gaps. 
Agencies with programs that have strategic or performance goals
similar to those of other federal programs should address the need
for coordination among such crosscutting efforts.  Annual performance
plans can be useful tools for identifying the need for coordination
and for discussing ways to address that need.  These plans can help
ensure that the goals are congruent and that crosscutting efforts are
mutually reinforcing. 


--------------------
\12 The Results Act defines a performance goal as a target level of
performance expressed as a tangible, measurable objective against
which actual achievement can be compared, including a goal expressed
as a quantitative standard, value, or rate.  It is likely that such
performance goals are self-measuring, requiring no separate measures. 
However, other performance goals, those that are more abstract and
not tangible and measurable on their own, should be expected to have
one or more separate performance measures or indicators as needed to
help define the goal for measurement.  In this guide, the terms
"measure" and "performance measure" also include the concept of what
are frequently called "performance indicators." In general,
performance measures are a tabulation, calculation, recording of
activity or effort, or assessment of results compared to intended
purpose, that can be expressed quantitatively or in another way that
indicates a level or degree of performance. 


      ISSUE 1:  DEFINING EXPECTED
      PERFORMANCE
-------------------------------------------------------- Chapter I:1.1

To what extent do the annual performance goals and measures provide a
succinct and concrete statement of expected performance for
subsequent comparison with actual performance? 

Guidance.  The Results Act requires an agency's annual performance
plan to contain both a set of annual goals that establishes its
intended performance and measures that can be used to assess progress
towards achieving those goals.  Some goals are self-measuring--that
is, they are expressed objectively and quantifiably--and thus do not
require the use of additional measures.  For example, a performance
goal to staff 300 airport control towers on a 24-hour basis in a
given year would not require additional measures.  Other goals, which
may be more abstract, require the defining of specific performance
measures in order to assess progress towards the goals.  For example,
a goal for tribal reservations to meet national standards for
maternal and child health would require the use of specific measures
reflecting those standards, such as morbidity and mortality rates for
this population, percentage of low birth weight babies, the
percentage of children receiving their full immunization series,
frequency of pediatric checkups, etc.  In either case, the goals and
measures should be objective and precise, and should allow for the
assessment of performance.  The measures should also be clearly
related to the performance they will be used to evaluate.  While the
number of measures for each goal at a given organizational level
should not be excessive, it is critical that they represent the
important dimensions of performance for that goal. 

An agency can use output goals, outcome goals, or some combination of
the two to reflect the agency's intended performance.  However, the
Results Act is clearly outcome-oriented, and thus an agency's
performance plan should include outcome goals whenever possible. 
Outputs can be defined as the direct products and services delivered
by a program.  Outcomes are the results of those products and
services.  For example, for a job training program whose objective is
to provide the target population with the skills needed to find and
keep a job, an output might be the percentage of participants who
have completed training, while an outcome might be the percentage of
participants employed 1 year after completing the training.  In
addition, results can be either intermediate outcomes representing
immediate benefits for program clients, such as obtaining employment
in their area of training, or end outcomes representing benefits to
the public at large, such as increased self-sufficiency. 


         CRITERION 1.1
------------------------------------------------------ Chapter I:1.1.1

The performance measures contained in the annual performance plan
should adequately indicate progress towards the performance goals
(Senate Report 103-58, p.  29). 

Guidance.  When performance goals are not self-measuring, performance
measures should translate those goals into concrete, observable
conditions that determine what data to collect to learn whether
progress was made towards achieving goals.  Such measures are meant
to cover the key aspects of performance that will enable agencies to
assess accomplishments, make decisions, realign processes, and assign
accountability.  To do this, the measures must clearly represent or
be related to the performance they are trying to assess.  For
example, a measure of the average processing time for tax returns
does not represent, nor is it clearly related to, the goal of
increasing the accuracy of tax return processing.  To adequately
indicate progress towards the performance goals, the measures need to
validly (or appropriately) represent the performance goals and
sufficiently cover the key aspects of the agency's performance. 

To be valid representations of the performance goals, the measures
should have a clearly apparent or commonly accepted relationship to
the intended performance or have been shown to be reasonable
predictors of the desired behaviors or events.  For example, because
the effectiveness of immunizations for childhood diseases is well
acknowledged, the percentage of children receiving their full
immunization series is considered a valid and appropriate measure of
their protection from those diseases, and thus a valid indicator (or
less direct measure) of their health status.  Also, the measures
should not represent too constrained a view of performance.  For
example, because only some people choose to appeal the results of an
audit of their tax return, the rate at which those cases are
overturned on appeal would not be a valid measure of the accuracy of
all tax return processing.  In addition, to ensure that they will
permit reliable measurement, the measures should be amenable to
applying standard procedures for collecting data or calculating
results--so that they would be likely to produce the same assessment
of results if applied repeatedly to the same situation. 

Valid measures should also not be unduly influenced by factors
outside the program's control--because that influence makes it
difficult to attribute changes in the measure to the effectiveness of
the program.  However, evaluators should note that certain measures
may be unavoidably influenced by factors outside the program but
still be appropriate to use.  For example, the health of the economy
may influence the rate of defaults on loans to students or
homeowners, but those measures assess an important component of loan
programs' performance and can also reflect a program's success in
managing its loans.  When no potential measure seems to be reasonably
free of such influences, a program may choose to use multiple
measures to avoid drawing mistaken conclusions about the program's
effectiveness. 

The measures sufficiently cover key aspects of performance when
individual measures or sets of measures represent the important
dimensions of their performance goals and when the measures reflect
the core functions of their related programs or activities.  For
example, if a program ranks the crashworthiness of vehicles by
measuring the extent of injury to passengers and damage to vehicles
in crash tests that use only front-end collisions, the measure may
not be valid for ranking the vehicles' overall crashworthiness. 
However, if front-end collisions are most frequently those collisions
producing the most injury and damage, then this measure of
crashworthiness may be the most appropriate one for assessing
programs that aim to reduce injury and damage from vehicle
collisions.  Further, the number of measures for each goal at a given
organizational level should not be so large that some measures are
redundant, or unnecessary for assessing progress toward the goal. 

In addition, because public programs often have multiple objectives
and must balance competing interests, a given program or activity
often should contain multiple goals and measures responding to
multiple priorities, such as quality, timeliness, cost, and outcomes. 
When complex program goals are broken down into a set of component
quantifiable measures, it is important to ensure that the overall
measurement of performance does not become biased by measures that
assess some priorities but neglect others.  Also, in reviewing
measures, evaluators should be sensitive to the possibility that the
use of or reliance on certain measures could produce unintended
consequences.  For example, for a welfare program that aims to
encourage the economic self-sufficiency of mothers yet only proposes
to measure the percentage who found employment within 6 months, an
unintended consequence might be a tendency of the program to provide
temporary rather than permanent positions for its clients. 

The following steps capture the main elements for assessing the
annual performance plan against this criterion of whether the
performance measures contained in the plan adequately indicate
progress towards the performance goals. 

The evaluator should

  -- assess whether the measures appear to be valid representations
     of the performance goals;

  -- assess whether the measures sufficiently cover key aspects of
     performance and adequately capture important distinctions
     between programs; and

  -- assess whether the agency's use of its measures, either
     individually or as a set, could result in promoting any
     unintended consequences. 


         CRITERION 1.2
------------------------------------------------------ Chapter I:1.1.2

The annual performance goals or their measures should be objective,
measurable, and quantifiable (Results Act, sec.  4(b)). 

Guidance.  For goals and measures to be considered objective, they
should be reasonably free of any significant bias or manipulation
that would distort the accurate assessment of performance.  To the
greatest extent possible, the goals and measures should not require
subjective considerations or judgments to dominate the measurement. 
The performance goals and measures usually should include a
quantifiable, numerical target level or other measurable value. 
Examples of quantifiable goals would be goals to have 50 percent of
participants in a job training program obtain a job related to their
training within the next year and hold the job for at least 6 months,
or to increase by 5 percentage points the percentage of children
under age 2 who have been adequately immunized.  However, without
adequate baseline data, goals like the latter one may not permit
subsequent comparison with actual performance.  If a goal is not
quantifiable, it may still be susceptible to measurement if a
determination of its level of achievement can be demonstrated and
independently verified.  For example, a goal that is not quantifiable
but is a statement of a tangible condition or event, such as landing
a person on the moon or discovering the cure for a disease, can be
measurable. 

If a goal cannot be expressed in an objective, specific, and
measurable form, the Results Act allows OMB to authorize agencies to
develop alternative forms of measurement.  An alternative form may be
either (1) separate descriptive statements of a minimally effective
program and a successful program, expressed with sufficient precision
and in such terms that would allow for an accurate, independent
determination of whether the actual performance meets the criteria of
the description; or (2) some other alternative that allows an
accurate, independent determination to be made of how actual
performance compares with the goal as stated. 

If OMB authorizes an agency's request to use an alternative form, OMB
has stated that the plan need not document the authorization.  Thus,
evaluators are on their own to identify such use of alternative
forms.  If the annual performance plan includes performance goals
that do not meet the characteristics of being objective, measurable,
and quantifiable, evaluators should assess those goals against the
characteristics for alternative forms of measurement allowed by the
Results Act.  If the agency determines that it is infeasible or
impractical to express any performance goals in any form for a
program activity, the performance plan should state why. 

Key features to look for include

  -- performance goals and measures that indicate specifically what
     should be observed, in which population or conditions, in what
     time frame (for example, an increase in immunizations of
     children under age 2 by 10 percent over the previous year);

  -- the use of baseline or benchmark data to allow for the
     assessment of progress towards goals;

  -- performance goals and measures that state a particular target
     level of performance, either as an absolute value (for example,
     20,000 served) or as a targeted level of improvement (10 percent
     increase over the previous year's level); and

  -- the use of alternative forms of goals and measures. 

The evaluator should

  -- assess whether the measures avoid limiting observations to
     certain types of cases or situations that may be more or less
     likely to show progress toward the goal, thus avoiding biasing
     the measurement;

  -- assess the extent to which the performance goals and measures
     allow for the comparison of actual performance with intended
     performance;

  -- when needed, determine whether baseline or trend data are
     contained or referred to in the plan so as to allow for the
     measurement of goals defined in terms of improvement;

  -- determine whether any attempts to benchmark against other
     organizations' performance are made and assess whether such
     attempts seem appropriate, based on previous and ongoing GAO
     work or other known sources;

  -- identify alternative forms used, if any; and

  -- if applicable, assess the alternative forms to determine whether
     such forms allow for the comparison of actual performance with
     intended performance (i.e., to assess whether or not the goal
     has been achieved). 


         CRITERION 1.3
------------------------------------------------------ Chapter I:1.1.3

Outcome goals should be included in the annual performance plan
whenever possible (Senate Report 103-58, p.  15). 

Guidance.  An agency's intended performance can be represented in its
performance plan by output goals, outcome goals, or some combination
of the two.  It is anticipated that agencies will include output
goals in their annual plans, and these output goals may predominate,
especially at this early stage.  However, agencies should make every
attempt to identify and use annual outcome goals whenever possible to
reflect the results of their activities. 

Key features to look for include

  -- goals expressed as the results of the services or goods produced
     by the program, such as actions that occur in response to, or
     conditions resulting from, program activities, and

  -- discussions of efforts to identify and use outcome goals, as
     well as explanations of why outcome goals were not used (e.g.: 
     outcomes will not be achieved on an annual basis; outputs are
     being used as intermediate outcomes; the program's primary
     objective is to produce an output; or it may not be possible to
     identify an outcome for the program). 

The evaluator should

  -- assess whether outcome goals are included in the performance
     plan and if so, the extent to which they are used;

  -- if outcome goals are not included, assess whether the plan
     provides a reasonable and credible explanation or rationale for
     why not; and

  -- based on the evaluator's understanding of program purposes,
     evaluate the plan for missed opportunities to identify or use
     outcome goals. 


      ISSUE 2:  CONNECTING
      MISSION, GOALS, AND
      ACTIVITIES
-------------------------------------------------------- Chapter I:1.2

How are the agency's annual performance goals linked to the agency's
mission, strategic goals, and program activities in its budget
request? 

Guidance.  A clear relationship should exist between an agency's
long-term strategic goals and mission and the performance goals in
the annual performance plan.  Successful organizations try to link
performance goals and measures to the organization's strategic goals
and, to the extent possible, have performance goals that will show
annual progress towards achieving their long-term strategic goals. 
For example, an annual performance goal to reduce the rate of
transportation-related fatalities per passenger-mile-traveled by 1
percent would show annual progress toward the strategic goal to
promote the public health and safety by working toward the
elimination of transportation-related deaths, injuries, and property
damage.  An agency's performance plan should show the relationship
between the annual performance goals and the strategic goals and
mission through a crosswalk, other illustration, or description.  If
the relationship is minimal or nonexistent, the evaluator should
point out that there may be a need to correct this situation, perhaps
by revising the strategic plan. 

However, not all annual goals have to link directly to the long-term
strategic goals.  Because agencies are required to have one or more
performance goals that cover each program activity in the agency's
budget, and because program activity structures may not be
mission-based, some performance goals may cover program activities
that have no direct link to the strategic goals or mission.  However,
in general, at least one performance goal should be related to each
strategic goal or, alternatively, some explanation of missing
relationships be given.  An amendment to the strategic plan,
documented in the annual plan, can be an acceptable way to establish
such a relationship.  Minor changes to the strategic plan can be made
directly in the annual performance plan.  In addition, the annual
plan should include performance goals for efforts to address major
management-related problems that are mission-critical or would
prevent achievement of the agency's strategic goals, whether these
problems are identified in the strategic plan or otherwise based on
evaluators' knowledge.  In general, the annual performance plan could
be an appropriate place for an agency to address shortcomings in its
strategic plan that GAO and others identified in reviews of that
plan. 


         CRITERION 2.1
------------------------------------------------------ Chapter I:1.2.1

The annual performance goals should reflect the strategic goals and
mission (OMB Circular A-11, sec.  220.5). 

Key features to look for include

  -- the mission statement and long-term strategic goals from the
     agency's current strategic plan, or a summary of them; any
     indication of whether they have been revised; and, if so, an
     explanation for the revision;

  -- a diagram, table, model, or description that clearly aligns the
     agency's annual performance goals and the current strategic
     goals and mission;

  -- descriptions of how annual performance goals contribute to
     achieving or supporting strategic goals, unless obvious;

  -- explanations for annual goals that do not relate to one or more
     of the strategic goals;

  -- explanations for strategic goals that do not have related annual
     goals; and

  -- amendments to the strategic plan, or explanations that the
     strategic plan has been revised, to correct the lack of these
     relationships. 

The evaluator should

  -- compare the mission statement and strategic goals (or summary of
     them) in the annual plan with those in the strategic plan to
     determine (1) if there are differences and (2) if a reasonable
     explanation is provided for any substantive changes made;

  -- assess the alignment of the annual performance goals in the
     performance plan with the strategic goals and mission and
     determine whether this alignment is clear and reasonable; for
     example, one indicator of a reasonable alignment would be if the
     annual performance goals can show progress towards achieving the
     strategic goals;

  -- determine whether there are any strategic goals that are not
     related to at least one performance goal and assess whether a
     reasonable explanation is provided if such relationships are
     missing;

  -- assess whether the agency has responded in its annual
     performance plan to previous reviews of that strategic plan; and

  -- determine whether performance goals are identified that cover
     mission-critical management problems or issues. 


         CRITERION 2.2
------------------------------------------------------ Chapter I:1.2.2

The annual performance plan should identify annual performance goals
that cover all of the program activities in the agency's budget
(Results Act, sec.  4(b)).\13

Guidance.  The annual performance plan should show how specific
performance goals are related to the specific program activities
contained in the agency's program and financing (P&F) schedules under
"obligations by program activity" in the Appendix to the Budget of
the United States Government (OMB Circular A-11, sec.  220.9).  The
Results Act requires each federal agency to prepare an annual
performance plan that covers these program activities while allowing
the agency to aggregate, disaggregate, or consolidate those program
activities for purposes of the plan.  An agency can have one or more
budget accounts, and each budget account has its own P&F schedule. 
The ï¿½obligations by program activityï¿½ section is the first section in
each P&F schedule.  This section shows obligations for the agencies'
specific program activities.  The number of program activities in
agencies' budget accounts varies from account to account, with many
accounts having no program activity breakdown other than "total
obligations" or another single activity.  (See figures I.2, I.3, and
I.4 for examples of P&F schedules from the budgets for the Department
of Energy, the Environmental Protection Agency, and the General
Accounting Office, respectively.)

   Figure I.2:  Excerpt from the
   Department of Energy's 1999
   Budget

   (See figure in printed
   edition.)

Source:  Budget of the United States Government, Fiscal Year
1999--Appendix (Washington, D.C.:  U.S.  Government Printing Office,
1998). 

   Figure I.3:  Excerpt from the
   Environmental Protection
   Agency's 1999 Budget

   (See figure in printed
   edition.)

Source:  Budget of the United States Government, Fiscal Year
1999--Appendix (Washington, D.C.:  U.S.  Government Printing Office,
1998). 

   Figure I.4:  Excerpt from the
   General Accounting Office's
   1999 Budget

   (See figure in printed
   edition.)

Source:  Budget of the United States Government, Fiscal Year
1999--Appendix (Washington, D.C.:  U.S.  Government Printing Office,
1998). 

Program activity structures can vary from agency to agency and,
within an agency, from budget account to budget account.  Program
activities, like budget accounts, may represent programmatic,
process, organizational, or other orientations.  Therefore, some of
the program activities that are currently used in an agency's budget
request may be either too detailed or too general, or they may lack a
clear relationship to the programs that are useful for measuring the
agency's performance.  The Results Act allows agencies, at their
discretion, to aggregate, disaggregate, or consolidate program
activities so that they align with the performance goals in the
performance plan. 

The Act assumes that agencies will use aggregation, disaggregation,
and consolidation to highlight important activities while avoiding
voluminous presentations that would overwhelm a reader and detract
from the plan's informative value and usefulness.  Aggregating
program activities consists of applying the same performance goals
and measures to several program activities within a budget account. 
Disaggregating consists of dividing individual program activities in
a P&F schedule into component parts and applying performance goals
and measures specifically to component parts.  Consolidating program
activities consists of applying a single set of performance goals and
measures to program activities from two or more budget accounts. 
(See figure I.5.) According to OMB, it is through aggregation or
consolidation that an annual plan can cover every program activity in
the P&F schedules while allowing an agency to omit goals for its
nonmajor activities.  The significance of any program activity
constituting a major function or operation for an agency is less
likely to be overlooked or minimized if the agency aggregates or
consolidates program activities. 

   Figure I.5:  Aligning Annual
   Performance Plans With the
   Budget Account and Program
   Activity Structure

   (See figure in printed
   edition.)

Because many P&F schedules are not mission- or goal-oriented, it may
be difficult for agencies to clearly link program activities to
performance goals.  As a by-product of Results Act implementation
efforts, an agency may develop and propose changes to the program
activity structure in its budget request, but any proposed changes
must be coordinated with OMB and the cognizant appropriations
subcommittees.  Although the agency's budget may show such proposed
changes, and changes may occur in the future to better align with
performance measurement, at this time, it is the evaluator's
responsibility to ensure that the major functions embedded in the
agency's program activity sections are covered by the performance
goals and measures. 

For key programs that do not appear in the program activity structure
of the president's budget (e.g., certain programs funded by Medicare
and Medicaid, such as the Rural Health Clinic program and the
Medicare Incentive Payment program), evaluators should determine
whether these programs constitute major functions or operations and
have goals that are unique and distinguishable from their
encompassing program activity.  If so, the encompassing program
activity should be disaggregated to show the results these programs
are expected to achieve.  In addition, the performance plan may
include performance goals for functions or operations that are staff
or support-type activities, even if such goals do not correspond to a
particular P&F schedule or program activity. 

Whether the program activities used in the performance plan directly
correspond to the budget or are aggregated, disaggregated, or
consolidated specifically for performance planning, they should
indicate the link between performance goals and the budgetary
resources the agency is requesting to pursue the goals.  Performance
goals should be set based on the funding proposed to achieve them. 

In addition, an agency may ask OMB to authorize it to have no
performance goals in its performance plans for one or more program
activities.  If it was determined to be infeasible or impractical to
express performance goals in any form for a given program activity,
the performance plan should state why. 

Key features to look for include

  -- a diagram, model, or description that clearly shows how each of
     the program activities in each budget account in the agency's
     budget is covered by the annual performance plan; and

  -- if the annual plan has aggregated, disaggregated, or
     consolidated the program activities from the agency's budget, a
     crosswalk that shows how the program activities in the agency's
     budget relate to the aggregated, disaggregated, or consolidated
     program activities. 

The evaluator should

  -- compare the presentation of the agency's program activities in
     its performance plan to the presentation in the P&F schedules to
     assess the annual plan's coverage of the budget's program
     activities;

  -- based on an evaluator's knowledge of the agency's programs,
     assess the reasonableness of any aggregation, disaggregation, or
     consolidation of program activities for purposes of the annual
     plan; and

  -- based on knowledge of the agency's programs, assess whether any
     major function or operation of the agency covered by the program
     activities has not been adequately addressed by the annual
     performance plan.  (If no performance goals have been
     established for a program activity, determine what explanation
     the agency has provided as to why it is infeasible or
     impractical to have any performance goals for the program
     activity). 


--------------------
\13 The term "program activity" refers to the listings of projects
and activities in the Appendix to the Budget of the United States
Government.  Program activity structures are intended to provide a
meaningful representation of the operations financed by a specific
budget account. 


      ISSUE 3:  RECOGNIZING
      CROSSCUTTING EFFORTS
-------------------------------------------------------- Chapter I:1.3

How are agencies coordinating efforts with related strategic or
performance goals?


         CRITERION 3.1
------------------------------------------------------ Chapter I:1.3.1

If applicable, the annual performance plan should identify
performance goals that reflect activities being undertaken to support
programs of a crosscutting nature (OMB Circular A-11, sec.  220.8). 

Guidance.  In many areas of the federal government, programs share
common purposes, although they may operate differently or address
different aspects of a broader national concern.  In their strategic
plans, agencies were encouraged to identify such programs and
crosscutting issues.  The annual performance plan provides an
opportunity, when applicable, for an agency to clarify the
relationship between programs that have related strategic or
performance goals, whether they are in the same or different federal
agencies.  The focus of an agency's performance plan should be on the
agency's performance goals and how it intends to achieve them. 
However, these performance goals should reflect the crosscutting
nature of programs when applicable.  When an agency's program
addresses a crosscutting issue of broad national concern, the agency
should discuss in its performance plan how achieving the program's
goals will contribute to addressing that crosscutting issue. 


         CRITERION 3.2
------------------------------------------------------ Chapter I:1.3.2

If applicable, the annual performance plan should evidence
coordination among crosscutting programs (congressional leadership
letter to the OMB Director, dated December 17, 1997). 

Guidance.  When the need for better coordination or integration of
efforts that crosscut programs or agencies has been noted in the
agency's strategic plan or identified in hearings or in GAO or
Congressional Research Service reports, the annual performance plan
should address this need.  The agency should discuss efforts to
coordinate its programs with other internal and external federal
programs performing related activities and provide evidence of such
coordination (e.g., evidence of joint planning and coordination,
relationship of programs' goals and measures, discussion of the
agency's contribution to the crosscutting issue). 

The evaluator should

  -- determine whether the agency's performance plan identifies
     performance goals for crosscutting issues raised in the agency's
     strategic plan;

  -- determine whether the agency's performance plan identifies
     performance goals for other crosscutting issues related to the
     agency's programs and identified in hearings or in GAO or
     Congressional Research Service reports; and

  -- determine whether the plan discusses how it has coordinated or
     will coordinate crosscutting efforts with other related federal
     programs. 


STRATEGIES AND RESOURCES
=========================================================== Chapter II


   CORE QUESTION 2
--------------------------------------------------------- Chapter II:1

How well does the performance plan discuss the strategies and
resources the agency will use to achieve its performance goals? 

   Figure II.1:  Summary of the
   Issues and Criteria for Core
   Question 2

   (See figure in printed
   edition.)

Guidance.  The Results Act prescribes that the performance plan
should discuss the operational processes, skills, technology, and
resources an agency will employ to achieve its performance goals. 
GAO and OMB use the term "strategies" when discussing the operational
processes, skills, and technology component of the Act's
requirements. 

The performance plan should communicate how an agency expects to
achieve its performance goals and the level of resources that will be
required.  The plan should describe both its ongoing or existing
strategies and resources and any significant changes to its
strategies and/or resources needed to accomplish the agency's
performance goals.  A performance plan, however, should not contain a
detailed list of an agency's strategies and resources. 

An agency's performance plan may include some performance measures
called "means-type measures" that relate to strategies and resources,
such as redistributing workload or replacing computer systems. 
However, according to OMB, these measures do not substitute for a
description of the agency's strategies and resources needed to
achieve its performance goals. 


      ISSUE 4:  CONNECTING
      STRATEGIES TO RESULTS
------------------------------------------------------- Chapter II:1.1

How clear and reasonable are the agency's strategies with respect to
its intended performance goals?


         CRITERION 4.1
----------------------------------------------------- Chapter II:1.1.1

The performance plan should briefly describe the agency's strategies
to accomplish its performance goals (Results Act, sec.  4(b); OMB
Circular A-11, sec.  220.12; and OMB checklist, pp.  6 and 8). 

Guidance.  An agency's strategies to help achieve its performance
goals include activities such as grant making, rulemaking,
inspections, and research as well as administrative processes such as
procurement, real property management, and human resources
management.  Additionally, strategies can include the application of
technology, such as introducing electronic funds transfers to
streamline payment processing or ensuring that all critical
information technology systems can function past the year 1999. 

The performance plan should briefly describe both existing and new
strategies to achieve a performance goal.  The discussion of new
strategies may involve an agency's efforts to improve efficiency and
effectiveness through such approaches as streamlining, contracting
out, privatization, partnering, business process reengineering, and
franchising.  An agency's performance plan should also discuss any
regulation, tax expenditure, and/or proposed legislation that affects
the achievement of a specific performance goal.  Additionally, the
strategies discussed in the performance plan should be consistent
with the strategies section of an agency's strategic plan. 

Experience has shown that, while this level of specificity is not
required by the Results Act, the better performance plans not only
describe the strategies needed, but also provide a rationale as to
how the strategies will contribute to accomplishing the expected
level of performance.  For example, an agency's simply stating that
it will administer block grants to states, without indicating what
the states will do with the money, would fail to link strategies to
intended results. 

Typically, the descriptions of strategies included in performance
plans should be brief.  However, an agency is expected to provide
more details on its strategies when it is planning a significant
change to its existing operational processes, skills, or technology. 
Also, the level of detail will vary depending on the requirements and
needs of particular congressional committees. 

Key features to look for include

  -- linkages between an agency's strategies and its performance
     goals;

  -- distinctions between existing and new strategies;

  -- a rationale as to how the strategies will contribute to
     improving performance;

  -- if applicable, a description of the agency's streamlining,
     contracting out, privatization, business process reengineering,
     franchising, and other approaches for achieving greater
     efficiency and effectiveness in agency operations;

  -- if applicable, a discussion of the agency's intended use of
     regulations or reliance on tax expenditures to achieve
     performance goals;

  -- if applicable, a description of any proposed legislation needed
     to help achieve performance goals; and

  -- references in the performance plan to other plans and documents
     (e.g., business plan, reorganization plans, training plans) that
     show how the identified strategies will contribute to achieving
     the agency's performance goals. 

The evaluator should

  -- based on an evaluator's knowledge of an agency, assess the
     reasonableness of the agency's rationale as to how the
     strategies will contribute to achieving the performance goals;

  -- identify changes to existing strategies or new initiatives and
     assess whether the agency provided a reasonable explanation of
     why the changes are necessary;

  -- review previous work by GAO, the agency's inspector general, and
     others to identify unresolved recommendations for improving the
     agency's processes, skills, and technology; and

  -- identify whether the agency's strategies sufficiently
     incorporate corrective actions recommended by GAO and/or the
     agency's inspector general. 


         CRITERION 4.2
----------------------------------------------------- Chapter II:1.1.2

If applicable, an agency's proposed waiver of administrative
procedural requirements and controls should be described in a manner
that quantifies expected performance improvements (Results Act, sec. 
5(a)). 

Guidance.  Agencies are authorized to apply, in their annual
performance plans, to OMB for waivers to administrative procedural
requirements and controls to provide agencies with managerial
flexibility.  The waiver request must describe and quantify any
anticipated effects on an agency's performance.  It also must be
endorsed by the agency that imposed the requirement or control, which
usually involves one of the four central management agencies--OMB,
the General Services Administration, the Office of Personnel
Management, and the Financial Management Service in the Department of
the Treasury.  Such waivers are intended to provide federal managers
with more flexibility to structure agency systems to better support
performance goals.  They may involve administrative requirements such
as staffing and the internal allocation and use of resources.  An
example of increased flexibility would be to allow an agency to
recapture operating funds that remain unspent because of increased
efficiencies and then use the funds to purchase new equipment or
expand employee training.  The waivers, however, do not involve
statutory requirements and controls. 

Key features to look for include

  -- a discussion by the agency of the waiver requested;

  -- a description of the managerial flexibility that would be
     achieved if the waiver is approved;

  -- a discussion of the anticipated effects on performance that will
     result from greater managerial flexibility;

  -- a description of the likelihood of the agency's achieving a
     specific performance goal if the waiver is not granted; and

  -- an endorsement of the waiver from the agency that imposed the
     administrative requirement or control. 

The evaluator should

  -- assess whether the agency provided reasonable descriptions of
     the managerial flexibility proposed and the impact on intended
     performance if the waiver is not granted;

  -- assess whether the agency's quantification of the anticipated
     effects on performance appears reasonable; and

  -- verify that the endorsement is included in the description. 


         CRITERION 4.3
----------------------------------------------------- Chapter II:1.1.3

The plan should discuss an agency's actions to address external
factors that are likely to affect its performance. 

Guidance.  Whether an agency achieves its performance goals can be
influenced by a number of external factors, such as emerging
economic, social, and technological trends and the role of third
parties (e.g., state and local governments).  Thus, the Results Act
requires agencies to identify, in their strategic plans, external
factors that could significantly affect performance.  GAO believes
that agencies' performance plans, although not required to by the
Results Act, should also discuss external factors in order to provide
additional context for anticipated performance.  OMB prescribes that,
if external factors are identified in an agency's performance plan,
they should be consistent with the external factors identified in the
agency's strategic plan. 

Key features to look for include

  -- consistency between the external factors affecting performance
     described in the performance plan and the external factors
     identified in the agency's strategic plan or previous reviews of
     that plan;

  -- an explanation of how the agency's performance will be
     positively or negatively affected by such external factors; and

  -- a discussion of partnerships or working relationships with other
     organizations aimed at achieving performance goals. 

The evaluator should

  -- determine whether the agency's discussion of external factors is
     consistent with its strategic plan and addresses any issues
     raised in reviews of that plan; and

  -- assess whether the agency provided a reasonable discussion of
     how it would mitigate or use the identified conditions to
     achieve its performance goals. 


      ISSUE 5:  CONNECTING
      RESOURCES TO STRATEGIES
------------------------------------------------------- Chapter II:1.2

What capital, human, financial, and other resources are being applied
to achieve the agency's performance goals?


         CRITERION 5.1
----------------------------------------------------- Chapter II:1.2.1

The performance plan should briefly describe the capital, human,
financial, and other resources being applied to achieve the
performance goals (Results Act, sec.  4(b); OMB Circular A-11, sec. 
220.12; and OMB checklist, p.  6). 

Guidance.  OMB Circular A-11 requires that the description be brief
and limited to resources (dollars and personnel) reflected in an
agency's budget request.  The level of detail, however, will vary
depending on the requirements and needs of congressional committees. 
Also, an agency may be expected to provide more details on the
resources it needs when the agency is requesting a significant
change.  The performance plan should also include prior year
resources available for the agency's current fiscal year performance
goals, as well as current year resources available for performance
goals to be achieved in a future year. 

Experience has shown that, while such specificity is not required by
the Results Act, the better performance plans not only describe the
resources needed, but also provide a rationale for how the resources
will contribute to accomplishing the expected level of performance. 

Identifying the specific resources associated with an individual
performance goal will pose a challenge for some agencies, as well as
for evaluators evaluating those agencies' performance plans.  For
example, funding from one budget activity may provide support for
more than one performance goal.  Agencies do not need to account for
every dollar allocated to a specific performance goal. 

To achieve a specific performance goal, agencies may need to invest
and acquire new capital assets.  When this is the case, OMB Circular
A-11 requires that performance plans make reference to the agency's
capital assets plan and briefly describe new capital acquisitions
that will significantly affect the achievement of performance goals. 
Capital assets are defined by OMB as land, structures, equipment
(including information technology), and intellectual property,
including software, that are used by the federal government and have
an estimated useful life of 2 years or more.  Some agencies (such as
the Department of Defense and the National Aeronautics and Space
Administration) rely more heavily on capital asset investments than
others to achieve their performance goals.  In addition to capital
asset plans, performance plans should refer to the agency's
information technology and financial management plans, where such
resources are identified as needed to achieve the performance goals. 

Key features to look for include

  -- an allocation of resources (dollars and personnel) to
     performance goals;

  -- a discussion of the effect increases or decreases in resources
     (dollars and/or personnel) may have on expected performance;

  -- references in the annual performance plan to the agency's
     capital assets, information technology, and financial management
     plans;

  -- a description of proposed major capital investments that will
     bear significantly on the achievement of performance goals;

  -- linkages between specific capital investments and individual
     performance goals;

  -- consistency between the capital assets' performance goal(s) and
     related annual performance goals; and

  -- consistency between the resource needs discussed in the
     performance plan and an agency's budget justification documents. 

The evaluator should

  -- assess the reasonableness of the performance plan's discussion
     of the proposed resources for achieving the performance goals
     and how the resources were allocated to each goal;

  -- assess whether the performance plan provides a clear rationale
     for how the resources will contribute to improving performance;

  -- verify that the performance plan references the proposals in the
     agency's capital assets and information technology plans when
     appropriate;

  -- determine whether the capital assets in the performance plan
     sufficiently reflect the goals and costs of those assets as
     presented in the capital assets plan; and

  -- assess whether the agency's performance plan or capital assets
     plan provides a plausible explanation of how a specific capital
     asset investment will contribute to achieving a performance
     goal. 


VERIFICATION AND VALIDATION
========================================================== Chapter III


   CORE QUESTION 3
-------------------------------------------------------- Chapter III:1

To what extent does the agency's performance plan provide confidence
that its performance information will be credible? 

   Figure III.1:  Summary of the
   Issues and Criteria for Core
   Question 3

   (See figure in printed
   edition.)

Guidance.  To be able to assess progress toward the achievement of
performance goals, the measures used must be valid and reliable. 
Reliability refers to the precision with which performance is
measured, while validity is the extent to which the measure
adequately represents actual performance.  The reliability and
validity of the measures are influenced by a number of factors,
including the quality of the data on which the measures are based. 
Performance measures may be derived from data collected and
maintained by an agency or obtained from external sources.  In order
for measures to be valid and reliable, the data on which they are
based must be free from significant error, especially bias.  This
section discusses the influence of data quality.  Other
considerations in the choice of reliable and valid measures are
discussed under question 1. 

To assess whether the goals and measures proposed in a performance
plan will provide a reasonable and appropriate basis for assessing
performance, the reader must be able to judge whether the proposed
performance data will be sufficiently free of error, in particular
bias.  To this end, the Results Act calls for the plan to describe
the procedures that will be used to verify and validate the measured
values of actual performance.  In this guide, verification means the
checking or testing of performance data to reduce the risk of using
data that contain significant errors.  Validation means the testing
of data to ensure that no error creates significant bias. 
Significant error, including bias, would affect conclusions about the
extent to which performance goals have been achieved. 

The reasonableness and appropriateness of proposed measures are also
influenced by the extent to which the needed data can be obtained at
a reasonable cost--i.e.  the extent to which benefits obtained from
providing the data outweigh the costs of producing it. 

The role of the evaluator is to

  -- assess whether the agency has provided sufficient information to
     permit an informed judgment by the reader of whether the
     performance data will be sufficiently free of bias and other
     significant error, and

  -- determine whether the verification and validation procedures and
     the data proposed by the agency are credible. 

The credibility of the verification and validation procedures and of
the data is to be determined by assessing whether there are any
problems with them that would influence conclusions about the
achievement of results. 


      ISSUE 6:  VERIFYING AND
      VALIDATING PERFORMANCE
------------------------------------------------------ Chapter III:1.1

How will the agency ensure that its performance information is
sufficiently complete, accurate, and consistent?


         CRITERION 6
---------------------------------------------------- Chapter III:1.1.1

The plan should describe credible procedures to verify and validate
performance information (Results Act, sec.  4(b); OMB A-11, sec. 
220.13). 

Guidance.  The Results Act requires that performance plans include
descriptions of procedures for verifying and validating the measured
values of actual performance (Results Act, sec.  4(b)).  OMB
encourages agencies to discuss the verification and validation
procedures with their OMB representatives (OMB A-11, sec.  220.13). 
Agencies should have in place or propose credible procedures for
ensuring that their performance data are reasonably complete,
accurate, and consistent. 

The data required for performance measurement can be collected on an
ongoing basis by agency personnel during the performance of duties or
by means of periodic studies.  For example, client data can be
recorded on an ongoing basis during the determination of eligibility
for services, or data can be collected in the processing of financial
transactions.  Client surveys are an example of periodic studies,
which may be required for some measures of performance--for example,
for measures of service quality. 

Error can occur at various points in the collection, maintenance,
processing, and reporting of data.  At data collection, information
can be recorded incorrectly.  For example, program personnel can
incorrectly record information about clients.  Inadequate procedures
for maintaining data, such as on computer information systems, can
result in the loss or alteration of data.  For instance, data from
client survey forms can be incorrectly entered into computer
information systems.  Processing and reporting of data can also add
error through, for example, the incorrect transfer of data from
computerized databases into analyzable or reportable forms. 

Error that is caused by a systematic source can introduce bias and
lead to inaccurate estimates of program performance.  For example, in
programs intended to reduce such inappropriate behaviors as alcohol
use, speeding, smoking, etc., clients entering these programs may be
asked to estimate the extent to which they engage in these behaviors,
as a baseline against which to judge program effectiveness.  Clients
may initially underestimate these undesirable behaviors, ultimately
leading to underestimates of program success. 

As another example, a program designed to support the development of
small businesses by providing financial assistance may do follow-up
of recipients to assess their success.  Some recipients may become
unreachable for a variety of reasons, including name changes,
relocation, or failure.  If many of those small enterprises that go
out of business are unreachable, they would not show up in the
database, and the data would then contain a greater proportion of
successful businesses than is the actual situation.  Thus, the
ability of the program to stimulate small enterprises could be
overestimated. 

The agency should have in place procedures for ensuring that the data
are free of significant levels of error and that bias is not
introduced.  These procedures can include internal controls over such
matters as data collection, maintenance, and entry.  Particularly
important are external assessments such as audits, evaluations, and
peer reviews.  One such external assessment, of particular importance
when goals and measures are financially oriented, is the audit of
agency annual financial statements, required under the Chief
Financial Officers (CFO) and Government Management Reform acts. 
Whichever verification and validation procedures are used, their
description needs to be sufficiently detailed to permit an assessment
of their adequacy. 

Key features to look for include

  -- a clear description of the data verification and validation
     procedures to be used to ensure that significant error,
     including bias, are not added during collection, maintenance, or
     processing of data;

  -- general procedures to control data quality, such as general
     controls for computerized information systems; and

  -- procedures specific to the data required for performance
     measures proposed in the performance plan, including data
     gathered both on an ongoing and a periodic basis. 

The described procedures are credible if they

  -- provide for periodic review of data collection, maintenance, and
     processing procedures by the agency to ensure that they are
     consistently applied and continue to be adequate;

  -- provide for periodic sampling and review of data to ensure their
     completeness, accuracy, and consistency;

  -- rely on independent audits or other established procedures for
     verifying and validating financial information when performance
     measures require the use of financial information; and

  -- address problems, in verification and validation procedures,
     known to GAO or the agency. 

Credibility is enhanced if the procedures call for formal evaluation
or audit of the information by parties external to the program in
question. 

The evaluator should

  -- review the performance plans to identify references to
     verification and validation procedures;

  -- review previous GAO work, or other available sources, for
     instances where procedures for verifying and validating agency
     performance information have been found deficient;

  -- where performance measurement requires financial information,
     review the annual audits of the agency's annual financial
     reports to identify instances where controls over related
     financial information were found to be deficient;

  -- interview agency staff to determine whether they are taking or
     are proposing to take actions to address previously identified
     deficiencies in verification and validation procedures;

  -- interview agency staff to determine the extent to which OMB has
     been consulted on proposed verification and validation
     procedures prior to submission of the plan, and inquire as to
     concerns raised by OMB and whether and how the agency addressed
     them; and

  -- assess whether GAO and OMB concerns were addressed in the
     validation and verification procedures described in the plan and
     whether the procedures are credible. 


      ISSUE 7:  RECOGNIZING DATA
      LIMITATIONS
------------------------------------------------------ Chapter III:1.2

To what extent does the performance plan identify significant data
limitations and their implications for assessing the achievement of
performance goals? 

Guidance.  The specific performance data required--and the means for
collecting, maintaining, and analyzing them--should be identified and
described in detail sufficient to allow an assessment of the extent
to which they can be relied upon (OMB checklist, p.  7). 

Evaluators should rely on previously conducted work and on
departmental sources to determine whether there are any known
problems with the data sources or the data that would cast doubt on
the credibility of the information.  Known limitations are those
identified by GAO in its work or in available documents such as
program evaluations, independent audits, inspector general
investigations, information systems analyses, etc.  For example,
external audits of agency annual financial reports, required under
the CFO and Government Management Reform acts, have identified
problems with data accuracy, completeness, and consistency. 

No data collection and maintenance process is free of error. 
Significant limitations are errors that would lead to inaccurate
assessment of goal achievement.  Proposals for performance data
recognize known limitations to the extent that

  -- they do not propose to use data with known problems that would
     lead to inaccurate assessment of the extent of achievement of
     performance goals, or

  -- they describe means taken or proposed to address these
     weaknesses. 

Potentially significant data limitations include

  -- inconsistencies in data collection from location to location,
     from one time period to another, or from one data source to
     another, when data from more than one source must be combined to
     create a measure (Inconsistencies can arise when standardized
     procedures are not used or followed.  For example, this could
     occur when data are not clearly defined.  It could also occur
     when definitions change, resulting in noncomparable data--for
     instance, if the number of people finding permanent employment
     is used to measure an outcome and the definition of ï¿½permanent
     employmentï¿½ changes in a way that affects the number of people
     fitting the category.);

  -- inaccuracies due to imprecise measurement and recording (for
     example, program personnel incorrectly recording client
     information); and

  -- incomplete data (for example, not all cases can be identified
     and recorded, data are backlogged and not up to date, data are
     lost when an information system crashes). 

The significance of these limitations increases if the error
constitutes bias.  For example, incomplete data can result in bias if
certain types of program clients are less likely to be recorded in a
database because they are harder to reach.  Certain types of data
collection are more prone to bias.  For example, management or grant
recipient subjective ratings of outcomes may lead to inaccuracies
that reflect their interests. 

Resolving limitations to the data needed for performance measurement
can pose serious challenges to agencies and take time.  Some agencies
may not have available all the needed data because of the challenges
involved.  GAO's previous review of the Department of Health and
Human Service's April 1997 draft strategic plan observed that some
federal data on the Medicaid program are of questionable accuracy.\14
Some of the problems stem from data collected from different states,
all of which do not use identical definitions for data categories. 
It is not expected that agencies will have resolved all significant
limitations, but that they are at least taking reasonable steps to
address them.  In some instances, however, previous work may have
revealed that the agency was not making reasonable progress in
resolving the challenges. 


--------------------
\14 HHS' Draft Strategic Plan (GAO/HEHS-97-173R, July 11, 1997). 


         CRITERION 7.1
---------------------------------------------------- Chapter III:1.2.1

Known significant limitations to data from agency sources should be
recognized in the performance plan. 

Key features to look for include

  -- a description of limitations to the data, the significance of
     the limitations for judging the extent to which goals have been
     achieved, and the steps being taken or proposed to address the
     problems (OMB checklist, p.  7; congressional leadership letter
     to the OMB Director, dated Dec.  17, 1997). 

The evaluator should

  -- review previous work by GAO, or other sources, to determine
     whether previously identified potentially significant
     limitations to the proposed performance information have not
     been discussed in the performance plan;

  -- where performance measurement requires financial information,
     review the annual audits of the agency's annual financial
     reports to identify instances where significant limitations have
     been found in related financial information;

  -- assess the means used to collect the data to determine whether
     the performance plan has not discussed weaknesses inherent in
     the form of data collection itself that could result in
     significant error (examples of such forms include management or
     client ratings and estimates of outcomes);

  -- assess whether actions described in the plan as being taken or
     planned to address the known data limitations appear reasonable;
     and

  -- interview agency officials to determine whether reasonable
     action, other than that described in the performance plans, is
     being taken or proposed to address potentially significant
     limitations or, alternatively, whether there is reason to
     believe that the limitations do not apply to the particular use
     intended for the data. 


         CRITERION 7.2
---------------------------------------------------- Chapter III:1.2.2

The performance plan should indicate when performance data will come
from sources external to the agency (OMB A-11, sec.  220.13) and
should recognize known significant limitations to external data. 

OMB requires that agencies identify in their performance plans data
received from external sources.  OMB specifies that agencies are not
required to develop an independent capacity for verifying or
validating performance data from external sources.  However, agencies
should collect available information from outside sources on the
consistency, accuracy, and completeness of the data, although they
are not required to include such information in the plan.  It is
expected that proposals to rely on external sources should not
include sources with known significant limitations, unless actions
are being taken or proposed to address these limitations.  Better
performance plans will also identify limitations to the data, discuss
their significance, and describe steps being taken or proposed to
address the problems (OMB checklist, p.  7; congressional leadership
letter to the OMB Director, dated Dec.  17, 1997). 

The evaluator should

  -- ascertain, based on previous reviews, whether external sources
     (for example, information produced by other agencies, state
     governments, or independent organizations) are used but not
     identified;

  -- determine whether the external sources to be relied on are
     recognized and generally accepted as reliable sources of this
     data (e.g., statistical offices, research organizations, etc.);

  -- interview agency officials to ascertain whether an effort was
     made to collect available information regarding the
     completeness, accuracy, and consistency of the information as
     well as on whatever limitations may have been identified;

  -- examine prior work to identify whether there are any significant
     problems with the completeness, accuracy, and consistency of the
     proposed external data; and

  -- assess whether known data limitations have been adequately taken
     into account in the performance plan. 


         CRITERION 7.3
---------------------------------------------------- Chapter III:1.2.3

The performance plan should discuss or reference any significant new
information systems or major changes to existing systems that have
been proposed to make more credible data available for performance
measures. 

When existing data contain significant limitations, providing
credible data may require new information systems or major changes to
existing systems.  In deciding whether to continue to rely on
existing data or embark on major development or modification efforts,
the agency will have to balance the incremental costs of different
data collection approaches against the need to ensure that the
collected data are complete, consistent, and accurate enough to
document performance and support decisionmaking. 

Excessively costly performance data may reduce the feasibility of
performance measurement, needlessly direct resources away from
program or service delivery, or place an undue burden on program
clients or third parties.  Conversely, costs may be offset by the
potential benefits of more efficient and effective programs and
greater accountability.  They may also be offset when performance
measurement is implemented as part of managing for results, through
the elimination or reduction of outdated management practices. 
Agencies can also reduce costs by building performance data
collection into the processes that govern daily operations, rather
than creating entirely new and separate data systems. 

Potential costs include the management and staff time required to
collect, maintain, analyze, and use performance data; the costs of
any contracts for data collection and analysis; related equipment
costs; the burden imposed on those required to report the data;
political and bureaucratic costs; and other important negative
consequences of performance measurement. 

Agencies are not required to report on the costs of collecting,
maintaining, and reporting performance information.  However, better
performance plans discuss the costs and benefits of proposed
information collection approaches, and of alternatives when the
proposed performance measures are costly to implement.  Although
evaluators are not expected to assess whether costs are reasonable,
recognition should be given to agencies that report the costs of
collecting performance data, the costs of alternative data sources,
or other information that allows the reader to assess whether the
benefits of collecting better data outweigh the costs. 

The evaluator should

  -- examine the agency's capital plan to assess whether new
     information systems, or significant modifications to existing
     systems, are proposed to implement performance measures and
     whether these proposals are appropriately referenced in the
     performance plan. 


*** End of document. ***