Agencies' Annual Performance Plans Under the Results Act: An Assessment
Guide to Facilitate Congressional Decisionmaking (Guidance, 02/01/98,
GAO/GGD/AIMD-10.1.18).
GAO published an assessment guide to facilitate congressional use of
agencies' performance plans developed under the Government Performance
and Results Act, focusing on the: (1) annual performance goals and
measures for each agency's program activities; (2) strategies and
resources needed to achieve annual performance goals; (3) procedures the
agency will use to verify and validate its performance data; and (4)
issues that need to be addressed, and key assessment questions that can
help congressional users elicit the cost and performance information on
the Results Act's planning and reporting requirements as well as related
guidance on annual performance plans.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GGD/AIMD-10.1.18
TITLE: Agencies' Annual Performance Plans Under the Results Act:
An Assessment Guide to Facilitate Congressional
Decisionmaking
DATE: 02/01/98
SUBJECT: Congressional/executive relations
Congressional oversight
Strategic planning
Agency missions
Data integrity
Reporting requirements
Interagency relations
IDENTIFIER: Government Performance and Results Act
GPRA
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
General Government Division
Accounting and Information Management Division
February 1998
Version 1
AGENCIES' ANNUAL PERFORMANCE PLANS
UNDER THE RESULTS ACT - AN
ASSESSMENT GUIDE TO FACILITATE
CONGRESSIONAL DECISIONMAKING
GAO/GGD/AIMD-10.1.18
GAO/GGD-98-999
Congressional Review of Performance Plans
(410193/935250)
Abbreviations
=============================================================== ABBREV
PREFACE
============================================================ Chapter 0
As Congress seeks to reduce the cost and improve the performance of
the federal government, there is a broad consensus that holding
executive agencies accountable for results is a key to better
management of programs. Further, the lack of precise information on
agencies' performance has handicapped congressional policymaking,
spending decisions, and oversight. Congress enacted the Government
Performance and Results Act of 1993 (Results Act) to help address
these concerns.
Performance-based management, as envisioned by the Results Act, is a
dynamic and complementary process of setting a strategic direction,
defining annual goals and measures, and reporting on performance.
Under the Results Act, agencies are to prepare multiyear strategic
plans that set the general direction for their efforts. Then,
agencies are to prepare annual performance plans beginning in fiscal
year 1999 that establish the connections between the long-term
strategic goals outlined in the strategic plans and the day-to-day
activities of managers and staff. Finally, the Act requires that
each agency report annually on the extent to which it is meeting
annual performance goals and the actions needed to achieve or modify
those goals that have not been met.
This guide is intended to facilitate congressional use of agencies'
annual performance plans. Although the Results Act does not require
a specific format for the performance plan, it does require the plan
to (1) identify annual performance goals and measures for each of an
agency's program activities, (2) discuss the strategies and resources
needed to achieve annual performance goals, and (3) provide an
explanation of the procedures the agency will use to verify and
validate its performance data. This guide is organized around three
core questions that correspond to these general requirements. For
each core question, we identify issues that need to be addressed and
present key assessment questions that can help congressional users
elicit the cost and performance information relevant to their
decisionmaking. We also provide information on the Results Act's
planning and reporting requirements as well as related guidance on
annual performance plans in the back of this guide.
We expect that as agencies and Congress gain experience in developing
and using annual performance plans, additional issues and questions
will emerge. Accordingly, after agencies have submitted their first
annual performance plans, we plan to update this guide and issue a
subsequent version to reflect agencies' and Congress' experiences.
In addition, the next version of this guide will include examples
showing how agencies are presenting information in their performance
plans.
We invite comments on this guide. Please contact us or J.
Christopher Mihm, Associate Director, Federal Management and
Workforce Issues, at (202) 512-8676 ([email protected]) or Susan J.
Irving, Associate Director, Budget Issues, at (202) 512-9142
([email protected]). Other contributors to this guide were Alan
M. Stapleton, Michael J. Curro, Lisa R. Shames, Laura E. Castro,
and Dorothy L. Self. An electronic version of this guide is
available from GAO's World Wide Web server at the following Internet
address: http://www.gao.gov.
L. Nye Stevens
Director, Federal Management and
Workforce Issues
General Government Division
Paul L. Posner
Director, Budget Issues
Accounting and Information Management
Division
INTRODUCTION
Annual performance plans are to systematically provide congressional
decisionmakers with information on the results to be achieved for a
proposed level of resources. Specifically, the plans should clearly
inform Congress and the public of the annual performance goals for
agencies' major programs and activities, the measures that will be
used to gauge performance, the strategies and resources required to
meet the performance goals, and the procedures that will be used to
verify and validate performance information.
With its requirement for annual performance plans, the Results Act
establishes the first statutory link between agencies' budget
requests and their performance planning efforts. The Act requires
that performance goals and measures be linked to program activities
in agencies' budget requests. Also, agencies' performance plans are
to be the basis for the federal government performance plan that the
Office of Management and Budget (OMB) must submit to Congress. The
federal government performance plan is intended to present a single,
cohesive picture of annual performance goals for the fiscal year and
must be provided to Congress as part of the president's budget
submission each February. Agencies' annual performance plans are to
be sent to Congress and made available to the public soon after
transmittal of the president's budget.
To ensure that the information in their performance plans is
presented clearly and concisely, agencies should seek to establish a
clear hierarchy of performance goals and measures. That is, lower
organizational levels of an agency may use goals and measures that
are different from those meaningfully or appropriately included in
the agency's annual performance plan. Thus, as performance plans are
compiled for higher organizational levels, there may be consolidation
and possible exclusion of some goals and measures, which does not
mean that those goals and measures are not important for guiding the
efforts of the lower levels. Agencies that seek to include an
excessive number of performance goals and measures in their annual
performance plans may risk creating a confusing excess of data that
will obscure rather than clarify performance issues.
The experiences of agencies to date suggest that the successful
implementation of the Results Act may be as difficult as it is
important.\1 Translating the use of agency resources into concrete
and measurable results will be a continual challenge that will
require both time and effort on the part of the agency. Our work
found that the strategic plans that agencies produced in September
1997 provide a workable foundation for the annual performance
planning and reporting phases of the Results Act. However, we also
found that agencies need to make continued progress in setting a
strategic direction, coordinating crosscutting programs, and ensuring
the capacity to gather and use reliable performance and cost data.\2
As agencies prepare their performance plans, they can identify
opportunities to address these planning challenges. With sustained
attention from Congress and agencies, annual performance plans can be
an invaluable tool for informing policy decisions, improving program
management, enhancing accountability, and helping to increase
American citizens' confidence in their government.
HOW THIS GUIDE WAS DEVELOPED
In May 1997, we developed a guide to facilitate congressional review
of agencies' strategic plans at the request of the Chairmen of the
Government Reform and Oversight Committee, Committee on
Appropriations, and Committee on the Budget in the House of
Representatives.\3 They saw a need for a tool to facilitate
congressional consultation on the development of agencies' strategic
plans. The guide we developed contained key questions to help
Congress determine how those plans could be improved to better
support congressional and agency decisionmaking.
Building on that guide, those Chairmen, the Speaker of the House, the
House Majority Leader, the Chairman of the House Committee on
Science, and the Chairmen of the Senate Committee on Budget and
Senate Committee on Governmental Affairs subsequently asked us to
develop a guide to help congressional decisionmakers both elicit the
information that Congress needs from agencies' annual performance
plans and assess the quality of those plans. Together, the answers
to the questions in this guide are intended to facilitate a complete
assessment of an agency's plan and address concerns likely to be
common across a variety of congressional users, although the
significance of individual questions may vary among the
appropriation, authorization, oversight, and budget committees. In
making decisions, Congress can factor in the cost and performance of
the agency along with policy considerations, such as whether the
performance goals are consistent with legislative priorities or the
proposed resources are reasonable to achieve the intended results.
We hope that this guide can also be helpful to agencies in developing
annual performance plans that comply with the Results Act; address
congressional needs; and promote sound, accountable management
practices.
We developed this guide on the basis of the Results Act requirements
for agency performance plans; guidelines contained in the OMB
Circular No. A-11, Part 2; and other relevant documents. The
Results Act has six requirements for the annual performance plans, as
shown in appendix. For purposes of this guide, we collapsed these
six requirements into three core questions. To identify
opportunities to help ensure that annual performance plans would be
most useful for congressional decisionmaking, we also relied on our
work over the last several years examining agencies' efforts to
implement the Results Act and, in particular, on our work identifying
best practices for the Act's effective implementation.\4 We also
developed a companion guide for evaluators to use in their assessment
of agency annual performance plans.
This guide benefited from the insights made by a wide range of
congressional reviewers representing appropriation, authorization,
budget, and oversight committees. We also received and incorporated
valuable comments from senior agency officials directly involved in
the development of annual performance plans, a senior official from
OMB, agency representatives to the Chief Financial Officers (CFO)
Council, individuals from the National Academy of Public
Administration's Consortium on Performance Measurement, and outside
experts.
--------------------
\1 The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).
\2 Managing For Results: Agencies' Annual Performance Plans Can Help
Address Strategic Planning Challenges (GAO/GGD-98-44, Jan. 30,
1998).
\3 Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate
Congressional Review, Version 1 (GAO/GGD-10.1.16, May 1997).
\4 See, for example, Executive Guide: Effectively Implementing the
Government Performance and Results Act (GAO/GGD-96-118, June 1996);
GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996);
Transforming The Civil Service: Building the Workforce of the Future
(GAO/GGD-96-35, Dec. 26, 1995); and Executive Guide: Improving
Mission Performance Through Strategic Information Management and
Technology (GAO/AIMD-94-115, May 1994).
ANNUAL PERFORMANCE PLAN ISSUES
============================================================ Chapter I
CORE QUESTION 1: ANNUAL
PERFORMANCE GOALS AND MEASURES
---------------------------------------------------------- Chapter I:1
To what extent does the agency's performance plan provide a clear
picture of intended performance across the agency?
The Results Act seeks to shift the focus of government accountability
away from a preoccupation with the activities of a federal program to
a focus on results of that program. Well-defined goals and measures
in annual performance plans can help Congress ensure that agencies
make that difficult transition and deliver desired program results.
For example, the plan can be used to set objective and measurable
expectations about what the agency will be held accountable for
achieving. Congress can also use the plan to evaluate whether the
annual goals will put the agency on a path toward achieving its
mission and long-term strategic goals. In addition, the plans can
aid in determining whether individual agencies can work effectively
and supportively with other agencies on crosscutting efforts.
A performance plan that provides Congress with the clear picture of
intended performance needed for decisionmaking addresses the
following issues:
Issue 1: Defining Expected Performance
To what extent do the annual performance goals and measures provide a
succinct and concrete statement of expected performance for
subsequent comparison with actual performance?
Issue 2: Connecting Mission, Goals, and Activities
How are the agency's annual performance goals linked to the agency's
mission, strategic goals, and program activities in its budget
request?
Issue 3: Recognizing Crosscutting Efforts
How are agencies coordinating efforts with related strategic or
performance goals?
ISSUE 1: DEFINING EXPECTED
PERFORMANCE
-------------------------------------------------------- Chapter I:1.1
To what extent do the annual performance goals and measures provide a
succinct and concrete statement of expected performance for
subsequent comparison with actual performance?
Annual performance plans can help Congress strengthen government
accountability. The performance goals and measures presented in the
annual performance plans will provide Congress a basis for comparing
a program's proposed level of performance against its actual
performance.
In developing annual performance goals, an agency should focus on the
results it expects its programs to achieve--the differences the
programs will make in people's lives. Outcome goals should be
included in an agency performance plan, whenever possible. However,
most plans will supplement outcome goals with output goals--such as
the number of activities--because, for example, the outcome goal may
not be achieved in the year covered by the plan. The Results Act
specifies that most annual performance goals should define an
objective, quantifiable, and measurable target level of performance
for each program activity. However, if an agency cannot set
performance goals that are objective, quantifiable, and measurable,
it may propose, and OMB may authorize, that a goal be expressed in an
alternative form, such as by describing a minimally effective program
and a successful program. As another option, an agency may propose,
and OMB may authorize, that no performance goals be established for a
particular program activity if it is infeasible or impractical.
Annual performance plans should also include performance measures to
show the progress the agency is making in achieving the goals.
Performance should be assessed using both nonfinancial and financial
performance measures, including program cost. We found that agencies
that were successful in measuring their performance generally had
developed performance measures with four characteristics.
1. Demonstrate results
Performance measures should tell each organizational level how well
it is achieving its goals. Our work has found that developing
measures that demonstrate results poses an especially difficult
challenge for agencies.\5 Agencies that exert a greater degree of
influence in accomplishing their intended programs, such as those
that run production operations, tend to have an easier time
identifying results-oriented performance measures. Other agencies,
such as those with research or regulatory programs, exert a lesser
degree of influence on the achievement of program results and face
challenges in their ability to identify meaningful measures.
Similarly, we observed that designing results-oriented performance
measures for intergovernmental programs, such as those using block
grants, is complicated by the broad range of objectives identified
for some programs and the discretion states have in achieving those
objectives.\6
2. Limited to the vital few
The number of measures for each goal at a given organizational level
should be limited to the vital few. Those vital few measures should
cover the key performance dimensions that will enable each
organizational level to assess accomplishments and costs, make
decisions, realign processes, and assign accountability.
Organizations that seek to manage an excessive number of performance
measures may risk creating a confusing excess of data that will
obscure rather than clarify performance issues. Agencies should seek
to establish a clear hierarchy of performance goals and measures.
For this reason, an agency will have more performance goals and
measures used by its lower organizational levels than can be
appropriately or meaningfully included in the agency's annual
performance plan.
3. Respond to multiple priorities
When developing performance measures, agencies must take into account
that most programs are forced to strike difficult balances among
competing demands. These competing demands generally include program
cost, service quality, customer satisfaction, and other stakeholder
concerns.
4. Link to responsible programs
Performance measures should be linked directly to the offices that
have responsibility for making programs work to reinforce
accountability and ensure that day-to-day activities contribute to
the results the organization is trying to achieve.
--------------------
\5 GAO/GGD-97-109, June 2, 1997.
\6 Block Grants: Issues in Designing Accountability Provisions
(GAO/AIMD-95-226, Sept. 1, 1995).
ISSUE 2: CONNECTING
MISSION, GOALS, AND
ACTIVITIES
-------------------------------------------------------- Chapter I:1.2
How are the agency's annual performance goals linked to the agency's
mission, strategic goals, and program activities in its budget
request?
Using agency performance plans can help Congress determine whether
the agency will be able to produce desired performance over the short
and long term and whether the benefits of performance justify the
expected costs. Our work has documented programs that have shown few
or no benefits over the long term despite the millions of dollars
that are spent annually on them.\7 By demonstrating how annual
performance goals relate to the strategic goals and the budget
request's program activities, annual performance plans can help
Congress ensure that agencies have a clear roadmap for efficiently
transforming their program activities into long-term performance.
However, we found that many agencies' strategic plans had not clearly
established linkages between their long-term strategic goals and
annual performance goals.\8 The Results Act envisioned that strategic
and annual performance planning cycles would be iterative, mutually
reinforcing processes. Consequently, agencies can continue to
strengthen this linkage as they develop their annual performance
plans. For example, the process of defining targeted levels of
performance within set time frames and providing baselines against
which to compare actual performance will likely identify
opportunities for agencies to revisit and improve upon their
strategic goals.
The Results Act expects that agencies' plans provide a means of
showing how budgetary resources will be used to achieve goals. The
Act requires that performance plans cover each program activity in an
agency's budget.\9 Program activity structures can differ from agency
to agency, and within an agency, from budget account to budget
account.\10
Program activities, like budget accounts, have developed over time in
response to specific needs and may represent programmatic, process,
organizational, or other orientations. Therefore, some of the
program activities that are currently used in an agency's budget
request may be either too detailed or too general, or they may lack a
clear relationship to the programs that are useful for measuring
agencies' performance.\11
The Results Act permits agencies to aggregate, disaggregate, or
consolidate the budget's program activities so that they align with
the goals presented in the performance plan. The adjusted program
activities that are used in the performance plan should continue to
represent the programs or functions of concern to both the agency and
Congress while being useful for performance planning. The Act
expects that agencies will use aggregation, disaggregation, and
consolidation to highlight important activities while avoiding
voluminous presentations that would overwhelm a reader. In addition,
agencies may propose changing their budget structures to conform to
the structures they use for annual performance planning purposes.
Approaches to aligning the annual performance plan with the budget
account structure by means of aggregating, disaggregating, and
consolidating program activities are illustrated in figure 1.
Aggregation combines program activities within one of an agency's
budget accounts. For example, performance goal 1 is defined for the
activities and funding represented by the aggregated budget program
activities 1 and 2 from budget account 1.
Disaggregation breaks a single program activity in one budget account
into one or more activities. For example, disaggregation divides
budget program activity 3 into two different activities, activity 3a
and 3b. As a result, the budget program activity 3 is now covered by
two goals; performance goal 2 covers the disaggregated activity 3a,
and performance goal 3 covers the disaggregated activity 3b.
Consolidation combines some or all of the program activities in two
separate budget accounts to form a single program activity that
appears in the performance plan. For example, performance goal 4
covers the consolidation of program activity 4 of budget account 1
and program activity 1 of budget account 2.
Figure 1: Aligning the Annual
Performance Plan With the
Budget Account and Program
Activity Structure
(See figure in printed
edition.)
--------------------
\7 Addressing the Deficit: Budgetary Implications of Selected GAO
Work for Fiscal Year 1998 (GAO/OCG-97-2, Mar. 14, 1997).
\8 GAO/GGD-98-44, January 30, 1998.
\9 The term "program activity" refers to the listings of projects and
activities in the appendix portion of the Budget of the United States
Government. Program activity structures are intended to provide a
meaningful representation of the operations financed by a specific
budget account.
\10 Budget Account Structure: A Descriptive Overview
(GAO/AIMD-95-179, Sept. 18, 1995).
\11 Performance Budgeting: Past Initiatives Offer Insights for GPRA
Implementation (GAO/AIMD-97-46, Mar. 27, 1997).
ISSUE 3: RECOGNIZING
CROSSCUTTING EFFORTS
-------------------------------------------------------- Chapter I:1.3
How are agencies coordinating efforts with related strategic or
performance goals?
Congress can use annual performance plans to identify areas in which
agencies should be coordinating efforts to efficiently and
effectively meet national concerns. The Results Act implies that
federal programs contributing to the same or similar outcomes should
be closely coordinated to ensure that goals are consistent and that
program efforts are mutually reinforcing.
Our work has shown that as the federal government has responded over
time to new needs and problems, many agencies have been given
responsibilities for addressing the same or similar national
issues.\12 Some of this shared responsibility was intended to
recognize that addressing some issues from a national perspective
would necessarily involve more than one agency or more than one
approach. However, in many program areas, our work has suggested
that significant overlap and fragmentation existed. Such program
overlap and fragmentation can waste scarce funds, confuse and
frustrate program customers, and limit overall program effectiveness.
We found that in agencies' strategic plans, coordination of
crosscutting programs continued to be a challenge.\13 Agency annual
performance plans should be useful tools for identifying such
crosscutting efforts, ensuring needed coordination with agencies
performing similar activities, and defining performance goals that
maximize the effectiveness and efficiency of crosscutting activities.
Complementary and, where appropriate, common performance measures can
permit comparisons of related programs' results and the tools used to
achieve those results. Although individual agency performance plans
provide the building blocks for recognizing crosscutting efforts,
resolving the fragmentation and overlap in these efforts is likely to
be difficult and require more than one cycle of performance plans.
--------------------
\12 Managing For Results: Using the Results Act to Address Mission
Fragmentation and Program Overlap (GAO/AIMD-97-146, Aug. 29, 1997).
\13 GAO/GGD-98-44, January 30, 1998.
CORE QUESTION 2: STRATEGIES
AND RESOURCES
---------------------------------------------------------- Chapter I:2
How well does the performance plan discuss the strategies and
resources the agency will use to achieve its performance goals?
Without a clear description of the strategies and resources an agency
plans to use in the upcoming fiscal year, it will be difficult for
Congress to assess the likelihood of the agency's success in
achieving its intended results. By describing the strategies to be
used to achieve results and the resources to be applied to those
strategies, the performance plan can help Congress understand and
assess the relationship between the agency's resources and results.
The Results Act states that annual performance plans are to describe
briefly the operational processes, skills, and technology, as well as
the human, capital, information, or other resources required to meet
performance goals. To effectively implement the Act, agencies will
need to consider how they can best deploy these critical resources to
create a synergy that effectively and efficiently achieves
performance goals.
A performance plan that demonstrates that the agency has the
necessary strategies and resources to achieve its performance goals
addresses the following issues:
Issue 4: Connecting Strategies to Results
How clear and reasonable are the agency's strategies with respect to
its intended performance goals?
Issue 5: Connecting Resources to Strategies
What capital, human, financial, and other resources are being applied
to achieve the agency's performance goals?
ISSUE 4: CONNECTING
STRATEGIES TO RESULTS
-------------------------------------------------------- Chapter I:2.1
How clear and reasonable are the agency's strategies with respect to
its intended performance goals?
Congress can use an annual performance plan to gauge whether an
agency has clear strategies that logically relate to the achievement
of goals. Our recent reviews of strategic plans found that agencies
often lacked clear and fully developed strategies to achieve intended
results.\14 The strategic plans frequently did not elaborate on
specific actions the agency was taking or planned to take to carry
out its mission, outline planned accomplishments, and schedule their
implementation. By indicating a connection between their performance
goals and day-to-day activities, agencies' annual performance plans
provide an opportunity for agencies to better articulate how they
plan to accomplish their goals. A clear connection between
performance goals and program offices also helps both to reinforce
accountability and to ensure that, in their day-to-day activities,
managers keep in mind the results their organizations and programs
are striving to achieve.
The Results Act encourages agencies to rethink how they are
delivering their services. Our work has found that a different
approach or method of delivering government goods and services could
yield significant savings and improve performance.\15 Agencies may
consider a range of approaches, from process changes, such as
business process reengineering, to fundamental structural changes,
such as partnering, franchising, or privatization.\16 Also, the Act
permits agencies to request managerial flexibility waivers from
certain nonstatutory requirements in exchange for improved
performance resulting from the waivers. Examples of such flexibility
are the delegation of additional procurement authority to line
managers or the lifting of limitations on personnel compensation and
remuneration by central management agencies.\17
The Results Act requires agencies to identify key external factors in
their strategic plans that could significantly affect performance.
Agencies can build on the discussions in their strategic plans and
provide additional context in their annual performance plans. The
efforts of agencies often are but one factor among many that may
influence whether, and the degree to which, their programs achieve
their intended results. For example, we reported that regulatory
agencies have had difficulty sorting out the interaction that
external factors have had on the results that those agencies were
trying to achieve.\18 Agencies should consider whether external
factors, such as legislation or the actions of outside entities,
might affect program performance. In response, agencies can develop
strategies to leverage or mitigate these conditions.
--------------------
\14 GAO/GGD-98-44, January 30, 1998.
\15 GAO/OCG-97-2, March 14, 1997.
\16 Business Process Reengineering Assessment Guide
(GAO/AIMD-10.1.15, Apr. 1997); Privatization: Lessons Learned by
State and Local Governments (GAO/GGD-97-48, Mar. 14, 1997); and
Managing For Results: Critical Actions For Measuring Performance
(GAO/T-GGD-95-187, June 20, 1995).
\17 GPRA: Managerial Accountability and Flexibility Pilot Did Not
Work As Intended (GAO/GGD-97-36, Apr. 10, 1997).
\18 Managing for Results: Regulatory Agencies Identified Significant
Barriers to Focusing on Results (GAO/GGD-97-83, June 24, 1997).
ISSUE 5: CONNECTING
RESOURCES TO STRATEGIES
-------------------------------------------------------- Chapter I:2.2
What capital, human, financial, and other resources are being applied
to achieve the agency's performance goals?
Annual performance plans can show Congress how agencies plan to
manage cost-effective mixtures of critical resources in pursuit of
performance goals. We have found that management problems have
needlessly cost billions of dollars in taxpayer funds and resulted in
missed opportunities to achieve better and less costly
performance.\19 For example, numerous poorly managed information
technology systems have produced multimillion-dollar cost overruns,
schedule slippages measured in years, and dismal results.
Similarly, because federal financial management has suffered from
decades of neglect and failed attempts to modernize outdated
financial systems, good information on the full costs of federal
operations is frequently absent or extremely difficult to
reconstruct. Full cost information is critical for making sound
analyses of how to pursue performance goals. Our work has found that
agencies have failed to accurately estimate the full actual and
future costs of multimillion-dollar programs and initiatives.\20
Estimating full costs was sometimes difficult because accounting
systems could not disaggregate costs and assign them to particular
programs or initiatives. Without reliable data on costs,
decisionmakers cannot determine whether a strategy is realistic and
cost effective.
Complementing the Results Act, a framework of reform legislation--the
Chief Financial Officers (CFO) Act and information technology reform
legislation, including the Paperwork Reduction Act of 1995 and the
Clinger-Cohen Act of 1996--is now in place to address these
long-standing problems. The CFO Act set expectations for the
development of better performance and cost measures and requires that
agencies prepare and annually revise plans to implement OMB's 5 year
financial management plan. Building on the CFO Act, the Federal
Financial Management Improvement Act requires agencies to comply with
federal accounting standards--including a requirement to develop and
implement cost accounting systems that can be used to relate the full
costs of various programs and activities to performance outputs by
fiscal year 1998. Information technology reform legislation is
directed at more effective management and use of information
technology to better support agencies' missions and improve program
performance.
To be most useful to congressional decisionmakers, plans required
under these initiatives, such as information technology, capital
assets, and financial management plans, should be linked to agencies'
annual performance plans. Similarly, significant initiatives
involving information, capital, and financial management resources
should be described in the annual performance plan and referenced
back to the agency's more detailed plans in these areas. Also, the
Results Act recognizes the importance of human resource management.
Agencies' performance plans that contain a description of how
workforce knowledge, skills, and abilities can contribute to the
achievement of program performance goals would be most useful to
congressional decisionmakers.
--------------------
\19 High-Risk Areas: Actions Needed to Solve Pressing Management
Problems (GAO/T-AIMD/GGD-97-60, Mar. 5, 1997).
\20 High-Risk Series: An Overview (GAO/HR-97-1, Feb. 1997).
CORE QUESTION 3: VERIFICATION
AND VALIDATION
---------------------------------------------------------- Chapter I:3
To what extent does the agency's performance plan provide confidence
that its performance information will be credible?
In passing the Results Act, Congress emphasized that the usefulness
of agencies' performance data for its decisionmaking ultimately
depends on the degree of confidence that Congress has in that data.
For Congress to know whether the intended performance has truly
occurred, agencies should produce performance data that is verified
and valid.
The Results Act requires that agencies describe in their annual
performance plans how they intend to verify and validate performance
data. The procedures should be credible and specific to ensure that
performance information is sufficiently complete, accurate, and
consistent to document performance and support decisionmaking.
However, some agencies have recognized that although such information
is indispensable to decisionmaking, collecting the data can be
difficult and costly. For this reason, they have explained the
limitations of their performance information and recognized the
implications of the limitations for assessing the achievement of
their goals.
A performance plan that fully describes whether an agency has
procedures to obtain credible performance data addresses the
following issues:
Issue 6: Verifying and Validating Performance
How will the agency ensure that its performance information is
sufficiently complete, accurate, and consistent?
Issue 7: Recognizing Data Limitations
To what extent does the performance plan identify significant data
limitations and their implications for assessing the achievement of
performance goals?
ISSUE 6: VERIFYING AND
VALIDATING PERFORMANCE
-------------------------------------------------------- Chapter I:3.1
How will the agency ensure that its performance information is
sufficiently complete, accurate, and consistent?
Congress needs sufficiently complete, accurate, and consistent
performance information to guide its decisions on how best to
allocate scarce resources and oversee federal programs. To assist
congressional decisionmakers, the Results Act requires agencies to
discuss the means that they will use to validate and verify
performance information. This discussion in the performance plans
can provide important contextual information for Congress. Adequate
and reliable performance data are critical for decisionmaking, but
the verification and validation of such data can be costly and
difficult and require agencies to consider the most cost-effective
means.
Both verification and validation of performance data are important
for reducing the risk of producing inaccurate data. Validation is an
effort to ensure that data are free of systematic error or bias and
that what is intended to be measured is actually measured. For
example, performance information about a program's ability to
stimulate the growth of small businesses could overestimate the
program's success if data were not collected from businesses that
participated but subsequently failed. Verification is a process of
checking or testing performance information to assess other types of
errors, such as errors in keying data.
The Results Act gives agencies discretion in determining how
verification and validation are performed. However, the CFO Act has
already established one procedure that can help improve the
credibility of agencies' financial data--the preparation and audit of
agency financial statements. The discipline and rigor of financial
statement audits have caused significant improvements in the state of
federal financial management, but much more is needed. Poor
recordkeeping and lack of controls often found in these audits have
resulted in inaccurate financial and program cost information both
being used to manage programs and provided to Congress for making
decisions. Therefore, it is important that agencies continue to make
progress toward the establishment of financial systems that can
generate the reliable data needed to assess financial performance.
Similarly, agencies should demonstrate that they have adequate
quality control systems to ensure the credibility of nonfinancial
data.
ISSUE 7: RECOGNIZING DATA
LIMITATIONS
-------------------------------------------------------- Chapter I:3.2
To what extent does the performance plan identify significant data
limitations and their implications for assessing the achievement of
performance goals?
Explaining the limitations of performance information can provide
Congress with a context for understanding and assessing agencies'
performance and the costs and challenges agencies' face in gathering,
processing, and analyzing needed data. This discussion on data
limitations can help identify the actions needed to improve the
agency's ability to measure its performance.
Agencies face many challenges in collecting credible data. For
example, performance data may not be complete because they are not
available from all reporting sites or may not be consistent because
multiple parties, who are providing the data, may use different
definitions of the reported information. Responding to these
concerns will require agencies to balance the cost of improved data
collection against the benefit of higher quality data to document
performance. Lastly, because agencies have frequently lacked a
results orientation and because of the costs and challenges of
collecting performance information, baseline and/or historical trend
data needed to define performance goals may be poor or nonexistent.
KEY ASSESSMENT QUESTIONS
=========================================================== Chapter II
ISSUE 1: DEFINING EXPECTED
PERFORMANCE
--------------------------------------------------------- Chapter II:1
To what extent do the annual performance goals and measures provide a
succinct and concrete statement of expected performance for
subsequent comparison with actual performance?
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
ISSUE 2: CONNECTING MISSION,
GOALS, AND ACTIVITIES
--------------------------------------------------------- Chapter II:2
How are the agency's annual performance goals linked to the agency's
mission, strategic goals, and program activities in its budget
request?
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
ISSUE 3: RECOGNIZING
CROSSCUTTING EFFORTS
--------------------------------------------------------- Chapter II:3
How are agencies coordinating efforts with related strategic or
performance goals?
(See figure in printed
edition.)
(See figure in printed
edition.)
ISSUE 4: CONNECTING STRATEGIES
TO RESULTS
--------------------------------------------------------- Chapter II:4
How clear and reasonable are the agency's strategies with respect to
its intended performance goals?
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
ISSUE 5: CONNECTING RESOURCES
TO STRATEGIES
--------------------------------------------------------- Chapter II:5
What capital, human, financial, and other resources are being applied
to achieve the agency's performance goals?
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
ISSUE 6: VERIFYING AND
VALIDATING PERFORMANCE
--------------------------------------------------------- Chapter II:6
How will the agency ensure that its performance information is
sufficiently complete, accurate, and consistent?
(See figure in printed
edition.)
(See figure in printed
edition.)
ISSUE 7: RECOGNIZING DATA
LIMITATIONS
--------------------------------------------------------- Chapter II:7
To what extent does the performance plan identify significant data
limitations and their implications for assessing the achievement of
performance goals?
(See figure in printed
edition.)
(See figure in printed
edition.)
THE RESULTS ACT'S PLANNING AND
REPORTING REQUIREMENTS
==================================================== Appendix Appendix
The Results Act is designed to improve the efficiency and
effectiveness of federal programs by establishing a system to set
goals for program performance and to measure results. Specifically,
the Act requires executive agencies to prepare multiyear strategic
plans, annual performance plans, and annual performance reports.
MULTIYEAR STRATEGIC PLANS
-------------------------------------------------- Appendix Appendix:1
The Results Act requires virtually every executive agency to develop
strategic plans covering a period of at least 5 years forward from
the fiscal year in which it is submitted and to update those plans at
least every 3 years. Agencies' first strategic plans were to be
submitted to Congress and the Director of OMB by September 30, 1997.
The strategic plans are to (1) include the agencies' mission
statements; (2) identify long-term general goals and objectives; (3)
describe how the agencies intend to achieve those goals through their
activities and through their human, capital, information, and other
resources; and (4) explain the key external factors that could
significantly affect the achievement of those goals. Under the Act,
strategic plans are the starting point for agencies to set annual
performance goals and to measure program performance in achieving
those goals. Consequently, strategic plans are also to include a
description of how long-term general goals will be related to annual
performance goals as well as a description of the program evaluations
that agencies used to establish their long-term general goals and a
schedule for subsequent evaluations. As part of the strategic
planning process, agencies are required to consult with Congress and
solicit the views of other stakeholders--those governmental and
nongovernmental entities potentially affected by, or interested in,
the agencies' activities.
ANNUAL PERFORMANCE PLANS
-------------------------------------------------- Appendix Appendix:2
Building on the decisions made as part of the strategic planning
process, the Results Act requires executive agencies to develop
annual performance plans covering each program activity set forth in
the agencies' budgets. The first annual performance plans, covering
fiscal year 1999, were to be submitted to OMB in the fall of 1997 and
to Congress after the President's budget in 1998. The Results Act
requires that each agency prepare an annual performance plan that
shall:
"(1) establish performance goals to define the level of performance
to be achieved by a program activity;
"(2) express such goals in an objective, quantifiable, and measurable
form unless authorized to be in an alternative form. . . ;
"(3) briefly describe the operational processes, skills and
technology, and the human, capital, information, or other resources
required to meet the performance goals;
"(4) establish performance indicators to be used in measuring or
assessing the relevant outputs, service levels, and outcomes of each
program activity;
"(5) provide a basis for comparing actual program results with the
established performance goals; and
"(6) describe the means to be used to verify and validate measured
values."\1
The Act authorizes agencies to apply for managerial flexibility
waivers in their annual performance plans. Agencies' authority to
request waivers of nonstatutory administrative procedural
requirements and controls is intended to provide federal managers
with more flexibility to structure agency systems to better support
performance goals. An example of increased flexibility would be to
allow an organization to recapture unspent operating funds because of
increased efficiencies and then to use these funds to purchase new
equipment or expand employee training. Another example might involve
delegating more authority to line managers to make procurement
decisions.
OMB is to use the performance plans that agencies submit to develop
an overall federal government performance plan. OMB is to submit
this governmentwide plan each year to Congress with the president's
budget. According to the Senate committee report accompanying the
Act, the overall federal government performance plan is to present to
Congress a single, cohesive picture of the federal government's
annual performance goals for the fiscal year.\2 The first overall
plan was due with the president's fiscal year 1999 budget.
--------------------
\1 32 U.S.C. 1115.
\2 S. Rep. No. 58, 103d Cong. 1st Sess. (1993).
ANNUAL PERFORMANCE REPORTS
-------------------------------------------------- Appendix Appendix:3
Finally, the Results Act requires each executive agency to prepare
annual reports on program performance for the previous fiscal year.
The first performance reports for fiscal year 1999 are due to
Congress and the president no later than March 31, 2000; subsequent
reports are due by March 31 for the years that follow. In each
report, an agency is to review and discuss its performance compared
with the performance goals it established in its annual performance
plan. When a goal is not met, the agency is to explain in the report
the reasons the goal was not met; plans and schedules for meeting the
goal; and, if the goal was impractical or not feasible, the reasons
for that and the actions recommended. According to the Senate
committee report on the Act, actions needed to accomplish a goal
could include legislative, regulatory, or other actions. If an
agency finds a goal to be impractical or not feasible, it is to
include a discussion of whether the goal should be modified.
In addition to evaluating the progress made toward achieving its
annual goals, an agency's program performance report is to evaluate
the agency's performance plan for the fiscal year in which the
performance report was submitted. Thus, in their fiscal year 1999
performance reports that are due by March 31, 2000, agencies are
required to evaluate their performance plans for fiscal year 2000 on
the basis of their reported performance in fiscal year 1999. This
evaluation is to help show how an agency's actual performance is
influencing its performance plan. The report also is to include (1)
the summary findings of program evaluations completed during the
fiscal year covered by the report and (2) the use and effectiveness
of any of the Results Act managerial flexibility waivers that an
agency received.
Agencies also are to include baseline and trend data in annual
performance reports to help ensure that their reports are complete
and that performance is viewed in context. Such data can show
whether performance goals are realistic given the past performance of
an agency. Such data can also assist users of reports to draw more
informed conclusions than they would if they compared only a single
year's performance against an annual goal, because users of reports
can see improvements or declines in an agency's performance over
prior years.\3 For fiscal years 2000 and 2001, agencies' reports are
to include data on the extent to which their performance achieved
their goals, beginning with fiscal year 1999. For each subsequent
year, agencies are to include performance data for the year covered
by the report and 3 prior years. Congress recognized that in some
cases not all the performance data will be available in time for the
required reporting date. In such cases, agencies are to provide
whatever data are available with a notation as to their incomplete
status. Subsequent annual performance reports are to include the
complete data as part of the trend information.
--------------------
\3 GAO/GGD-96-66R, February 14, 1996.
RELATED GUIDANCE ON ANNUAL
PERFORMANCE PLANS
============================================================ Chapter 2
This appendix contains a compilation of related guidance, including
the Results Act, GAO reports, and OMB documents, and is arranged by
the issues discussed in this guide.
ISSUE 1: DEFINING
EXPECTED PERFORMANCE
------------------------------------------------------ Chapter 2:0.0.1
The Government Performance and Results Act (Results Act), 31 U.S.C.
1115(a)(1), 1115(a)(2), 1115(a)(4), 1115(a)(5), 1115(b), and 1115(c).
Senate Committee on Governmental Affairs Report accompanying the
Results Act (Senate Report 103-58, June 16, 1993), pp. 15-16,
"Performance Plans"; p. 29, "Performance Goals"; pp. 29-30,
"Performance Indicators"; and p. 30, "Alternative Forms of
Measurement."
OMB Circular A-11, secs. 220.1, 220.4, 220.10(a), 220.10(b),
220.10(c), 220.14, 220.16, 220.17, 221.4(a), 221.4(b), and 221.4(d).
OMB Checklist for Agency Annual Performance Plans (Nov. 24, 1997),
pp.1-2, "Coverage of Program Activities"; pp. 3-4, "Annual
Performance Goals"; p. 4, "Performance Indicators"; and p. 5,
"Alternative Form of Measurement."
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997), pp.
55-57, 61-63, and 71-72.
Executive Guide: Effectively Implementing the Government Performance
and Results Act (GAO/GGD-96-118, June 1996), pp. 24-26.
Implementation of the Government Performance and Results Act (GPRA),
A Report on the Chief Financial Officer's Role and Other Issues
Critical to the Governmentwide Success of GPRA, Chief Financial
Officers Council, GPRA Implementation Committee, May 1995.
ISSUE 2: CONNECTING
MISSION, GOALS, AND
ACTIVITIES
------------------------------------------------------ Chapter 2:0.0.2
Results Act, 5 U.S.C. 306(c), 31 U.S.C. 1115(a), and 31 U.S.C.
1115(c).
Senate Committee on Governmental Affairs Report accompanying the
Results Act (Senate Report 103-58, June 16, 1993), p. 15,
"Performance Plans"; p. 29, "Performance Goals"; and p. 31,
"Coverage of Program Activities."
OMB Circular A-11, secs. 210.2(c), 210.4, 220.3, 220.4, 220.5,
220.6, 220.7, 220.8, 220.9(a), 220.9(b), 220.9(d), 220.9(e),
220.10(c), 221.3, 221.4(b).
OMB Checklist for Agency Annual Performance Plans (Nov. 24, 1997),
pp.1-2, "Coverage of Program Activities"; pp. 3-4, "Annual
Performance Goals"; p. 7, "Mission Statement and General Goals and
Objectives"; and p. 8, "Budget Account Restructuring."
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997), pp.
90-93.
Performance Budgeting: Past Initiatives Offer Insights for GPRA
Implementation (GAO/AIMD-97-46, Mar. 27, 1997).
Integrating Performance Measurement into the Budget Process, Chief
Financial Officers Council, GPRA Implementation Committee
Subcommittee Project, September 22, 1997.
ISSUE 3: RECOGNIZING
CROSSCUTTING EFFORTS
------------------------------------------------------ Chapter 2:0.0.3
OMB Circular A-11, secs. 220.8, 220.10(b), and 221.4(c).
OMB Checklist for Agency Annual Performance Plans (Nov. 24, 1997),
p. 8, "Cross-cutting Programs."
Managing for Results: Using the Results Act to Address Mission
Fragmentation and Program Overlap (GAO/AIMD-97-146, Aug. 29, 1997).
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997), pp.
53-55.
ISSUE 4: CONNECTING
STRATEGIES TO RESULTS
------------------------------------------------------ Chapter 2:0.0.4
Results Act, 31 U.S.C. 1115(a)(3) and 31 U.S.C. 9703.
Senate Committee on Governmental Affairs Report accompanying Results
Act (Senate Report 103-58, June 16, 1993), pp. 15-16, "Performance
Plans"; pp. 17-18, "Managerial Flexibility Waivers"; and pp. 34-36,
"Section 5 - Managerial Accountability and Flexibility."
OMB Circular A-11, secs. 220.10(b), 220.12(a), 220.12(b), 220.12(c),
and 221.4(b).
OMB Checklist for Agency Annual Performance Plans (Nov. 24, 1997),
p. 6, "Means and Strategies"; p. 8, "Tax Expenditures and
Regulation"; and p. 8, "External Factors."
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997), pp.
63-66.
Business Process Reengineering Assessment Guide, Version 3
(GAO/AIMD-10.1.15, Apr. 1997).
Privatization: Lessons Learned by State and Local Governments
(GAO/GGD-97-48, Mar. 14, 1997).
Executive Guide: Effectively Implementing the Government Performance
and Results Act (GAO/GGD-96-118, June 1996), pp. 18-21 and 24-26.
ISSUE 5: CONNECTING
RESOURCES TO STRATEGIES
------------------------------------------------------ Chapter 2:0.0.5
Results Act, 31 U.S.C. 1115(a)(3).
Senate Committee on Governmental Affairs Report accompanying Results
Act (Senate Report 103-58, June 16, 1993), pp. 15-16, "Performance
Plans"; and p. 30, "Performance Indicators."
OMB Circular A-11, secs. 220.1, 220.9(a), 220.9(e), 220.10(c),
220.11(a), 220.11(b), 220.11(c), 220.12(a), 220.12(d), and Part 3.
OMB Checklist for Agency Annual Performance Plans (Nov. 24, 1997),
p. 5, "Future Year Performance"; p. 5, "Performance Goals Funded By
Prior Year Appropriations"; and p. 6, "Means and Strategies."
OMB Capital Programming Guide, v. 1.0 (July 1997).
Executive Guide: Measuring Performance and Demonstrating Results of
Information Technology Investments (GAO/AIMD-97-163, Sept. 1997).
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997), pp.
90-97.
Year 2000 Computing Crisis: An Assessment Guide (GAO/AIMD-10.1.14,
Sept. 1997).
Assessing Risks and Returns: A Guide for Evaluating Federal
Agencies' IT Investment Decision-making, Version 1 (GAO/AIMD-10.1.13,
Feb. 1997).
Information Technology Investment: Agencies Can Improve Performance,
Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, Sept. 30, 1996).
Executive Guide: Effectively Implementing the Government Performance
and Results Act (GAO/GGD-96-118, June 1996), pp. 18-21 and 39-46.
Transforming the Civil Service: Building the Workforce of the
Future--Results of a GAO-Sponsored Symposium (GAO/GGD-96-35, Dec.
26, 1995).
Federal Accounting Standards Board (FASAB) Volume 1 Original
Statements: Statements of Federal Financial Accounting Concepts and
Standards, Statement of Federal Financial Accounting Standards No.
1, Objectives of Federal Financial Reporting (GAO/AIMD-21.1.1, Mar.
1997), pp. 11-62.
FASAB Volume 1 Original Statements: Statements of Federal Financial
Accounting Concepts and Standards, Statement of Federal Financial
Accounting Standards No. 4, Managerial Cost Accounting Standards
(GAO/AIMD-21.1.1, Mar. 1997), pp. 331-394.
ISSUE 6: VERIFYING AND
VALIDATING PERFORMANCE
------------------------------------------------------ Chapter 2:0.0.6
Results Act 31, U.S.C. 1115 (a)(6).
Senate Committee on Governmental Affairs Report accompanying Results
Act (Senate Report 103-58, June 16, 1993), p. 30, "Verification and
Validation."
OMB Circular A-11, secs. 220.7, 220.13, and 221.5.
OMB Checklist for Agency Annual Performance Plans (Nov. 24, 1997),
p. 7, "Verification and Validation."
Executive Guide: Effectively Implementing the Government Performance
and Results Act (GAO/GGD-96-118, June 1996), pp. 27-29.
GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996) pp. 6-8
and 11.
FASAB Volume 1 Original Statements: Statements of Federal Financial
Accounting Concepts and Standards (GAO/AIMD-21.1.1, Mar. 1997).
Budget and Financial Management: Progress and Agenda for the Future
(GAO/T-AIMD-96-80, Apr. 23, 1996).
Executive Guide: Information Security Management (GAO/AIMD-98-21,
Nov. 1997).
ISSUE 7: RECOGNIZING
DATA LIMITATIONS
------------------------------------------------------ Chapter 2:0.0.7
OMB Circular A-11, sec. 221.5.
OMB Checklist for Agency Annual Performance Plans (Nov. 24, 1997),
p. 7, "Verification and Validation."
Managing for Results: Regulatory Agencies Identified Significant
Barriers to Focusing on Results (GAO/GGD-97-83, June 24, 1997).
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997), pp.
61-75.
Managing for Results: Analytic Challenges in Measuring Performance
(GAO/HEHS/GGD-97-138, May 30, 1997).
Measuring Performance: Strengths and Limitations of Research
Indicators (GAO/RCED-97-91, Mar. 21, 1997).
Executive Guide: Effectively Implementing the Government Performance
and Results Act (GAO/GGD-96-118, June 1996), pp. 27-29.
GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996).
Block Grants: Issues in Designing Accountability Provisions
(GAO/AIMD-95-226, Sept. 1, 1995).
*** End of document. ***