Results-Oriented Government: GPRA Has Established a Solid
Foundation for Achieving Greater Results (10-MAR-04, GAO-04-38).
Now that the Government Performance and Results Act (GPRA) has
been in effect for 10 years, GAO was asked to address (1) the
effect of GPRA in creating a governmentwide focus on results and
the government's ability to deliver results to the American
public, (2) the challenges agencies face in measuring performance
and using performance information in management decisions, and
(3) how the federal government can continue to shift toward a
more results-oriented focus.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-38
ACCNO: A09453
TITLE: Results-Oriented Government: GPRA Has Established a Solid
Foundation for Achieving Greater Results
DATE: 03/10/2004
SUBJECT: Agency missions
Federal agencies
Performance measures
Productivity in government
Reporting requirements
Strategic planning
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-38
United States General Accounting Office
GAO
Report to Congressional Requesters
March 2004
RESULTS-ORIENTED GOVERNMENT
GPRA Has Established a Solid Foundation for Achieving Greater Results
a
GAO-04-38
Highlights of GAO-04-38, a report to congressional requesters
Now that the Government Performance and Results Act (GPRA) has been in
effect for 10 years, GAO was asked to address (1) the effect of GPRA in
creating a governmentwide focus on results and the government's ability to
deliver results to the American public, (2) the challenges agencies face
in measuring performance and using performance information in management
decisions, and (3) how the federal government can continue to shift toward
a more results-oriented focus.
GAO recommends that the Office of Management and Budget (OMB) improve its
guidance and oversight of GPRA implementation, as well as develop a
governmentwide performance plan. GAO also believes Congress should
consider amending GPRA to require that (1) agencies update their strategic
plans at least once every four years, consult with congressional
stakeholders at least once every new Congress, and make interim updates to
strategic and performance plans as appropriate; and (2) the President
develop a governmentwide strategic plan. OMB generally agreed with our
recommendations, but stated that the President's Budget can serve as both
a governmentwide strategic and annual plan. However, we believe the budget
provides neither a long-term nor an integrated perspective on the federal
government's performance.
March 2004
RESULTS-ORIENTED GOVERNMENT
GPRA Has Established a Solid Foundation for Achieving Greater Results
GPRA's requirements have established a solid foundation of
results-oriented performance planning, measurement, and reporting in the
federal government. Federal managers surveyed by GAO reported having
significantly more of the types of performance measures called for by GPRA
(see figure below). GPRA has also begun to facilitate the linking of
resources to results, although much remains to be done in this area to
increase the use of performance information to make decisions about
resources. We also found agency strategic and annual performance plans and
reports we reviewed have improved over initial efforts.
Although a foundation has been established, numerous significant
challenges to GPRA implementation still exist. Inconsistent top leadership
commitment to achieving results within agencies and OMB can hinder the
development of results-oriented cultures in agencies. Furthermore, in
certain areas, federal managers continue to have difficulty setting
outcome-oriented goals, collecting useful data on results, and linking
institutional, program, unit, and individual performance measurement and
reward systems. Finally, there is an inadequate focus on addressing issues
that cut across federal agencies.
OMB, as the focal point for management in the federal government, is
responsible for overall leadership and direction in addressing these
challenges. OMB has clearly placed greater emphasis on management issues
during the past several years. However, it has showed less commitment to
GPRA implementation in its guidance to agencies and in using the
governmentwide performance plan requirement of GPRA to develop an
integrated approach to crosscutting issues. In our view, governmentwide
strategic planning could better facilitate the integration of federal
activities to achieve national goals.
Percentage of Federal Managers Who Reported Having Specific Types of
Performance Measures Called for by GPRA
www.gao.gov/cgi-bin/getrpt?GAO-04-38.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Patricia A. Dalton at (202)
512-6806 or [email protected].
Contents
Transmittal Letter 1
Executive Summary Purpose 3 3
Background 4
Results in Brief 6
Principal Findings 10
Recommendations for Executive Action 19
Matters for Congressional Consideration 20
Agency Comments 20
Chapter 1 Introduction Impact of Emerging Trends and Fiscal
Challenges GPRA Background 22 23 25
Scope and Methodology 30
Chapter 2 31 GPRA Established the GPRA Statutory Requirements Laid a
Foundation for Agencywide Results-Oriented Management 31
Foundation for a More Results-Oriented Federal Government
Chapter 3
Agencies Have
Addressed Many
Critical Performance
Planning and Reporting
Challenges, but
Weaknesses Persist
46 Quality of Selected Strategic Plans Reflects Improvements over Initial
Drafts 46 Fiscal Year 2004 Annual Performance Plans Addressed Some
Weaknesses of Earlier Plans, but Still Have Room for Significant
Improvement 53 Strengths and Weaknesses of Selected Agencies' Fiscal Year
2002
Annual Performance Reports 60
Chapter 4 68 Challenges to GPRA Top Leadership Does Not Consistently Show
Commitment to
Achieving Results 69
Implementation Persist
Contents
Managers Report Mixed Results in Use of Performance Information 75
Managers Continue to Confront a Range of Human Capital Management
Challenges 79 Persistent Challenges in Setting Outcome-Oriented Goals,
Measuring Performance, and Collecting Useful Data 88 Crosscutting Issues
Hinder Successful GPRA Implementation 92 Managers View Congress' Use of
Performance Information as
Limited 96
Chapter 5 Conclusions and Agenda for Achieving a Sustainable, 100 100
Governmentwide Focus on Results
Recommendations Recommendations for Executive Action 109
Matters for Congressional Consideration 110
Agency Comments 110
Appendixes
Appendix I:
Appendix II:
Appendix III:
Objectives, Scope, and Methodology 113 Methodology for Governmentwide
Survey 113 Methodology for Focus Groups 116 Methodology for Interviews
with Political Appointees 118 Methodology for Selecting Agencies to Review
for Changes in the
Quality of Their Strategic Plans, Annual Performance Plans, and
Annual Performance Reports 119
Focus Group Participants Agreed GPRA Provides a
Framework for Federal Agencies to Become More Results
Oriented 121
GPRA Accomplishments 122
Views on Delivering Results to the American Public Were
Mixed 124 Alternate Views on GPRA's Effect 125 Challenges in Implementing
and Overseeing GPRA Activities 126 Suggestions to Address GPRA Challenges
127
Observations on Agencies' Strategic Plans 130 Required Elements of Agency
Strategic Plans 130 Observations on Changes in the Quality of Education's
Strategic
Plan 134 Observations on Changes in the Quality of DOE's Strategic Plan
139
Contents
Observations on Changes in the Quality of HUD's Strategic Plan
Observations on Changes in the Quality of SBA's Strategic Plan
Observations on Changes in the Quality of SSA's Strategic Plan
Observations on Changes in the Quality of DOT's Strategic Plan
Appendix IV: Observations on Agencies' Annual Performance Plans
Key Elements of Information for Annual Performance Plans Observations on
Changes in the Quality of Education's Annual Performance Plan Observations
on Changes in the Quality of DOE's Annual Performance Plan Observations on
Changes in the Quality of HUD's Annual Performance Plan Observations on
Changes in the Quality of SBA's Annual Performance Plan Observations on
Changes in the Quality of SSA's Annual Performance Plan Observations on
Changes in the Quality of DOT's Annual Performance Plan
Appendix V: Observations on Agencies' Annual Performance and
Accountability Reports
Observations on the Quality of Education's Fiscal Year 2002 Performance
and Accountability Report Observations on the Quality of DOE's Fiscal Year
2002 Annual Performance and Accountability Report Observations on the
Quality of HUD's Fiscal Year 2002 Annual Performance and Accountability
Report Observations on the Quality of SBA's Fiscal Year 2002 Annual
Performance and Accountability Report Observations on the Quality of SSA's
Fiscal Year 2002 Annual Performance and Accountability Report Observations
on the Quality of DOT's Fiscal Year 2002 Annual Performance and
Accountability Report
Appendix VI: GAO Federal Managers' Survey Data Appendix VII: Agencies
Subject to the Chief Financial Officers Act Appendix VIII: Comments from
the Office of Management and Budget Appendix IX: Comments from the
Department of Energy
GAO Comments
142 146 150 154
160 160
164
169
172
175
179
183
188
192
200
204
208
213
217
223
236
238
240 242
Contents
Appendix X: Comments from the Department of Housing and Urban Development
245 GAO Comments 251
Appendix XI: Comments from the Social Security Administration 255 GAO
Comments 260
Appendix XII: GAO Contact and Staff Acknowledgments 263 GAO Contact 263
Acknowledgments 263
Related GAO Products 264
GPRA/Managing for Results 264
Strategic Human Capital Management 266
Linking Resources to Results 266
Measuring Performance 267
Data Credibility 268
Using Performance Information 268
Tables Table 1: Table 2: Table 3: Table 4:
Table 5: Table 6: Table 7: Table 8: Table 9:
Agencies' Progress in Addressing Required Elements of
Strategic Planning under GPRA
Characterizations of Agencies' Fiscal Year 1999 and 2004
Annual Performance Plans
Characterizations of Agencies' 2002 Annual Performance
Reports
Summary of Characteristics of Agencies Selected for
Review of Strategic Plans, Annual Performance Plans, and
Annual Performance Reports
Agencies' Progress in Addressing Required Elements of
Strategic Planning under GPRA
Education's Progress in Addressing Required Elements of
Strategic Planning under GPRA
DOE's Progress in Addressing Required Elements of
Strategic Planning under GPRA
HUD's Progress in Addressing Required Elements of
Strategic Planning under GPRA
SBA's Progress in Addressing Required Elements of
Strategic Planning under GPRA
48 54 61
120 133 134 139 143 146 150 154
Table 10: SSA's Progress in Addressing Required Elements of Strategic
Planning under GPRA
Table 11: DOT's Progress in Addressing Required Elements of Strategic
Planning under GPRA
Contents
Table 12: Characterizations of Agencies' Annual Performance Plans 163
Table 13: Characterizations of Agencies' Fiscal Year 2002 Annual
Performance and Accountability Reports 192
Figures Figure 1:
Figure 2:
Figure 3:
Figure 4: Figure 5:
Figure 6: Figure 7: Figure 8: Figure 9:
Composition of Spending as a Share of GDP Assuming
Discretionary Spending Grows with GDP after 2003 and
All Expiring Tax Provisions Are Extended
Percentage of Federal Managers Who Reported That
There Were Performance Measures for the Programs with
Which They Were Involved
Percentage of Federal Managers Who Reported Having
Specific Types of Performance Measures to a Great or
Very Great Extent
Percentage of Federal Managers Who Reported Their
Awareness of GPRA
Percentage of Federal Managers Who Reported
Hindrances to Measuring Performance or Using the
Performance Information to a Great or Very Great
Extent
Percentage of Federal Managers and SES Managers Who
Reported That OMB Paid Attention to Their Agency's
Efforts under GPRA to a Great or Very Great Extent
Percentage of Federal Managers Who Reported They
Considered Strategic Goals to a Great or Very Great
Extent When Allocating Resources
Percentage of Federal Managers Who Reported They
Considered Performance Information to a Great or Very
Great Extent When Allocating Resources
Percentage of Federal Managers Who Reported That
Funding Decisions Were Based on Results or
Outcome-Oriented Performance Information to a Great
or Very Great Extent
24
35
36 37
38 39 43 44
45 70
71
Figure 10: Percentage of Federal Managers Who Reported to a Great or Very
Great Extent Their Top Leadership Has a Strong Commitment to Achieving
Results
Figure 11: Percentage of SES and Non-SES Managers Who Reported to a Great
or Very Great Extent Their Agency Top Leadership Demonstrated Strong
Commitment to Achieving Results
Contents
Figure 12: Percentage of Federal Managers Who Reported Using Information
Obtained from Performance Measurement to a Great or Very Great Extent for
Various Management Activities 77
Figure 13: Percentage of Federal Managers Responding "Yes" about Being
Involved in the Following Activities 79
Figure 14: Percentage of Federal Managers Reporting to a Great or Very
Great Extent That Managers/Supervisors at Their Levels Had the
Decision-Making Authority They Needed to Help the Agency Accomplish Its
Strategic Goals 81
Figure 15: Percentage of Federal Managers, SES, and Non-SES in 2003
Reporting to a Great or Very Great Extent That They Were Held Accountable
for the Accomplishment of Agency Strategic Goals 82
Figure 16: Percentage of Federal Managers in Each Survey Year Who Reported
That during the Past 3 Years Their Agencies Provided, Arranged, or Paid
for Training That Would Help Them Accomplish Specific Tasks 84
Figure 17: Percentage of Federal Managers Who Reported to a Great or Very
Great Extent That Employees in Their Agencies Received Positive
Recognition for Helping Their Agencies Accomplish Their Strategic Goals 86
Figure 18: Percentage of Federal Managers Reporting to a Great or Very
Great Extent That a Lack of Ongoing Congressional Commitment or Support
for Using Performance Information in Making Program/Funding Decisions Is a
Hindrance 98
Figure 19: Summary of Education's Performance Indicators for Fiscal Year
2002 194
Figure 20: Inputs: Allocating Funds for Education's Objective to Ensure
That All Students Read on Grade Level by the Third Grade 198
Figure 21: Summary of DOE's Performance Indicators for Fiscal Year 2002
201 Figure 22: Summary of HUD's Performance Indicators for Fiscal Year
2002 206 Figure 23: Summary of SBA's Performance Goals for Fiscal Year
2002 209 Figure 24: Summary of SSA's Performance Goals for Fiscal Year
2002 214 Figure 25: Summary of DOT's Performance Indicators for Fiscal
Year 2002 218
Contents
Abbreviations
AP advanced placement
CDBG Community Development Block Grants
CFO Chief Financial Officer
CRS Congressional Research Service
DOE Department of Energy
DOT Department of Transportation
EPA Environmental Protection Agency
FAA Federal Aviation Administration
FSA Federal Student Assistance
FTE full-time employee
GM general management
GPRA Government Performance and Results Act of 1993
GS general schedule
HHS Department of Health and Human Services
HOME Home Investment Partnership Program
HUD Department of Housing and Urban Development
ICH Interagency Council on the Homeless
IG Inspector General
IT information technology
IRS Internal Revenue Service
JARC Job Access and Reverse Commute
NAEP National Assessment for Educational Progress
NASA National Aeronautics and Space Administration
OASI Old Age and Survivors Insurance
OMB Office of Management and Budget
OPM Office of Personnel Management
PART Program Assessment Rating Tool
PMA President's Management Agenda
SBA Small Business Administration
SBDC Small Business Development Centers
SES Senior Executive Service
SSA Social Security Administration
SSI Supplemental Security Income
SSN Social Security number
VA Department of Veterans Affairs
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
Comptroller General of the United States
United States General Accounting Office Washington, D.C. 20548
March 10, 2004
Congressional Requesters
As you requested, we have assessed the effectiveness of the Government
Performance and Results Act (GPRA), in light of its 10-year anniversary in
2003. Our review focused on GPRA's accomplishments, challenges to its
continued implementation, and an agenda for achieving a sustainable,
governmentwide focus on results.
Upon issuance, we will send copies to the Director of the Office of
Management and Budget and executive branch agencies (see appendix VII for
a list). We will also make copies available to others upon request. In
addition, this report will be available at no charge on the GAO web site
at http://www.gao.gov.
If you have any questions concerning this report, please contact Patricia
A. Dalton at (202) 512-6806 or [email protected]. The major contributors to
this report are listed in appendix XII.
David M. Walker Comptroller General of the United States
List of Requesters
The Honorable Susan M. Collins
Chairman
The Honorable Joe Lieberman
Ranking Minority Member
Committee on Governmental Affairs
United States Senate
The Honorable George V. Voinovich
Chairman
The Honorable Richard Durbin
Ranking Minority Member
Subcommittee on Oversight of Government Management, the Federal
Workforce, and the District of Columbia Committee on Governmental Affairs
United States Senate
The Honorable Peter G. Fitzgerald
Chairman
The Honorable Daniel K. Akaka
Ranking Minority Member
Subcommittee on Financial Management, the Budget, and International
Security Committee on Governmental Affairs United States Senate
The Honorable Tom Davis
Chairman
The Honorable Henry A. Waxman
Ranking Minority Member
Committee on Government Reform
House of Representatives
The Honorable Todd Russel Platts
Chairman
The Honorable Edolphus Towns
Ranking Minority Member
Subcommittee on Government Efficiency and Financial Management
Committee on Government Reform
House of Representatives
Executive Summary
Purpose From defending the homeland against terrorists, to preventing the
spread of infectious diseases, to providing a reliable stream of social
security income to retirees and supporting the transition from welfare to
work, the federal government provides funding and services to the American
public that can affect their lives in critical ways every day. However,
the federal government is in a period of profound transition and faces an
array of challenges and opportunities to enhance performance, ensure
accountability, and position the nation for the future. A number of
overarching trends, such as diffuse security threats and homeland security
needs, increasing global interdependency, the shift to knowledge-based
economies, and the looming fiscal challenges facing our nation drive the
need to reconsider the proper role for the federal government in the 21st
century, how the government should do business (including how it should be
structured), and in some instances, who should do the government's
business.
Without effective short-and long-term planning, which takes into account
the changing environment and needs of the American public and the
challenges they face and establishes goals to be achieved, federal
agencies risk delivering programs and services that may or may not meet
society's most critical needs. At a cost to taxpayers of over $2 trillion
annually, the federal government should be able to demonstrate to the
American public that it can anticipate emerging issues, develop sound
strategies and plans to address them, and be accountable for the results
that have been achieved.
Concerned that the federal government was more focused on program
activities and processes than the results to be achieved, Congress passed
the Government Performance and Results Act of 1993 (GPRA).1 The act
required federal agencies to develop strategic plans with long-term,
outcome-oriented goals and objectives, annual goals linked to achieving
the long-term goals, and annual reports on the results achieved. Now that
GPRA has been in effect for 10 years, you asked us to assess the
effectiveness of GPRA in creating a focus on results in the federal
government. Specifically, this report discusses (1) the effect of GPRA
over the last 10 years in creating a governmentwide focus on results and
the government's ability to deliver results to the American public,
including an assessment of the changes in the overall quality of agencies'
strategic plans,
1Pub. L. No. 103-62.
Executive Summary
annual performance plans, and annual performance reports; (2) the
challenges agencies face in measuring performance and using performance
information in management decisions; and (3) how the federal government
can continue to shift toward a more results-oriented focus.
To meet our objectives, we reviewed our extensive prior work on GPRA best
practices and implementation and collected governmentwide data to assess
the government's overall focus on results. We conducted a random,
stratified, governmentwide survey of federal managers comparable to
surveys we conducted in 1997 and 2000. We also held eight in-depth focus
groups-seven comprised of federal managers from 23 federal agencies and
one with GPRA experts. We also interviewed top appointed officials from
the current and previous administrations. Finally, we judgmentally
selected a sample of six agencies to review for changes in the quality of
their strategic plans, performance plans, and performance reports since
their initial efforts. The agencies we selected included the Departments
of Education (Education), Energy (DOE), Housing and Urban Development
(HUD), and Transportation (DOT) and the Small Business (SBA) and Social
Security Administrations (SSA). In making this selection, we chose
agencies that collectively represented the full range of characteristics
in the following four areas: (1) agency size (small, medium, large); (2)
primary program type (direct service, research, regulatory, transfer
payments, and contracts or grants); (3) quality of fiscal year 2000
performance plans based on our previous review;2 and (4) type of agency
(cabinet department and independent agency). Appendix I contains a more
detailed discussion of our scope and methodology. We performed our work in
Washington, D.C., from January through November 2003 in accordance with
generally accepted government auditing standards.
Background GPRA is the centerpiece of a statutory framework that Congress
put in place during the 1990s to help resolve the long-standing management
problems that have undermined the federal government's efficiency and
effectiveness and to provide greater accountability for results. GPRA was
intended to address several broad purposes, including strengthening the
confidence of the American people in their government; improving federal
program effectiveness, accountability, and service delivery; and enhancing
2U.S. General Accounting Office, Managing for Results: Opportunities for
Continued Improvements in Agencies' Performance Plans, GAO/GGD/AIMD-99-215
(Washington, D.C.: July 20, 1999).
Executive Summary
congressional decision making by providing more objective information on
program performance.
GPRA requires executive agencies to complete strategic plans in which they
define their missions, establish results-oriented goals, and identify the
strategies that will be needed to achieve those goals. GPRA requires
agencies to consult with Congress and solicit the input of others as they
develop these plans. Through this strategic planning requirement, GPRA has
required federal agencies to reassess their missions and long-term goals
as well as the strategies and resources they will need to achieve their
goals. Agencies developed their first strategic plans in fiscal year 1997,
and are required to update the plans every 3 years since then.
GPRA also requires executive agencies to prepare annual performance plans
that articulate goals for the upcoming fiscal year that are aligned with
their long-term strategic goals. These performance plans are to include
results-oriented annual goals linked to the program activities displayed
in budget presentations as well as the indicators the agency will use to
measure performance against the results-oriented goals. Agencies developed
their first annual performance plans in fiscal year 1999 and are required
to issue plans annually thereafter to correspond with budget submissions
to Congress.
Finally, GPRA requires agencies to measure performance toward the
achievement of the goals in the annual performance plan and report
annually on their progress in program performance reports. If a goal was
not met, the report is to provide an explanation and present the actions
needed to meet any unmet goals in the future. These reports are intended
to provide important information to agency managers, policymakers, and the
public on what each agency accomplished with the resources it was given.
Agencies issued their first annual performance reports on their fiscal
year 1999 performance in fiscal year 2000 and are required to issue a
report on each subsequent performance plan.
The Office of Management and Budget (OMB) plays an important role in the
management of federal government performance and specifically GPRA
implementation. Part of OMB's overall mission is to ensure that agency
plans and reports are consistent with the President's budget and
administration policies. OMB is responsible for receiving and reviewing
agencies' strategic plans, annual performance plans, and annual
performance reports. To improve the quality and consistency of these
documents, OMB issues annual guidance to agencies for their preparation,
Executive Summary
including guidelines on format, required elements, and submission
deadlines. GPRA requires OMB to prepare a governmentwide performance plan,
based on agencies' annual performance plan submissions. OMB also played an
important role in the pilot phase of GPRA implementation by designating
agencies for pilot projects in performance measurement, managerial
accountability and flexibility, and performance budgeting, and assessing
the results of the pilots. Finally, GPRA provides OMB with authority to
grant agencies waivers to certain administrative procedures and controls.
Recent OMB guidance-OMB Circular A-11, July 2003-requires agencies to
submit "performance budgets" in lieu of annual performance plans for their
fiscal year 2005 budget submission to OMB and Congress. According to OMB,
performance budgets should satisfy all the statutory requirements of GPRA
for annual performance plans. In addition, agencies are to include all
performance goals used in the assessment of program performance done under
OMB's Program Assessment Rating Tool (PART) process.3 Moreover, the
guidance states that until all programs have been assessed by PART, the
performance budget will also for a time include performance goals for
agency programs that have not yet been assessed using PART. The
expectation is that agencies are to substitute new or revised performance
goals resulting from OMB's review for goals it deemed unacceptable.
Results in Brief Among the purposes of GPRA cited by Congress was to
improve federal program effectiveness and service delivery by promoting a
new focus on results, service quality, and customer satisfaction by
setting program goals measuring performance against goals, and reporting
publicly on progress. Furthermore, GPRA was to improve congressional
decision making by providing more objective information on achieving
objectives, and on the relative effectiveness and efficiency of federal
programs and spending. Ten years after enactment, GPRA's requirements have
laid a solid foundation of results-oriented agency planning, measurement,
and reporting that have
3PART is a diagnostic tool developed by OMB that it has been using to rate
the effectiveness of federal programs with a particular focus on program
results. OMB's goal is to review all federal programs over a 5-year period
using the PART tool. OMB used the tool to review approximately 400
programs between the fiscal year 2004 budget cycle and the fiscal year
2005 budget cycle-234 programs were assessed last year and 173 were
assessed this year. Some reassessed programs were combined for review for
the 2005 budget, which is why the number of programs assessed over the 2
years does not add up to exactly 400 programs.
Executive Summary
begun to address these purposes. Focus group participants and high-level
political appointees, as well as OMB officials we interviewed, cited
positive effects of GPRA that they generally attributed to GPRA's
statutory requirements for planning and reporting. Performance planning
and measurement have slowly yet increasingly become a part of agencies'
cultures. The results of our stratified, random sample survey of federal
managers indicate that since GPRA went into effect governmentwide in 1997,
federal managers reported having significantly more of the types of
performance measures called for by GPRA-particularly outcome-oriented
performance measures. Survey data also suggested that more federal
managers, especially at the Senior Executive Service (SES) level, believed
that OMB was paying attention to their agencies' efforts under GPRA. GPRA
has also begun to facilitate the linking of resources to results, although
much remains to be done in this area.
Beginning with agencies' initial efforts to develop effective strategic
plans in 1997 and annual performance plans and reports for fiscal year
1999, Congress, GAO, and others have commented on the quality of those
efforts and provided constructive feedback on how agency plans and reports
could be improved. According to our current review of the strategic plans,
annual performance plans, and annual performance reports of six selected
agencies, these documents reflect much of the feedback that was provided.
For example, goals are more quantifiable and results oriented, and
agencies are providing more information about goals and strategies to
address performance and accountability challenges and the limitations to
their performance data. However, certain serious weaknesses persist, such
as lack of detail on how annual performance goals relate to strategic
goals and how agencies are coordinating with other entities to address
common challenges and achieve common objectives.
While a great deal of progress has been made in making federal agencies
more results oriented, numerous challenges still exist. As we have noted
before, top leadership commitment and sustained attention to achieving
results, both within the agencies and at OMB, is essential to GPRA
implementation. While one might expect an increase in agency leadership
commitment since GPRA was implemented governmentwide beginning in fiscal
year 1997, federal managers reported that such commitment has not
significantly increased. Furthermore, although OMB has recently
demonstrated leadership in its review of performance information from a
budgetary perspective using the PART tool, it is unclear whether the
results of those reviews, such as changes in program performance measures,
will complement and be integrated with the long-term, strategic focus of
GPRA.
Executive Summary
OMB provided significantly less guidance on GPRA implementation for the
fiscal year 2005 budget, compared to the very detailed guidance provided
in prior years. Without consistent guidance from OMB on meeting GPRA
requirements and following best practices, it may be difficult to maintain
the improvements in agency performance plans and reports or bring about
improvements in areas where weaknesses remain. The commitment of top
leadership within agencies, OMB, and Congress is critical to the success
of strategic planning efforts. However, GPRA specifies time frames for
updating strategic plans that do not correspond to presidential or
congressional terms. As a result, an agency may be required to update its
strategic plan a year before a presidential election and without input
from a new Congress. A strategic plan should reflect the policy priorities
of an organization's leaders and the input of key stakeholders if it is to
be an effective management tool.
Managers reported they had more performance measures, but indications that
managers are making greater use of this information to improve performance
are mixed. Additionally, managers reported several human capital-related
challenges that impede results-oriented management, including a lack of
authority and training to carry out GPRA requirements, as well as a lack
of recognition for completing these tasks. Unfortunately, most existing
federal performance appraisal systems are not designed to support a
meaningful performance-based pay system in that they fail to link
institutional, program, unit, and individual performance measurement and
reward systems. Fewer than half of federal managers reported receiving
relevant training in critical results-oriented management-related tasks.
Managers also reported significant challenges persist in setting
outcomeoriented goals, measuring performance, and collecting useful data.
In some agencies, particularly those that have a research and development
component, managers reported difficulties in establishing meaningful
outcome measures. Managers also identified difficulties in distinguishing
between the results produced by the federal program and results caused by
external factors or nonfederal actors, such as with grant programs. Timely
and useful performance information is not always available to federal
agencies, making it more difficult to assess and report on progress
achieved. Finally, agency officials believe that Congress could make
greater use of performance information to conduct oversight and to inform
appropriations decisions. GPRA provides a vehicle for Congress to
explicitly state its performance expectations in outcome-oriented terms
when establishing new programs or in exercising oversight of existing
programs that are not achieving desired results.
Executive Summary
Mission fragmentation and overlap contribute to difficulties in addressing
crosscutting issues, particularly when those issues require a national
focus, such as homeland security, drug control, and the environment. GPRA
requires a governmentwide performance plan, where these issues could be
addressed in a centralized fashion, but OMB has not issued a distinct plan
since 1999. Most recently, the President's fiscal year 2004 budget focused
on describing agencies' progress in addressing the President's Management
Agenda (PMA) and the results of PART reviews of agency programs. Such
information is important and useful, but is not adequate alone to provide
a broader and more integrated perspective of planned performance on
governmentwide outcomes. GAO has previously reported on a variety of
barriers to interagency cooperation, such as conflicting agency missions,
jurisdiction issues, and incompatible procedures, data, and processes. A
strategic plan for the federal government, supported by a set of key
national indicators to assess the government's performance, position, and
progress, could provide an additional tool for governmentwide
reexamination of existing programs, as well as proposals for new programs.
Such a plan could be of particular value in linking agencies' long-term
performance goals and objectives horizontally across the government. In
addition, it could provide a basis for integrating, rather than merely
coordinating, a wide array of federal activities.
To address these challenges, continued and sustained commitment and
leadership are needed. OMB, as the primary focal point for overall
management in the federal government, can provide this leadership and
direction working with the various management councils and work groups of
the government. Also, governmentwide planning could better facilitate the
integration of federal activities to achieve national goals.
GAO recommends that the Director of OMB (1) fully implement GPRA's
requirement to develop a governmentwide performance plan; (2) articulate
and implement an integrated, complementary relationship between GPRA and
PART; (3) provide clearer and consistent guidance to executive branch
agencies on how to implement GPRA; (4) continue to maintain a dialogue
with agencies about their performance measurement practices with a
particular focus on grant-making, research and development, and regulatory
functions to identify and replicate successful approaches agencies are
using to measure and report on their outcomes, including the use of
program evaluation tools; and, work with executive branch agencies to
identify the barriers to obtaining timely data to show progress against
performance goals and the best ways to report information when there are
unavoidable lags in data availability; and (5) work with agencies to
ensure
Executive Summary
they are making adequate investments in training on performance planning
and measurement, with a particular emphasis on how to use performance
information to improve program performance.
We also suggest that Congress consider amending GPRA to require that
updates to agency strategic plans be submitted at least once every 4
years, 12-18 months after a new administration begins its term.
Additionally, consultations with congressional stakeholders on existing
strategic plans should be held at least once every new Congress and
revisions should be made as needed. Further, we suggest Congress use these
consultations and its oversight role to clarify its performance
expectations for agencies. Congress should also consider amending GPRA to
require the President to develop a governmentwide strategic plan.
In commenting on a draft of this report, OMB generally agreed with our
findings and conclusions. OMB agreed to implement most of our
recommendations, but stated that the President's Budget represents the
executive branch's governmentwide performance plan and could also serve as
a governmentwide strategic plan. However, because of the budget's focus on
agency-level expenditures for the upcoming fiscal year, we believe that
the President's Budget provides neither a long-term nor an integrated
perspective on the federal government's performance. OMB's comments appear
in appendix VIII. Our response appears in chapter 5. We also provided
relevant sections of the draft to the six agencies whose plans and reports
we reviewed. DOE, HUD, and SSA disagreed with some of our observations,
and we changed or clarified relevant sections of the report, as
appropriate. Written comments from DOE, HUD, and SSA are reprinted in
appendixes IX, X, and XI, respectively, along with our responses.
Principal Findings
GPRA Laid the Foundation Prior to enactment of GPRA, our 1992 review of
the collection and use of for a More Results-Oriented performance data by
federal agencies revealed that, although many Federal Government agencies
collected performance information at the program level, few
agencies had results-oriented performance information to manage or make
Executive Summary
strategic policy decisions for the agency as a whole.4 GPRA addressed
agencies' shortcomings by creating a comprehensive and consistent
statutory foundation of required agencywide strategic plans, annual
performance plans, and annual performance reports. Participants in eight
focus groups comprised of experts on GPRA and federal managers from 23
agencies cited the creation of this statutory foundation as one of the key
accomplishments of GPRA. One of the premises of GPRA is that both
congressional and executive branch oversight of federal agency performance
were seriously hampered by a lack of adequate resultsoriented goals and
performance information. As noted above, prior to the enactment of GPRA
few agencies reported their performance information externally. OMB
officials we interviewed as part of our current review suggested that OMB
has been a key consumer of the performance information produced under GPRA
and that it has provided a foundation for their efforts to oversee agency
performance.
Federal managers' views of GPRA's effect on the federal government's
ability to deliver results to the American public were mixed. When asked
about the direct effects of GPRA on the public, 23 percent of the federal
managers surveyed agreed to a moderate or greater extent that GPRA
improved their agency's ability to deliver results to the American public.
High-level political appointees we interviewed cited a number of examples
of how the structure of GPRA created a greater focus on results in their
agencies. Participants in our focus groups had mixed perceptions of GPRA's
effect on their agency's ability to deliver results to the American
public. Participants indicated GPRA has had a positive effect by shifting
the focus of federal management from program activities and processes to
achieving the intended results of those programs. Another major
accomplishment of GPRA cited by focus group participants is that GPRA
improved the transparency of government results to the American public.
Other focus group participants had difficulty attributing the results
their agencies achieved directly to GPRA's requirements.
4U.S. General Accounting Office, Program Performance Measures: Federal
Agency Collection and Use of Performance Data, GAO/GGD-92-65 (Washington,
D.C.: May 4, 1992).
Executive Summary
Focus group and survey results suggest that performance planning and
measurement have slowly, but increasingly, become a part of agencies'
cultures. Compared to the results of our 1997 governmentwide survey of
federal managers, in our 2003 governmentwide survey more managers reported
having performance measures for their programs. When we asked managers who
said they had performance measures which of the five types of measures
they had to a great or very great extent, they reported increases in all
five types of measures between 1997 and 2003,5 all of which were
statistically significant.
Similarly, focus group participants commented on certain cultural changes
that had taken place within their agencies since the passage of GPRA in
which the "vocabulary" of performance planning and measurement-e.g., a
greater focus on performance measurement, orientation toward outcomes over
inputs and outputs, and an increased focus on program evaluation- had
become more pervasive. This perception is partly born out by our survey
results. Consistent with our survey results indicating increases in
results-oriented performance measures, we also observed a significant
decline in the percentage of federal managers who agreed that certain
factors hindered measuring performance or using the performance
information. Finally, our survey data suggested that more federal
managers, especially at the SES level, believed that OMB was paying
attention to their agencies' efforts under GPRA, but with no corresponding
increase in their concern that OMB would micromanage the programs in their
agencies.
5Types of measures were defined in the questionnaire as follows:
performance measures that tell us how many things we produce or services
we provide (output measures); performance measures that tell us if we are
operating efficiently (efficiency measures); performance measures that
tell us whether or not we are satisfying our customers (customer service
measures); performance measures that tell us about the quality of the
products or services we provide (quality measures); and performance
measures that would demonstrate to someone outside of our agency whether
or not we are achieving our intended results (outcome measures).
Executive Summary
Agencies have begun to establish a link between results and resources. Our
1998 assessment of fiscal year 1999 performance plans found that agencies
generally covered the program activities in their budgets, but most plans
did not identify how the funding for those program activities would be
allocated to performance goals.6 However, our subsequent reviews of
performance plans indicate that agencies have made progress in
demonstrating how their performance goals and objectives relate to program
activities in the budget.
We reviewed a sample of six agencies' strategic plans (Education, DOE,
HUD, DOT, SBA, and SSA) and found the quality of the selected plans
reflected improvements over these agencies' initial strategic plans. Our
1997 review of agencies' draft strategic plans found that a significant
amount of work remained to be done by executive branch agencies if their
strategic plans were to fulfill the requirements of GPRA, serve as a basis
for guiding agencies, and help congressional and other policymakers make
decisions about agency activities and programs.7 The six strategic plans
we looked at for this 2003 review reflected many new and continuing
strengths as well as improvements over the 1997 initial draft plans, but
we continued to find certain persistent weaknesses. Of the six elements
required by GPRA, the plans generally discussed all but one-program
evaluation, an area in which we have found agencies often lack capacity.
Although the strategic plans listed the program evaluations agencies
intended to complete over the planning period, they generally did not
address how the agencies planned to use their evaluations to establish new
or revise existing strategic goals, as envisioned by GPRA. Finally,
although not required by GPRA, the strategic plans would have benefited
from more complete discussions of how agencies planned to coordinate and
collaborate with other entities to address common challenges and achieve
common or complementary goals and objectives.
6U.S. General Accounting Office, Managing for Results: An Agenda to
Improve the Usefulness of Agencies' Annual Performance Plans,
GAO/GGD/AIMD-98-228 (Washington, D.C.: Sept. 8, 1998).
7U.S. General Accounting Office, Managing for Results: Critical Issues for
Improving Agencies' Strategic Plans, GAO/GGD-97-180 (Washington, D.C.:
Sept. 16, 1997).
Executive Summary
The six selected agencies' fiscal year 2004 annual performance plans
addressed some weaknesses of earlier plans, but there is still significant
room for improvement. During our review of agencies' first annual
performance plans, which presented agencies' annual performance goals for
fiscal year 1999,8 we found that substantial further development was
needed for these plans to be useful in a significant way to congressional
and other decision makers. Most of the 2004 plans that we reviewed showed
meaningful improvements over the fiscal year 1999 plans by showing a
clearer picture of intended performance, providing strategies and
resources that were more specifically related to achieving agency goals,
and providing a greater level of confidence that performance data would be
credible. But these plans also contained a number of serious weaknesses,
such as inadequate discussion of coordination and collaboration and
inconsistent or limited discussions of procedures used to verify and
validate performance data, which limited their quality and undermined
their usefulness.
Our review of the six agencies' fiscal year 2002 performance reports
showed a number of strengths and improvements over their fiscal year 1999
performance reports, as well as areas that needed improvement. As we found
in our earlier reviews, the six agencies' fiscal year 2002 reports
generally allowed for an assessment of progress made in achieving agency
goals. In addition, the majority of agencies discussed the progress
achieved in addressing performance and accountability challenges
identified by agency inspectors general and GAO. However, as with the
fiscal year 1999 reports, many of the weaknesses we identified in the
agencies' fiscal year 2002 reports were related to the significant number
of performance goals not achieved or for which performance data were
unavailable. In addition, the majority of the reports we reviewed did not
include other GPRA requirements, such as a summary of the findings from
program evaluations. Finally, only one of the six agencies clearly linked
its costs to the achievement of performance goals or objectives.
Challenges to GPRA While a great deal of progress has been made in making
federal agencies
Implementation Exist more results oriented, numerous challenges still
exist to effective implementation of GPRA. We observed in our 1997 report
that we would expect to see managers' positive perceptions on items, such
as the extent
8GAO/GGD/AIMD-98-228.
Executive Summary
to which top leadership is committed to achieving results, become more
prevalent and the gap between SES and non-SES managers begin to narrow as
GPRA and related reforms are implemented. However, these changes do not
appear to be happening to the extent anticipated. The need for strong,
committed, and sustained leadership extends to OMB as well. OMB has shown
a commitment to improving the management of federal programs, both through
its leadership in reviewing agency program performance using the PART tool
as well as through the PMA. As part of the President's budget preparation,
PART clearly must serve the President's interests. However, it is not well
suited to addressing crosscutting (or horizontal) issues or to looking at
broad program areas in which several programs address a common goal. GPRA
was designed to address the needs of many users of performance
information, including (1) Congress to provide oversight and inform
funding decisions, (2) agency managers to manage programs and make
internal resource decisions, and (3) the public to provide greater
accountability. It is not yet clear the extent to which PART performance
goals and measures will compete with agencies' long-term, strategic GPRA
goals and objectives that were established in consultation with Congress
and other stakeholders.
We also found that, while the quality of agency plans and reports have
improved overall since their initial efforts, they continue to suffer from
certain persistent weaknesses as noted above. However, OMB's July 2003
guidance for preparation and submission of annual performance plans is
significantly shorter and less detailed than its 2002 guidance.
Consistent, more explicit OMB guidance on preparing GPRA documents can
help ensure that gains in the quality of GPRA documents are maintained and
provide a resource for agency managers to make further improvements in
those documents.
We also found that timing issues may affect the development of agency
strategic plans that are meaningful and useful to top leadership. The
commitment and sustained attention of top leadership within agencies, OMB,
and Congress is critical to the success of strategic planning efforts. A
strategic plan should reflect the policy priorities of an organization's
leaders and the input of key stakeholders if it is to be an effective
management tool. However, GPRA specifies time frames for updating
strategic plans that do not correspond to presidential or congressional
terms. As a result, an agency may be required to update its strategic plan
a year before a presidential election and without input from a new
Congress. If a new president is elected, the updated plan is essentially
moot and agencies must spend additional time and effort revising it to
reflect new
Executive Summary
priorities. Our focus group participants, including GPRA experts, strongly
agreed that this timing issue should be addressed by adjusting time frames
to correspond better with presidential and congressional terms.
The benefit of collecting performance information is only fully realized
when this information is actually used by managers to bring about desired
results. However, federal managers reported mixed results in the use of
performance information. Focus group participants and survey respondents
noted that although many federal managers understand and use
results-oriented management concepts in their day-to-day activities, such
as strategic planning and performance measurement, they do not always
connect these concepts to the requirements of GPRA. According to our 2003
survey results, the reported use of performance information to a great or
very great extent for nine management activities, such as setting program
priorities or setting individual job expectations for staff, ranging from
41 to 66 percent, has not changed significantly since our first survey in
1997. One exception was the reported use to a great or very great extent
of performance information to adopt new program approaches or change work
processes, which was significantly lower than the 1997 results. GPRA's
usefulness to agency leaders and managers as a tool for management and
accountability was cited as a key accomplishment numerous times by focus
group participants. However, a number of alternative views indicated that
the usefulness of GPRA as a management tool has been limited. Our survey
data also indicate that managers' perceive their participation in
activities related to the development and use of performance information
has been limited.
Federal managers continue to confront a range of important human capital
management challenges. These managers report that they are held
accountable for program results, but may not have the decision-making
authority they need to accomplish agency goals. Moreover, fewer than half
of managers reported receiving relevant training. Managers also perceive a
lack of positive recognition for helping agencies achieve results.
Unfortunately, most existing federal performance appraisal systems are not
designed to support a meaningful performance-based pay system in that they
fail to link institutional, program, unit, and individual performance
measurement and reward systems. In our view, one key need is to modernize
performance management systems in executive agencies so that they link to
the agency's strategic plan, related goals, and desired outcomes and are
therefore capable of adequately supporting more performancebased pay and
other personnel decisions.
Executive Summary
Managers reported persistent challenges in setting outcome-oriented goals,
measuring performance, and collecting useful data. Focus group
participants and survey respondents noted that outcome-oriented
performance measures were especially difficult to establish when the
program or line of effort was not easily quantifiable. In some agencies,
particularly those that have a research and development component,
managers reported difficulties in establishing meaningful outcome
measures. Managers also identified difficulties in distinguishing between
the results produced by the federal program and results caused by external
factors or nonfederal actors, such as with grant programs. Finally,
managers reported that timely and useful performance information is not
always available.
Crosscutting issues continue to be a challenge to GPRA implementation. Our
review of six agencies' strategic and annual performance plans showed some
improvement in addressing their crosscutting program efforts, but a great
deal of improvement is still necessary. We have previously reported and
testified that GPRA could provide OMB, agencies, and Congress with a
structured framework for addressing crosscutting policy initiatives and
program efforts. OMB could use the provision of GPRA that calls for OMB to
develop a governmentwide performance plan to integrate expected
agency-level performance. It could also be used to more clearly relate and
address the contributions of alternative federal strategies.
Unfortunately, this provision has not been fully implemented. Instead, OMB
has used the President's Budget to present high-level information about
agencies and certain program performance issues. The current
agency-by-agency focus of the budget does not provide the integrated
perspective of government performance envisioned by GPRA. For example, the
fiscal year 2004 budget identified budget requests and performance
objectives by agency, such as the U.S. Department of Defense, as opposed
to crosscutting governmentwide themes. From this presentation, one could
assume that the only activities the U.S. government planned to carry out
in support of national defense were those listed under the chapter
"Department of Defense." However, the chapter on the fiscal year 2004
budget discussing "the Department of State and International Assistance
Programs," contains a heading titled, "Countering the Threat from Weapons
of Mass Destruction." And while OMB may have a technical reason for not
classifying this task as being related to national defense or homeland
security, it is unclear that a lay reader could make that distinction. The
fiscal year 2005 budget also identified budget requests and performance
objectives by agency, not by crosscutting theme.
Executive Summary
A strategic plan for the federal government could provide an additional
tool for governmentwide reexamination of existing programs, as well as
proposals for new programs. If fully developed, a governmentwide strategic
plan could potentially provide a cohesive perspective on the longterm
goals of the federal government and provide a much needed basis for fully
integrating, rather than merely coordinating, a wide array of federal
activities. Successful strategic planning requires the involvement of key
stakeholders. Thus, it could serve as a mechanism for building consensus.
Further, it could provide a vehicle for the President to articulate
long-term goals and a road map for achieving them. In addition, a
strategic plan could provide a more comprehensive framework for
considering organizational changes and making resource decisions. The
development of a set of key national indicators could be used as a basis
to inform the development of governmentwide strategic and annual
performance plans. The indicators could also link to and provide
information to support outcome-oriented goals and objectives in
agency-level strategic and annual performance plans.
Focus group members believed that one of the main challenges to GPRA
implementation was the reluctance of Congress to use performance
information when making decisions, especially appropriations decisions.
However, less than one quarter of federal managers in the 2003 survey
shared that concern. Further, a recent Congressional Research Service
review suggests that Congress uses performance information to some extent,
as evidenced by citations in legislation and committee reports. While
there is concern regarding Congress' use of performance information, it is
important to make sure that this information is useful. In other words,
the information presented and its presentation must meet the needs of the
user. Regular consultation with Congress about both the content and format
of performance plans and reports is critical.
As a key user of performance information, Congress also needs to be
considered a partner in shaping agency goals at the outset. GPRA provides
a vehicle for Congress to explicitly state its performance expectations in
outcome-oriented terms when consulting with agencies on their strategic
plans or when establishing new programs or exercising oversight of
existing programs that are not achieving desired results. This would
provide important guidance to agencies that could then be incorporated in
agency strategic and annual performance plans.
Executive Summary
Recommendations for Executive Action
GAO recommends that the Director of OMB implement five suggestions to
improve its guidance and oversight of GPRA implementation.
To provide a broader perspective and more cohesive picture of the federal
government's goals and strategies to address issues that cut across
executive branch agencies, we recommend that the Director of OMB fully
implement GPRA's requirement to develop a governmentwide performance plan.
To achieve the greatest benefit from both GPRA and PART, we recommend that
the Director of OMB articulate and implement an integrated and
complementary relationship between the two. GPRA is a broad legislative
framework that was designed to be consultative with Congress and other
stakeholders, and allows for varying uses of performance information. PART
looks through a particular lens for a particular use-the executive budget
formulation process.
To improve the quality of agencies' strategic plans, annual performance
plans, and performance reports and help agencies meet the requirements of
GPRA, we recommend that the Director of OMB provide clearer and consistent
guidance to executive branch agencies on how to implement GPRA. Such
guidance should include standards for communicating key performance
information in concise as well as longer formats to better meet the needs
of external users who lack the time or expertise to analyze lengthy,
detailed documents.
To help address agencies' performance measurement challenges, we recommend
the Director of OMB engage in a continuing dialogue with agencies about
their performance measurement practices with a particular focus on
grant-making, research and development, and regulatory functions to
identify and replicate successful approaches agencies are using to measure
and report on their outcomes, including the use of program evaluation
tools. Additionally, we recommend that the Director of OMB work with
executive branch agencies to identify the barriers to obtaining timely
data to show progress against performance goals and the best ways to
report information where there are unavoidable lags in data availability.
Governmentwide councils, such as the President's Management Council and
the Chief Financial Officers Council, may be effective vehicles for
working on these issues.
Executive Summary
To facilitate the transformation of agencies' management cultures to be
more results oriented, we recommend that the Director of OMB work with
agencies to ensure they are making adequate investments in training on
performance planning and measurement, with a particular emphasis on how to
use performance information to improve program performance.
Matters for Congressional Consideration
GAO also identified two matters for congressional consideration to improve
the governmentwide focus on results.
To ensure that agency strategic plans more closely align with changes in
the federal government leadership, Congress should consider amending GPRA
to require that updates to agency strategic plans be submitted at least
once every 4 years, 12-18 months after a new administration begins its
term. Additionally, consultations with congressional stakeholders should
be held at least once every new Congress and interim updates made to
strategic and performance plans as warranted. Congress should consider
using these consultations along with its traditional oversight role and
legislation as opportunities to clarify its performance expectations for
agencies. This process may provide an opportunity for Congress to develop
a more structured oversight agenda.
To provide a framework to identify long-term goals and strategies to
address issues that cut across federal agencies, Congress should consider
amending GPRA to require the President to develop a governmentwide
strategic plan.
Agency Comments We provided a copy of the draft report to OMB for comment.
OMB's written comments are reprinted in appendix VIII. In general, OMB
agreed with our findings and conclusions. OMB agreed to implement most of
our recommendations, noting that these recommendations will enhance their
efforts to make the government more results oriented. OMB agreed to (1)
work with agencies to ensure they are provided adequate training in
performance management, (2) revise its guidance to clarify the integrated
and complementary relationship between GPRA and PART, and (3) continue to
use PART to improve agency performance measurement practices and share
those practices across government.
In response to our recommendation that OMB fully implement GPRA's
requirement to develop a governmentwide performance plan, OMB stated
Executive Summary
that the President's Budget represents the executive branch's
governmentwide performance plan. However, according to GAO's review, the
agency-by-agency focus of the budget over the past few years does not
provide an integrated perspective of government performance, and thus does
not meet GPRA's requirement to provide a "single cohesive picture of the
annual performance goals for the fiscal year." To clarify this point, we
added an example that illustrates the lack of integration between
crosscutting issues in the budget.
In response to our matter for congressional consideration that Congress
should consider amending GPRA to require the President to develop a
governmentwide strategic plan, OMB noted that the budget serves as the
governmentwide strategic plan. However, the President's Budget focuses on
establishing agency budgets for the upcoming fiscal year. Unlike a
strategic plan, it provides neither a long-term nor an integrated
perspective on the federal government's activities. A governmentwide
strategic plan should provide a cohesive perspective on the long-term
goals of the federal government and provide a basis for fully integrating,
rather than primarily coordinating, a wide array of existing and
relatively short-term federal activities.
We provided relevant sections of the draft report to Education, DOE, HUD,
SBA, SSA, and DOT. Education and SBA did not provide any comments, while
DOT provided minor technical comments. DOE, HUD, and SSA disagreed with
some of our observations on their strategic plans, performance plans, and
performance reports; we changed or clarified relevant sections of the
report, as appropriate. Written comments from DOE, HUD, and SSA are
reprinted in appendixes IX, X, and XI, respectively, along with our
responses.
Chapter 1
Introduction
From defending the homeland against terrorists, to preventing the spread
of infectious diseases, to providing a reliable stream of social security
income to retirees and supporting the transition from welfare to work, the
federal government provides funding and services to the American public
that can affect their lives in critical ways every day. However, the
federal government is in a period of profound transition and faces an
array of challenges and opportunities to enhance performance, ensure
accountability, and position the nation for the future. A number of
overarching trends, such as diffuse security threats and homeland security
needs, increasing global interdependency, the shift to knowledge-based
economies, and the looming fiscal challenges facing our nation, drive the
need to reconsider the proper role for the federal government in the 21st
century, how the government should do business (including how it should be
structured), and in some instances, who should do the government's
business.
Without effective short-and long-term planning, which takes into account
the changing environment and needs of the American public and the
challenges they face and establishes goals to be achieved, federal
agencies risk delivering programs and services that may or may not meet
society's most critical needs. At a cost to taxpayers of over $2 trillion
annually, the federal government should be able to demonstrate to the
American public that it can anticipate emerging issues, develop sound
strategies and plans to address them, and be accountable for the results
that have been achieved.
Concerned that the federal government was more focused on program
activities and processes than the results to be achieved, Congress passed
the Government Performance and Results Act of 1993 (GPRA).1 The act
required federal agencies to develop strategic plans with long-term
strategic goals, annual goals linked to achieving the long-term goals, and
annual reports on the results achieved. Now that GPRA has been in effect
for 10 years, you asked us to assess the effectiveness of GPRA in creating
a focus on results in the federal government. Specifically, this report
discusses (1) the effect of GPRA over the last 10 years in creating a
governmentwide focus on results and the government's ability to deliver
results to the American public, including an assessment of the changes in
the overall quality of agencies' strategic plans, annual performance
plans, and annual performance reports; (2) the challenges agencies face in
1Pub. L. No. 103-62.
Chapter 1 Introduction
measuring performance and using performance information in management
decisions; and (3) how the federal government can continue to shift toward
a more results-oriented focus.
Impact of Emerging Trends and Fiscal Challenges
With the 21st century challenges we are facing, it is more vital than ever
to maximize the performance of federal agencies in achieving their
long-term goals. The federal government must address and adapt to major
trends in our country and around the world. At the same time, our nation
faces serious long-term fiscal challenges. Increased pressure also comes
from world events: both from the recognition that we cannot consider
ourselves "safe" between two oceans-which has increased demands for
spending on homeland security-and from the United States (U.S.) role in
combating terrorism in an increasingly interdependent world. To be able to
assess federal agency performance and hold agency managers accountable for
achieving their long-term goals, we need to know what the level of
performance is. GPRA planning and reporting requirements can provide this
essential information.
Our country's transition into the 21st century is characterized by a
number of key trends, including
o the national and global response to terrorism and other threats to our
personal and national security;
o the increasing interdependence of enterprises, economies, markets,
civil societies, and national governments, commonly referred to as
globalization;
o the shift to market-oriented, knowledge-based economies;
o an aging and more diverse U.S. population;
o rapid advances in science and technology and the opportunities and
challenges created by these changes;
o challenges and opportunities to maintain and improve the quality of
life for the nation, communities, families, and individuals; and
o the changing and increasingly diverse nature of governance structures
and tools.
Chapter 1 Introduction
As the nation and government policymakers grapple with the challenges
presented by these evolving trends, they do so in the context of rapidly
building fiscal pressures. GAO's long-range budget simulations show that
this nation faces a large and growing structural deficit due primarily to
known demographic trends and rising health care costs. The fiscal
pressures created by the retirement of the baby boom generation and rising
health costs threaten to overwhelm the nation's fiscal future. As figure 1
shows, by 2040, absent reform or other major tax or spending policy
changes, projected federal revenues will likely be insufficient to pay
more than interest on publicly held debt. Further, our recent shift from
surpluses to deficits means the nation is moving into the future in a more
constrained fiscal position.
Figure 1: Composition of Spending as a Share of GDP Assuming Discretionary
Spending Grows with GDP after 2003 and All Expiring Tax Provisions Are
Extended
50 Percent of GDP
40
30
20
10
0 2003 2015 2030 2040 Fiscal year
Net interest Social Security Medicare and Medicaid All other spending
Source: GAO's January 2004 analysis.
Notes: Although all expiring tax cuts are extended, revenue as a share of
gross domestic product (GDP) increases through 2013 due to (1) real
bracket creep, (2) more taxpayers becoming subject to the Alternative
Minimum Tax, and (3) increased revenue from tax-deferred retirement
accounts. After
Chapter 1 Introduction
2013, revenue as a share of GDP is held constant. This simulation assumes
that currently scheduled Social Security benefits are paid in full
throughout the simulation period.
The United States has had a long-range budget deficit problem for a number
of years, even during recent years in which we had significant annual
budget surpluses. Unfortunately, the days of surpluses are gone, and our
current and projected budget situation has worsened significantly. The
bottom line is that our projected budget deficits are not manageable
without significant changes in "status quo" programs, policies, processes,
and operations.
GPRA Background GPRA is the centerpiece of a statutory framework that
Congress put in place during the 1990s to help resolve the long-standing
management problems that have undermined the federal government's
efficiency and effectiveness and to provide greater accountability for
results. In addition to GPRA, the framework comprises the Chief Financial
Officers Act of 1990, as amended by the Government Management Reform Act
of 1994, and information technology reform legislation, including the
Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996.
Together, these laws provide a powerful framework for developing and
integrating information about agencies' missions and strategic priorities,
the resultsoriented performance goals that flow from those priorities,
performance data to show the level of achievement of those goals, and the
relationship of reliable and audited financial information and information
technology investments to the achievement of those goals.
GPRA was intended to address several broad purposes, including
strengthening the confidence of the American people in their government;
improving federal program effectiveness, accountability, and service
delivery; and enhancing congressional decision making by providing more
objective information on program performance.
Chapter 1 Introduction
The basic requirements of GPRA for the preparation of strategic plans,
annual performance plans, and annual program performance reports by
executive branch agencies are the following:
Chapter 1 Introduction
The Office of Management and Budget (OMB) plays an important role in the
management of the federal government's performance, and specifically GPRA
implementation. Part of OMB's overall mission is to ensure that agency
plans and reports are consistent with the President's Budget and
administration policies. OMB is responsible for receiving and reviewing
agencies' strategic plans, annual performance plans, and annual
performance reports. To improve the quality and consistency of these
documents, OMB issues annual guidance to agencies for their preparation,
including guidelines on format, required elements, and submission
deadlines.2 GPRA requires OMB to prepare the overall governmentwide
performance plan, based on agencies' annual performance plan submissions.
OMB also played an important role in the pilot phase of GPRA
implementation by designating agencies for pilot projects in performance
measurement, managerial accountability and flexibility, and performance
budgeting, and assessing the results of the pilots. Finally, GPRA provides
OMB with authority to grant agencies waivers to certain administrative
procedures and controls.
Recent OMB guidance3 requires agencies to submit "performance budgets" in
lieu of annual performance plans for their budget submission to OMB and
Congress. Performance budgets are to meet all the statutory requirements
of GPRA for annual performance plans. In addition, agencies are to include
all performance goals used in the assessment of program performance done
under OMB's Program Assessment Rating Tool (PART) process.4 Moreover, the
guidance states that until all programs have been assessed by PART, the
performance budget will also for a time include performance goals for
agency programs that have not yet been assessed using PART. The
expectation is that agencies are to substitute new or revised performance
goals resulting from OMB's review for goals it deemed unacceptable.
2The guidance on the preparation of strategic plans, annual performance
plans, and program performance reports is contained in OMB Circular A-11,
Part 6.
3OMB Circular A-11, July 2003.
4PART is a diagnostic tool developed by OMB that it has been using to rate
the effectiveness of federal programs with a particular focus on program
results. OMB's goal is to review all federal programs over a 5-year period
using the PART tool. OMB used the tool to review approximately 400
programs between the fiscal year 2004 budget cycle and the fiscal year
2005 budget cycle-234 programs were assessed last year and 173 were
assessed this year. Some reassessed programs were combined for review for
the 2005 budget, which is why the number of programs assessed over the 2
years does not add up to exactly 400 programs.
Chapter 1 Introduction
In crafting GPRA, Congress recognized that managerial accountability for
results is linked to managers having sufficient flexibility, discretion,
and authority to accomplish desired results. GPRA authorizes agencies to
apply for managerial flexibility waivers in their annual performance plans
beginning with fiscal year 1999. The authority of agencies to request
waivers of administrative procedural requirements and controls is intended
to provide federal managers with more flexibility to structure agency
systems to better support program goals. The nonstatutory requirements
that OMB can waive under GPRA generally involve the allocation and use of
resources, such as restrictions on shifting funds among items within a
budget account. Agencies must report in their annual performance reports
on the use and effectiveness of any GPRA managerial flexibility waivers
that they receive.
OMB was to designate at least five agencies from the first set of pilot
projects to test managerial accountability and flexibility during fiscal
years 1995 and 1996. We previously reported on the results of the pilot
project to implement managerial flexibility waivers and found that the
pilot did not work as intended.5 OMB did not designate any of the seven
departments and one independent agency that submitted a total of 61 waiver
proposals as GPRA managerial accountability and flexibility pilots. For
about threequarters of the waiver proposals, OMB or other central
management agencies determined that the waivers were not allowable for
statutory or other reasons or that the requirement for which the waivers
were proposed no longer existed. For the remaining proposals, OMB or other
central management agencies approved waivers or developed compromises by
using authorities that were already available independent of GPRA.
Under GPRA, another set of pilot projects, which were scheduled for fiscal
years 1998 and 1999, were to test performance budgeting-i.e., the
presentation of the varying levels of performance that would result from
different budget levels. We previously reported that OMB initially
deferred these pilots-originally to be designated in fiscal years 1998 and
1999-to give federal agencies time to develop the capability of
calculating the effects of marginal changes in cost or funding on
performance.6 When the
5U.S. General Accounting Office, GPRA: Managerial Accountability and
Flexibility Pilot Did Not Work as Intended, GAO/GGD-97-36 (Washington,
D.C.: Apr. 10, 1997).
6U.S. General Accounting Office, Managing for Results: Agency Progress in
Linking Performance Plans With Budgets and Financial Statements,
GAO-02-236 (Washington, D.C.: Jan. 4, 2002).
Chapter 1 Introduction
pilots began in August 1999, OMB designed them as case studies prepared by
OMB staff to demonstrate how performance information could be used to
compare alternatives and to develop funding recommendations for
incorporation into the President's fiscal year 2001 budget submission.
On January 18, 2001, OMB reported the results of five performance
budgeting pilots that explored agencies' capabilities of more formally
assessing the effects of different funding levels on performance goals.
OMB selected the pilots to reflect a cross section of federal functions
and capabilities so that a representative range of measurement and
reporting issues could be explored. In its report, OMB concluded that
legislative changes were not needed. OMB reported that the pilots
demonstrated that assuring further performance measurement improvements
and steadily expanding the scope and quality of performance measures is
paramount, and that the existing statute provides sufficient latitude for
such improvement.
Overall, OMB concluded that the pilots raised several key challenges about
performance budgeting at the federal level including, for example, the
following:
o In many instances, measuring the effects of marginal, annual budget
changes on performance is not precise or meaningful.
o While continuing to change from an almost total reliance on output
measures to outcome measures, it will be much more difficult to associate
specific resource levels with those outcomes, particularly over short
periods of time.
o Establishing clear linkages between funding and outcomes will vary by
the nature of the program and the number of external factors.
o Delays in the availability of performance data, sometimes caused by
agencies' reliance on nonfederal program partners for data collection,
will continue to present synchronization problems during budget
formulation.
Chapter 1 Introduction
Scope and Methodology
To meet the three objectives stated earlier, we reviewed our extensive
prior work on GPRA best practices and implementation and collected
governmentwide data to assess the government's overall focus on results.
We conducted a random, stratified, governmentwide survey of federal
managers comparable to surveys we conducted in 1997 and 2000. We also held
eight in-depth focus groups-seven comprised of federal managers from 23
federal agencies and one with GPRA experts. We also interviewed top
appointed officials from the current and previous administrations.
Finally, we judgmentally selected a sample of six agencies to review for
changes in the quality of their strategic plans, performance plans, and
performance reports since their initial efforts. The agencies we selected
were the Departments of Education (Education), Energy (DOE), Housing and
Urban Development (HUD), and Transportation (DOT) and the Small Business
(SBA) and Social Security Administrations (SSA). In making this selection,
we chose agencies that collectively represented the full range of
characteristics in the following four areas: (1) agency size (small,
medium, large); (2) primary program types (direct service, research,
regulatory, transfer payments, and contracts or grants); (3) quality of
fiscal year 2000 performance plan based on our previous review (low,
medium, high);7 and (4) type of agency (cabinet department and independent
agency). Appendix I contains a more detailed discussion of our scope and
methodology.
We performed our work in Washington, D.C., from January through November
2003 in accordance with generally accepted government auditing standards.
Major contributors to this report are listed in appendix
XII.
7GAO/GGD/AIMD-99-215. Based on how we had rated agencies' annual
performance plans on their picture of performance, specificity of
strategies and resources, and the degree of confidence that performance
information will be credible, we assigned numeric values to each agencies'
rating (e.g., clear=3, general=2, limited=1, unclear=0) and added them up
to determine overall quality of high, medium, or low. An agency's plan was
considered high quality if its score was between 7-9, a score of 5-6 was
considered medium quality, and a score of 3-4 was low. No agencies
received a score lower than 3.
Chapter 2
GPRA Established the Foundation for a More Results-Oriented Federal
Government
Among the purposes of GPRA cited by Congress was to improve federal
program effectiveness and service delivery by promoting a new focus on
results, service quality, and customer satisfaction by setting program
goals, measuring performance against goals, and reporting publicly on
progress. Furthermore, GPRA was to improve congressional decision making
by providing better information on achieving objectives, and on the
relative effectiveness and efficiency of federal programs and spending.
Ten years after enactment, GPRA's requirements have laid a foundation of
resultsoriented agency planning, measurement, and reporting that have
begun to address these purposes. Focus group participants, high-level
political appointees, and OMB officials we interviewed cited positive
effects of GPRA that they generally attributed to GPRA's statutory
requirements for planning and reporting. Our survey results indicate that
since GPRA went into effect governmentwide in 1997, federal managers
reported having significantly more of the types of performance measures
called for by GPRA-particularly outcome-oriented performance measures.
GPRA has also begun to facilitate the linking of resources to results,
although much remains to be done in this area.
GPRA Statutory Requirements Laid a Foundation for Agencywide
Results-Oriented Management
Prior to enactment of GPRA, our 1992 review of the collection and use of
performance data by federal agencies revealed that, although many agencies
collected performance information at the program level, few agencies had
results-oriented performance information to manage or make strategic
policy decisions for the agency as a whole.1 Federal agencies surveyed
indicated that many had a single, long-term plan that contained goals,
standards, or objectives for the entire agency or program. Many of these
agencies also reported they collected a wide variety of performance
measures. However, in validating the survey responses with a sample of
agencies, we found that measures were typically generated and used by
program-level units within an agency and focused on measuring work
activity levels and outputs or compliance with statutes. Little of this
performance information was transparent to Congress, OMB, or the public
and few of the agencies we visited used performance measures to manage
toward long-term objectives. Few of the agencies surveyed had the
infrastructure in place, such as a unified strategic plan with measurable
goals, an office that collected performance measures, and regular
consolidated reports, to tie plans and measures.
1GAO/GGD-92-65.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
GPRA addressed these shortcomings by creating a comprehensive and
consistent statutory foundation of required agencywide strategic plans,
annual performance plans, and annual performance reports. In contrast to
prior federal government efforts to measure performance, GPRA explicitly
emphasized that, in addition to performance indicators that agencies may
need to manage programs on a day-to-day basis, such as quantity, quality,
timeliness, and cost, agencies also needed outcome-oriented goals and
measures that assess the actual results, effects, or impact of a program
or activity compared to its intended purpose.
Expert and agency focus group participants cited the creation of this
statutory foundation as one of the key accomplishments of GPRA.
Participants agreed that GPRA created a framework in statute for federal
agencies to plan their activities in order to become more results oriented
and provided a managerial tool for program accountability. Using this
framework, agencies could develop and focus on strategies to carry out the
programs they administer; set goals and identify performance indicators
that will inform them whether or not they achieved the performance they
expected; and determine what impact, if any, their programs have had on
the American public. According to the experts in one of our focus groups,
comparing federal agencies' current mission statements contained in their
strategic plans to what they were in the past demonstrates that the
agencies have done some "soul searching" to get a better sense of what
their role is (or should be) and how they can achieve it. Given that GPRA
is in statute, participants indicated that the use of this planning
framework is likely to be sustained within agencies.
One of the premises of GPRA is that both congressional and executive
branch oversight of federal agency performance were seriously hampered by
a lack of adequate results-oriented goals and performance information. As
noted above, prior to the enactment of GPRA few agencies reported their
performance information externally. OMB officials we interviewed as part
of our current review suggested that OMB has been a key consumer of the
performance information produced under GPRA and that it has provided a
foundation for their efforts to oversee agency performance.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
For example, during the development of the fiscal year 2004 budget, OMB
used PART to review and rate 234 federal programs. We recently reported
that one of PART's major impacts was its ability to highlight OMB's
recommended changes in program management and design.2 PART reviews look
at four elements-program purpose and design, strategic planning, program
management, and program results/accountability-and rate the program on how
well each of these elements is executed. However, without the foundation
of missions, goals, strategies, performance measures, and performance
information generated under GPRA, such oversight would be difficult to
carry out.
Participants in most of our focus groups also agreed that GPRA has been a
driving force behind many cultural changes that have occurred within
federal agencies. Highlighting the focus on results, participants stated
that GPRA had stimulated a problem-solving approach within federal
agencies and encouraged agency managers to think creatively when
developing performance indicators for their programs. GPRA has also
changed the dialogue within federal agencies; front-line managers and
staff at lower levels of the organization now discuss budget issues in
connection with performance. Similarly, experts noted that information
about performance management and resource investments are more frequently
communicated between agency officials and Congress than in the past.
Within agencies, GPRA documents can provide a context of missions, goals,
and strategies that political appointees can use to articulate agencies'
priorities.
Views on GPRA's Effect on the Federal Government's Ability to Deliver
Results to the American Public Were Mixed
A key purpose of GPRA was "to improve the confidence of the American
people in the capability of the Federal Government, by systematically
holding Federal agencies accountable for achieving program results." When
asked about the direct effects of GPRA on the public in our 2003 survey,
an estimated 23 percent of federal managers agreed to a moderate or
greater extent that GPRA improved their agency's ability to deliver
results to the American public; a larger percentage-38 percent-chose a "no
basis to judge/not applicable" category.
When a similar question was posed in our focus groups with experts and
federal managers, participants' views were generally mixed. Some federal
2U.S. General Accounting Office, Performance Budgeting: Observations on
the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004
Budget, GAO-04-174 (Washington, D.C.: Jan. 30, 2004).
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
managers in our focus groups agreed that GPRA has had a positive effect on
raising awareness on many performance issues, and that in and of itself is
a way of delivering results. The information gathered and reported for
GPRA allows agencies to make better-informed decisions, which improves
their ability to achieve results. Other participants stated that while
certain aspects of GPRA-related work have been positive, agencies' ability
to deliver results and public awareness of their activities cannot always
be exclusively attributed to GPRA. For example, some participants stated
that many agencies rely on grant recipients to carry out their work, and
delivering results to the American public depends, to a large extent, on
the diligence of these organizations to implement their programs; such
results would not change dramatically if GPRA were no longer a
requirement.
A number of the political appointees we interviewed cited examples of
outcomes they believe would not have occurred without the structure of
GPRA. For example, a former deputy secretary of the Department of Veterans
Affairs (VA) stated that "the Results Act brought about a fundamental
rethinking of how we managed our programs and processes. . . . We
developed a strategic plan that was veteranfocused. . . . We made every
effort to define program successes from the veteran's perspective." A
former Chief Financial Officer (CFO) cited Customs Service goals to reduce
the quantity of illegal drugs flowing into the United States and the Food
and Drug Administration's focus on speeding up the approval of new drugs
as examples of outcomes that can make a big difference in people's lives.
Another major accomplishment of GPRA cited by our focus group participants
is that GPRA improved the transparency of government results to the
American public. As noted above, prior to GPRA, few agencies reported
performance results outside of their agencies. Focus group participants
indicated a key accomplishment of GPRA was its value as a communication
tool by increasing the transparency to the public of what their agencies
did in terms the public could understand. For example, information on
agencies' strategic plans, performance goals, measures, and results are
easily obtainable from agency Web sites. One focus group participant
commented that GPRA helps bureaucrats explain to nonbureaucrats what the
federal government does in terms they can better understand. Other
comments indicated that because of GPRA agencies could now tell Congress
and the American public what they are getting for their money.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
More Managers Reported A fundamental element in an organization's efforts
to manage for results is Having Performance its ability to set meaningful
goals for performance and to measure Measures performance against those
goals. From our 2003 survey we estimate that
89 percent of federal managers overall said there were performance
measures for the programs they were involved with. This is a statistically
significantly higher percentage than the 76 percent of managers who
answered yes to this item on our 1997 survey. (See fig. 2.)
Figure 2: Percentage of Federal Managers Who Reported That There Were
Performance Measures for the Programs with Which They Were Involved
Percent
100
90 89
80
70
60
50
40
30
20
10
0 1997a 2000 2003a Year
Source: GAO. aThere was a statistically significant difference between
1997 and 2003 surveys.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
Moreover, when we asked managers who said they had performance measures
which of five types of measures they had to a great or very great extent,
they reported increases in all five types of measures between 1997 and
2003,3 all of which were statistically significant. (See fig. 3.) Notably,
managers indicated the existence of outcome measures, defined as
"performance measures that demonstrate to someone outside the organization
whether or not intended results are being achieved," grew from a low of 32
percent in 1997 to the current estimate of 55 percent, a level that is on
par with output measures for the first time since we began our survey.
Figure 3: Percentage of Federal Managers Who Reported Having Specific
Types of Performance Measures to a Great or Very Great Extent
Percent
60 54 55
50
40
30
20
10
0 Outputa Efficiencya Customer servicea Qualitya Outcomea
Performance measures
1997
2000
2003
Source: GAO.
aThere was a statistically significant difference between 1997 and 2003
surveys.
3Types of measures were defined in the questionnaire as follows:
performance measures that tell us how many things we produce or services
we provide (output measures); performance measures that tell us if we are
operating efficiently (efficiency measures); performance measures that
tell us whether or not we are satisfying our customers (customer service
measures); performance measures that tell us about the quality of the
products or services we provide (quality measures); and performance
measures that would demonstrate to someone outside of our agency whether
or not we are achieving our intended results (outcome measures).
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
Similarly, focus group participants commented on certain cultural changes
that had taken place within their agencies since the passage of GPRA in
which the "vocabulary" of performance planning and measurement-e.g., a
greater focus on performance management; orientation toward outcomes over
inputs and outputs; and an increased focus on program evaluation- had
become more pervasive. This perception is partly born out by our survey
results. Since 1997 those reporting a moderate to extensive knowledge of
GPRA and its requirements shifted significantly from 26 percent to 41
percent in 2003, while those reporting no knowledge of GPRA declined
significantly from 27 percent to 20 percent. (See fig. 4.)
Figure 4: Percentage of Federal Managers Who Reported Their Awareness of
GPRA Percent
100
90
80
70
60
50 41
40
30
20
10
0 1997a 2000 2003a
Year
Moderate to extensive knowledge
No knowledge Source: GAO.
aThere was a statistically significant difference between 1997 and 2003
surveys.
Consistent with our survey results indicating increases in
results-oriented performance measures and increasing GPRA knowledge, we
also observed a significant decline in the percentage of federal managers
who agreed that certain factors hindered measuring performance or using
the performance information. For example, as shown in figure 5, of those
who expressed an
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
opinion, the percentage of managers who noted that determining meaningful
measures was a hindrance to a great or very great extent was down
significantly from 47 percent in 1997 to 36 percent in 2003. Likewise, the
percentage that agreed to a great or very great extent that different
parties are using different definitions to measure performance was a
hindrance also declined significantly from 49 percent in 1997 to 36
percent in 2003.
Figure 5: Percentage of Federal Managers Who Reported Hindrances to
Measuring
Performance or Using the Performance Information to a Great or Very Great
Extent
Percent
60
50 49
40
30
20
10
0
Difficulty Different parties
are
determining using different
meaningful definitions to
measure
measuresa performancea
1997
2000
2003
Source: GAO.
Note: Percentages are based on those respondents answering on the extent
scale. aThere was a statistically significant difference between 1997 and
2003.
Finally, our survey data suggested that more federal managers, especially
at the Senior Executive Service (SES) level, believed that OMB was paying
attention to their agencies' efforts under GPRA. Moreover, there was no
corresponding increase in their concern that OMB would micromanage the
programs in their agencies. In our survey, we asked respondents to assess
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
the extent to which OMB pays attention to their agencies' efforts under
GPRA. As seen in figure 6, in 2003, the percentage of respondents who
responded "Great" or "Very Great" to this question (31 percent) was
significantly higher than in 2000 (22 percent). Of those, SES respondents
showed an even more dramatic increase, from 33 to 51 percent. We also
asked respondents the extent to which their concern that OMB would
micromanage programs in their agencies was a hindrance to measuring
performance or using performance information. The percentage among those
expressing an opinion that it was a hindrance to a great or very great
extent was low-around 24 percent in 2003-with no significant difference
between 2000 and 2003.
Figure 6: Percentage of Federal Managers and SES Managers Who Reported
That OMB Paid Attention to Their Agency's Efforts under GPRA to a Great or
Very Great Extent
Percent
100
90
80
70
60
51
50
40
30
20 22
10
0 2000a 2003a
Year
Federal managers SES Non-SES Source: GAO.
aThere was a statistically significant difference between 2000 and 2003
surveys.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
GPRA Has Begun to Establish a Link between Resources and Results
Among its major purposes, GPRA aims for a closer and clearer linkage
between requested resources and expected results. The general concept of
linking performance information with budget requests is commonly known as
performance budgeting. Budgeting is and will remain an exercise in
political choice, in which performance can be one, but not necessarily the
only, factor underlying decisions. However, efforts to infuse performance
information into resource allocation decisions can more explicitly inform
budget discussions and focus them-both in Congress and in agencies-on
expected results, rather than on inputs.
GPRA established a basic foundation for performance budgeting by requiring
that an agency's annual performance plan cover each program activity in
the President's budget request for that agency. GPRA does not specify any
level of detail or required components needed to achieve this coverage.
Further, GPRA recognizes that agencies' program activity structures are
often inconsistent across budget accounts and thus gives agencies the
flexibility to consolidate, aggregate, or disaggregate program activities,
so long as no major function or operation of the agency is omitted or
minimized. In addition, OMB guidance has traditionally required agencies
to display, by budget program activity, the funding level being applied to
achieve performance goals. OMB's guidance on developing fiscal year 2005
performance budgets also encourages a greater link between performance and
funding levels, however, it places greater emphasis on linking agencies'
long-term and annual performance goals to individual programs. At a
minimum, agencies are to align resources at the program level, but they
are encouraged to align resources at the performance goal level. Resources
requested for each program are to be the amounts needed to achieve program
performance goal targets.
Our 1998 assessment of fiscal year 1999 performance plans found that
agencies generally covered the program activities in their budgets, but
most plans did not identify how the funding for those program activities
would be allocated to performance goals.4 However, our subsequent reviews
of performance plans indicate that agencies have made progress in
demonstrating how their performance goals and objectives relate to program
activities in the budget.
4GAO/GGD/AIMD-98-228.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
Over the first 4 years of agency efforts to implement GPRA, we observed
that agencies continued to tighten the required link between their
performance plans and budget requests.5 Of the agencies we reviewed over
this period, all but three met the basic requirement of GPRA to define a
link between their performance plans and the program activities in their
budget requests, and most of the agencies in our review had moved beyond
this basic requirement to indicate some level of funding associated with
expected performance described in the plan. Most importantly, more of the
agencies we reviewed each year-almost 75 percent in fiscal year 2002
compared to 40 percent in fiscal year 1999-were able to show a direct link
between expected performance and requested program activity funding
levels-the first step in defining the performance consequences of
budgetary decisions. However, we have also observed that the nature of
these linkages varied considerably. Most of the agencies in our review of
fiscal year 2002 performance plans associated funding requests with
higher, more general levels of expected performance, rather than the more
detailed "performance goals or sets of performance goals" suggested in OMB
guidance.
Although not cited by our group of experts, participants at six of our
seven focus groups with federal managers cited progress in this area as a
key accomplishment of GPRA. However, the participants also commented that
much remains to be done in this area. The comments ranged from the
general-GPRA provides a framework for planning and budgeting, to the more
specific-GPRA created a definition of programs and how they will help the
agency achieve its goals/objectives and the amount of money that will be
required to achieve said goals/objectives. One of the comments implied
that GPRA has helped to prioritize agency efforts by helping agencies
align their efforts with programs or activities that make a difference. A
political appointee we interviewed echoed this comment, stating that GPRA
was pushing the department to think about what it gets out of the budget,
not just what it puts into it-12 to 15 years ago the "so what" was missing
from the budget process. Another political appointee we interviewed stated
that the department was in the process of tying its goals to its budget
formulation and execution processes and linking program costs to
departmental goals. A former political appointee
5GAO-02-236; U.S. General Accounting Office, Performance Budgeting:
Initial Experiences Under the Results Act in Linking Plans With Budgets,
GAO/AIMD/GGD-99-67 (Washington, D.C.: Apr. 12, 1999); and Performance
Budgeting: Fiscal Year 2000 Progress in Linking Plans With Budgets,
GAO/AIMD-99-239R (Washington, D.C.: July 30, 1999).
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
discussed how his department used program performance information to
inform a major information systems investment decision.
Furthermore, GAO case studies on the integration of performance
information in budget decision making found that performance information
has been used to inform the allocation of resources and for other
management purposes at selected agencies. For example, the Veterans Health
Administration provides its health care networks with performance
information on patterns of patient care and patient health outcomes, which
can be used to analyze resource allocation and costs and reallocate
resources as appropriate.6 Officials at the Administration for Children
and Families said that training and technical assistance and salaries and
expense funds are often allocated based on program and performance needs.7
The Nuclear Regulatory Commission monitors performance against targets and
makes resource adjustments, if needed, to achieve those targets.8
Although there has been progress in formally establishing the linkages
between budgets and plans, our survey results are somewhat conflicting and
have not reflected any notable changes either in managers' perceptions
governmentwide as to their personal use of plans or performance
information when allocating resources, or in their perceptions about the
use of performance information when funding decisions are made about their
programs. Our 2003 survey data show that a large majority of federal
managers reported that they consider their agency's strategic goals when
they are allocating resources. As shown in figure 7, on our 2003 survey,
an estimated 70 percent of all federal managers agreed to a great or very
great extent that they considered their agency's strategic goals when
allocating resources. However, using our 1997 survey responses as a
baseline, it was not a statistically significant increase over 64 percent
of the managers who responded comparably then. As shown in figure 8, a
similar, but somewhat
6U.S. General Accounting Office, Managing for Results: Efforts to
Strengthen the Link Between Resources and Results at the Veterans Health
Administration, GAO-03-10 (Washington, D.C.: Dec. 10, 2002).
7U.S. General Accounting Office, Managing for Results: Efforts to
Strengthen the Link Between Resources and Results at the Administration
for Children and Families, GAO-0309 (Washington, D.C.: Dec. 10, 2002).
8U.S. General Accounting Office, Managing for Results: Efforts to
Strengthen the Link Between Resources and Results at the Nuclear
Regulatory Commission, GAO-03-258 (Washington, D.C.: Dec. 10, 2002).
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
smaller, majority (60 percent) of managers who expressed an opinion on our
2003 survey agreed to a great or very great extent that they used
information from performance measurement when they were involved in
allocating resources. In 1997, the comparable response was about the same
at 62 percent. When we asked managers on another item, however, about the
extent to which they perceived funding decisions for their programs being
based on results or outcome-oriented performance information, only 25
percent of federal managers in 2003 endorsed this view to a great or very
great extent. In 1997, 20 percent of managers expressed a comparable view,
again not a significant increase. (See fig. 9.)
Figure 7: Percentage of Federal Managers Who Reported They Considered
Strategic
Goals to a Great or Very Great Extent When Allocating Resources
Percent
100
90
80
70
70
60
50
40
30
20
10
0 1997 2000 2003
Year
Source: GAO.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
Figure 8: Percentage of Federal Managers Who Reported They Considered
Performance Information to a Great or Very Great Extent When Allocating
Resources Percent
100
90
80
70
62 60
60
50
40
30
20
10
0
1997 2000 2003
Year
Source: GAO. Note: Percentages are based on those respondents answering on
the extent scale.
Chapter 2
GPRA Established the Foundation for a More
Results-Oriented Federal Government
Figure 9: Percentage of Federal Managers Who Reported That Funding
Decisions Were Based on Results or Outcome-Oriented Performance
Information to a Great or Very Great Extent
Percent
100
90
80
70
60
50
40
30 25
20
10
0
1997 2000 2003
Year
Source: GAO.
Chapter 3
Agencies Have Addressed Many Critical Performance Planning and Reporting
Challenges, but Weaknesses Persist
Beginning with federal agencies' initial efforts to develop effective
strategic plans in 1997 and annual performance plans and reports for
fiscal year 1999, Congress, GAO, and others have commented on the quality
of those efforts and provided constructive feedback on how agency plans
and reports could be improved. On the basis of our current review of the
strategic plans, annual performance plans, and annual performance reports
of six selected agencies-Education, DOE, HUD, DOT, SBA, and SSA-we found
that these documents reflect much of the feedback that was provided. For
example, goals were more quantifiable and results oriented, and agencies
were providing more information about goals and strategies to address
performance and accountability challenges and the limitations to their
performance data. However, certain weaknesses, such as lack of detail on
how annual performance goals relate to strategic goals and how agencies
are coordinating with other entities to achieve common objectives,
persist. A detailed discussion of our scope and methodology and the
results of our reviews of the six agencies' most recent strategic plans,
annual performance plans, and annual performance reports compared to
initial efforts are contained in appendixes III, IV, and V, respectively.
Quality of Selected Strategic Plans Reflects Improvements over Initial
Drafts
Under GPRA, strategic plans are the starting point and basic underpinning
for results-oriented management. GPRA requires that an agency's strategic
plan contain six key elements: (1) a comprehensive agency mission
statement; (2) agencywide long-term goals and objectives for all major
functions and operations; (3) approaches (or strategies) and the various
resources needed to achieve the goals and objectives; (4) a description of
the relationship between the long-term goals and objectives and the annual
performance goals; (5) an identification of key factors, external to the
agency and beyond its control, that could significantly affect the
achievement of the strategic goals; and (6) a description of how program
evaluations were used to establish or revise strategic goals and a
schedule for future program evaluations.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
Our 1997 review of agencies' draft strategic plans found that a
significant amount of work remained to be done by executive branch
agencies if their strategic plans were to fulfill the requirements of
GPRA, serve as a basis for guiding agencies, and help congressional and
other policymakers make decisions about activities and programs.1 Our
assessment of 27 agencies' initial draft strategic plans revealed several
critical strategic planning issues that needed to be addressed. These
planning issues were as follows:
o Most of the draft plans did not adequately link required elements in
the plans, such as strategic goals to annual performance goals.
o Long-term strategic goals often tended to have weaknesses.
o Many agencies did not fully develop strategies explaining how their
long-term strategic goals would be achieved.
o Most agencies did not reflect in their draft plans the identification
and planned coordination of activities and programs that cut across
multiple agencies.
o The draft strategic plans did not adequately address program
evaluations.
We noted that Congress anticipated that it may take several planning
cycles to perfect the process and that strategic plans would be
continually refined as various planning cycles occur. We also recognized
that developing a strategic plan is a dynamic process and that agencies,
with input from OMB and Congress, were continuing to improve their plans.
Agencies have now had 6 years to refine their strategic planning
processes. Although the six strategic plans we looked at for this review
reflected many new and continuing strengths as well as improvements over
the 1997 initial drafts, we continued to find certain persistent
weaknesses. As depicted in table 1, of the six elements required by GPRA,
the plans generally discussed all but one-program evaluation, an area in
which we have found capacity is often lacking in federal agencies.2
Although the strategic plans generally
1GAO/GGD-97-180.
2U.S. General Accounting Office, Managing for Results: Challenges Agencies
Face in Producing Credible Performance Information, GAO/GGD-00-52
(Washington, D.C.: Feb. 4, 2000).
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
listed the program evaluations agencies planned to complete over the
planning period, they generally did not address how the agencies planned
to use their evaluations to establish new or revise existing strategic
goals, as envisioned by GPRA. Finally, although not required by GPRA, the
strategic plans would have benefited from more complete discussions of how
agencies planned to coordinate with other entities to address common
challenges or achieve common or complementary goals. Appendix III provides
a more detailed discussion of (1) the required and other useful elements
we reviewed to assess strategic plan strengths and weaknesses and (2)
changes in the quality of the six agencies' strategic plans we reviewed.
Table 1: Agencies' Progress in Addressing Required Elements of Strategic
Planning under GPRA Element included in agency strategic plan? Agency strategic
plans Plan year
Mission statement Long-term goals Strategies Relationship between longterm
goals and annual goals External factors Evaluations
Department of Education 1997 X X X X X
2002XXX X X
Department of Energy 1997 X X X
2003a X X X X X
Department of Housing and Urban 1997 X
Development 2003 X X X X X
Small Business Administration 1997 X X X X
2001b X X X X X
Social Security Administration 1997 X X X X X X
2003 X X X X X X
Department of Transportation 1997 X X X
2003a X X X X X
Sources: GAO analysis of agencies' strategic plans in effect at the time
of our review. See also, U.S. General Accounting Office, The Results Act:
Observations on the Department of Education's June 1997 Draft Strategic
Plan, GAO/HEHS-97-176R (Washington, D.C.: July 18, 1997); Results Act:
Observations on DOE's Draft Strategic Plan, GAO/RCED-97-199R (Washington,
D.C.: July 11, 1997); The Results Act: Observations on the Department of
Transportation's Draft Strategic Plan, GAO/RCED-97-208R (Washington, D.C.:
July 30, 1997); The Results Act: Observations on the Social Security
Administration's June 1997 Draft Strategic Plan, GAO/HEHS-97-179R
(Washington, D.C.: July 22, 1997); The Results Act: Observations on the
Small Business Administration's Draft Strategic Plan, GAO/RCED-97-205R
(Washington, D.C.: July 11, 1997); and The Results Act: Observations on
the Department of Housing and Urban Development's Draft Strategic Plan,
GAO/RCED-97-224R (Washington, D.C.: Aug. 8, 1997).
aThe 2003 plans for DOE and DOT were in draft form during the time of our
review.
bAt the time of our review, the most recent SBA strategic plan was for
fiscal years 2001-2008. SBA released a new strategic plan for fiscal years
2003-2008 in October 2003.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
Strategic Planning Strengths and Improvements from Initial Draft Plans
Consistent with our review of agencies' 1997 strategic plans, the recent
strategic plans we reviewed generally contained mission statements that
were results oriented, distinct from other agencies, and covered the
agencies' major activities. DOT's mission statement had improved by
reflecting additional language from its enabling legislation that we
recommended adding during our 1997 review. Still improvement could be made
in this area as is shown by DOE's mission statement. DOE's mission was
results oriented but did not address the department's activities related
to energy supply and conservation.
Our review of the current strategic plans also revealed improvements in
the development of agencies' long-term, strategic goals-essential for
resultsoriented management. Although GPRA does not require that all of an
agency's long-term, strategic goals be results oriented, the intent of
GPRA is to have agencies focus their strategic goals on results to the
extent feasible. In addition, as required by GPRA, the goals should be
expressed in a manner that could be used to gauge success in the future
and should cover an agency's major functions or activities. All of the
strategic plans we reviewed contained long-term, strategic goals that
demonstrated improvements in the quality of their 1997 goals. Agencies'
long-term strategic goals generally covered their missions, were results
oriented, and were expressed in a manner that could be used to gauge
future success. For example, SBA improved the quality of its long-term
goals by focusing more on key outcomes to be achieved and less on process
improvements, as was the case in its 1997 plan. In some cases, we observed
strategic goals that addressed the agency's organizational capacity to
achieve results, such as SSA's long-term goal to strategically manage and
align staff to support its mission.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
We also found improvements in how agencies' current plans addressed
performance and accountability challenges we had identified, a key
weakness we identified in our earlier review. Each of the agency plans we
reviewed discussed the long-term goals and strategies to address the
challenges that we had identified. For example, Education's strategic plan
contained a long-term strategic goal to modernize the Federal Student
Assistance programs and address identified problems in this area, which we
have designated as high risk since 1990.3 SSA noted that it considered
GAO-identified performance and accountability challenges when it
determined its strategic goals and objectives, however not all of the
challenges are clearly addressed in the plan.
A third area of improvement we observed was in the description of the
strategies agencies planned to use to achieve their long-term strategic
goals. In our review of agencies' 1997 draft strategic plans, we found
that many agencies did not fully develop strategies explaining how their
longterm strategic goals would be achieved. In contrast, all six of the
current strategic plans we reviewed contained strategies that appeared
logically linked to achieving the agencies' long-term goals.
Other strengths and improvements we observed in meeting GPRA's basic
requirements involved the reporting of external factors that could affect
the achievement of the long-term goals and the identification of
crosscutting activities, although as indicated below these discussions
could be improved. The six agencies reviewed for this report each reported
on external factors in current strategic plans. For example, for each of
the strategic objectives in DOT's strategic plan, DOT lists factors
external to its control and how those factors could affect the achievement
of its objectives. Although not a requirement, some of the better plans we
reviewed discussed strategies to ameliorate the effect of external
factors. For example, for an external factor on teacher certification
under a goal on reading, Education's plan states that the agency "will
work with the states and national accreditation bodies to encourage the
incorporation of research-based reading instruction into teacher
certification requirements."
3Since 1990, GAO has periodically reported on government operations that
it identifies as "high risk" because of the greater vulnerabilities to
fraud, waste, abuse, and mismanagement. See U.S. General Accounting
Office, High-Risk Series: An Update, GAO03-119 (Washington, D.C.: January
2003).
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
We have frequently reported that a focus on results, as envisioned by
GPRA, implies that federal programs contributing to the same or similar
results should be closely coordinated to ensure that goals are consistent
and, as appropriate, program efforts are mutually reinforcing. This means
that federal agencies are to look beyond their organizational boundaries
and coordinate with other agencies to ensure that their efforts are
aligned. During our 1997 review, we found that most agencies did not
reflect in their draft plans the identification and planned coordination
of activities and programs that cut across multiple agencies. In contrast,
each of the six current agency strategic plans that we reviewed identified
at least some activities and programs that cut across multiple agencies.
For example, SBA's 1997 plan contained no evidence of how the agency
coordinated with other agencies, but the current plan contained a separate
section describing crosscutting issues in the areas of innovation and
research assistance, international trade assistance, business development
assistance, veterans affairs, and disaster assistance.
Critical Strategic Planning Issues Needing Further Improvement
First, consistent with our 1997 review, the strategic plans we reviewed
did not adequately link required elements in the plans. Although all of
the agencies we reviewed provided some information on the relationship
between their long-term and annual goals, the extent of information
provided on how annual goals would be used to measure progress in
achieving the long-term goals varied greatly. In the case of DOE, the plan
provides a very brief description of the overall relationship between its
long-term and annual goals with examples, but does not demonstrate how it
will assess progress for each of its long-term goals and objectives.
Another plan, DOT's, refers the reader to the annual performance plan for
information about annual goals. We have reported that this linkage is
critical for determining whether an agency has a clear sense of how it
will assess progress toward achieving its intended results.
Second, although the agencies' descriptions of their strategies had
improved since our initial reviews, with few exceptions, their strategies
generally did not include information on how the agencies plan to align
their activities, core processes, human capital, and other resources to
support their mission-critical outcomes and whether they have the right
mix of activities, skills, and resources to achieve their goals. Such
information is critical to understanding the viability of the strategies.
Furthermore, none of the agencies discussed alternative strategies they
had considered in developing their plans. Without such discussions, it is
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
unclear whether agency planning processes were truly strategic or simply a
recasting of existing activities, processes, etc.
HUD was the only agency that provided any details of how it intended to
coordinate with other agencies to achieve common or complementary goals
for its crosscutting programs or activities. For example, to support its
goal of "Equal Opportunity in Housing," HUD's plan states that HUD and the
Department of Justice continue to coordinate their fair housing
enforcement activities, especially with respect to responding quickly and
effectively to Fair Housing Act complaints that involve criminal activity
(e.g., hate crimes), a pattern and practice of housing discrimination, or
the legality of state and local zoning or other land use laws or
ordinances. We have reported that mission fragmentation and program
overlap are widespread throughout the federal government.4 As such,
interagency coordination is important for ensuring that crosscutting
programs are mutually reinforcing and efficiently implemented.
Finally, the draft strategic plans did not adequately address program
evaluations. In combination with an agency's performance measurement
system, program evaluations can provide feedback to the agency on how well
its activities and programs contributed to achieving strategic goals. For
example, evaluations can be a potentially critical source of information
for Congress and others in assessing (1) the appropriateness and
reasonableness of goals; (2) the effectiveness of strategies by
supplementing performance measurement data with impact evaluation studies;
and (3) the implementation of programs, such as identifying the need for
corrective action. Evaluations are important because they potentially can
be critical sources of information for ensuring that goals are reasonable,
strategies for achieving goals are effective, and that corrective actions
are taken in program implementation. Five out of the six current plans
that we reviewed included a discussion of program evaluations, however for
most of these plans the discussions lacked critical information required
by GPRA, such as a discussion of how evaluations were used to establish
strategic goals or a schedule of future evaluations. For example, DOE's
plan stated that internal, GAO, and Inspector General (IG) evaluations
were used as resources to develop its draft strategic plan, but specific
program evaluations were not identified.
4U.S. General Accounting Office, Managing for Results: Using the Results
Act to Address Mission Fragmentation and Program Overlap, GAO/AIMD-97-146
(Washington, D.C.: Aug. 29, 1997).
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
Fiscal Year 2004 Annual Performance Plans Addressed Some Weaknesses of
Earlier Plans, but Still Have Room for Significant Improvement
According to our review of agencies' first annual performance plans, which
presented agencies' annual performance goals for fiscal year 1999,5 we
found that substantial further development was needed for these plans to
be useful in a significant way to congressional and other decision makers.
Most of the fiscal year 1999 plans that we reviewed contained major
weaknesses that undermined their usefulness in that they (1) did not
consistently provide clear pictures of agencies' intended performance, (2)
generally did not relate strategies and resources to performance, and (3)
provided limited confidence that agencies' performance data will be
sufficiently credible. Although all of the fiscal year 1999 plans
contained valuable information for decision makers, their weaknesses
caused their usefulness to vary considerably within and among plans.
As shown in table 2, our current review of agencies' fiscal year 2004
performance plans found that five agencies-Education, HUD, SBA, SSA, and
DOT-improved their efforts to provide a clear picture of intended
performance, with SSA and DOT being the clearest. Furthermore, the same
five agencies improved the specificity of the strategies and resources
they intended to use to achieve their performance goals, with DOT being
the most specific. Finally, the same five agencies-Education, HUD, SBA,
SSA, and DOT-made improvements in the area of greatest weakness- reporting
on how they will ensure performance data will be credible. However, only
DOT's plan provided a full level of confidence that the performance data
the agency intended to collect would be credible. Appendix IV provides a
more detailed discussion of (1) the required and other useful elements we
reviewed to assess the clarity of the picture of intended performance, the
specificity of the strategies and resources, and the level of confidence
in the performance data and (2) changes in the quality of the six
agencies' annual performance plans we reviewed.
5GAO/GGD/AIMD-98-228.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
Table 2: Characterizations of Agencies' Fiscal Year 1999 and 2004 Annual
Performance Plans
Picture of intended performance (unclear, limited, general, clear) Strategies
and resources (no, limited, general, specific) Data credible (no, limited,
general, full)
Agency 1999 2004 1999 2004 1999 2004
Department of Limited General Limited General Limited General
Education
Department of Limited Limited General General Limited Limited
Energy
Department of Limited General Limited General Limited General
Housing
and Urban
Development
Small Business Limited General Limited General Limited General
Administration
Social Security Limited Clear Limited General No General
Administration
Department of General Clear General Specific Limited Full
Transportation
Sources: GAO analysis of agencies' fiscal year 2004 annual performance
plans and U.S. General Accounting Office, Results Act: Observations on the
Department of Education's Fiscal Year 1999 Annual Performance Plan,
GAO/HEHS-98-172R (Washington, D.C.: June 8, 1998); Results Act:
Observations on DOE's Annual Performance Plan for Fiscal Year 1999,
GAO/RCED-98-194R (Washington, D.C.: May 28, 1998); Results Act:
Observations on the Department of Housing and Urban Development's Fiscal
Year 1999 Annual Performance Plan, GAO/RCED-98-159R (Washington, D.C.:
June 5, 1998); Results Act: Observations on the Small Business
Administration's Fiscal Year 1999 Annual Performance Plan,
GAO/RCED-98-200R (Washington, D.C.: May 28, 1998); The Results Act:
Observations on the Social Security Administration's Fiscal Year 1999
Annual Performance Plan, GAO/HEHS-98-178R (Washington, D.C.: June 9,
1998); and Results Act: Observations on the Department of Transportation's
Annual Performance Plan for Fiscal Year 1999, GAO/RCED-98-180R
(Washington, D.C.: May 12, 1998).
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
Plans Generally Provided a Clearer Picture of Intended Performance, Except
for Crosscutting Areas
At the most basic level, an annual performance plan is to provide a clear
picture of intended performance across the agency. Such information is
important to Congress, agency managers, and others for understanding what
the agency is trying to achieve, identifying subsequent opportunities for
improvement, and assigning accountability. Our current review of agencies'
fiscal year 2004 performance plans found that five of the six agencies
provided a clearer picture of intended performance than their fiscal year
1999 plans did, although only two of the 2004 plans-DOT's and
SSA's-received the highest rating possible. As shown in table 2, except
for DOT, the six agencies we reviewed for this report initially provided a
limited picture of intended performance. Most of the fiscal year 1999
performance plans we previously reviewed had at least some objective,
quantifiable, and measurable goals, but few plans consistently included a
comprehensive set of goals that focused on the results that programs were
intended to achieve. Moreover, agencies did not consistently follow OMB's
guidance that goals for performance and accountability challenges be
included in the plans. Agencies' plans generally showed how their missions
and strategic goals were related to their annual performance goals and
covered all of the program activities in the agencies' budget requests.6
In addition, many agencies took the needed first step of identifying their
crosscutting efforts, with some including helpful lists of other agencies
with which they shared a responsibility for addressing similar national
issues. However, the plans generally did not go further to describe how
agencies expected to coordinate their efforts with other agencies.
The fiscal year 2004 plans improved the picture of performance by making
annual goals and performance measures more results oriented, objective,
and quantifiable. For example, Education's plan included a measure for the
number of states meeting their eighth grade mathematics achievement
targets under the long-term goal to improve mathematics and science
achievement for all students. We previously criticized Education's 1999
plan for lacking such outcome-oriented measures. Another overall
improvement we observed was that all of the plans described intended
efforts to address performance and accountability challenges we and others
had previously identified. For instance, to address the governmentwide
high-risk area of strategic human capital management,
6Program activity refers to the list of projects and activities in the
appendix portion of the Budget of the United States Government. Program
activity structures are intended to provide a meaningful representation of
the operations financed by a specific budget account.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
HUD states that to develop its staff capacity, it will complete a
comprehensive workforce analysis in 2004 to serve as the basis to fill
mission critical skill gaps through succession planning, hiring, and
training initiatives in a 5-year human capital management strategy. The
clarity of DOE's plan remained limited because its annual goals were not
clearly linked to its mission, the long-term goals in its strategic plan,
or the program activities in its budget request.
Although five of the six agencies improved the clarity of the picture of
intended performance, improvement is still needed in reporting on
crosscutting efforts. In both the 1999 and 2004 plans, many agencies
identified their crosscutting efforts, with some including helpful lists
of other agencies with which they shared a responsibility for addressing
similar national issues. Our review of fiscal year 2004 plans shows that
the six agencies we reviewed still did not discuss how they expected to
coordinate with other agencies to address common challenges or to achieve
common or complementary performance goals. As we have reported previously,
improved reporting on crosscutting efforts can help Congress use the
annual performance plan to evaluate whether the annual goals will put the
agency on a path toward achieving its mission and longterm strategic
goals. In addition, the plans can aid in determining efforts to reduce
significant program overlap and fragmentation that can waste scarce
resources, confuse and frustrate program customers, and limit overall
program effectiveness.
None of the six agencies' plans indicated an intention to request waivers
of specific administrative procedural requirements and controls that may
be impeding an agencies' ability to achieve results. This provision of
GPRA allows agencies greater managerial flexibility in exchange for
accountability for results. We previously reported on the results of the
pilot project to implement this provision of GPRA and found that the pilot
did not work as intended.7 OMB did not designate any of the seven
departments and one independent agency that submitted a total of 61 waiver
proposals as GPRA managerial accountability and flexibility pilots. For
about three-quarters of the waiver proposals, OMB or other central
management agencies determined that the waivers were not allowable for
statutory or other reasons or that the requirement for which the waivers
were proposed no longer existed. For the remaining proposals, OMB or other
central management agencies approved waivers or developed
7GAO/GGD-97-36.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
compromises by using authorities that were already available independent
of GPRA.
Plans More Specifically Related Strategies and Resources to Performance
Goals
To judge the reasonableness of an agency's proposed strategies and
resources, congressional and other decision makers need complete
information on how the proposed strategies and resources will contribute
to the achievement of agency goals. Agencies generally improved their
plans by better relating strategies and resources to performance.
Education's, HUD's, SBA's, and SSA's 1999 plans had a limited discussion,
while DOE's and DOT's 1999 plans had a general discussion. In 2004, five
of the six plans-Education's, DOE's, HUD's, SBA's, and SSA's-provided
general discussions of how their strategies and resources would contribute
to achieving their performance goals. DOT's 2004 plan improved to include
a specific discussion.
Our review of the 1999 plans found that most agencies' performance plans
did not provide clear strategies that described how performance goals
would be achieved. In contrast, the 2004 performance plans we reviewed
generally provided lists of the agencies' current array of programs and
initiatives. Several plans provided a perspective on how these programs
and initiatives were necessary or helpful for achieving results. For
example, DOE and HUD included in their plans a "means and strategies"
section for each of their goals that described how the goal would be
achieved. One strategy DOE identified to meet its goal of contributing
unique, vital facilities to the biological environmental sciences was to
conduct peer reviews of the facilities to assess the scientific output,
user satisfaction, the overall cost-effectiveness of each facility's
operations, and their ability to deliver the most advanced scientific
capability.
In addition, each of the agencies' plans identified the external factors
that could influence the degree to which goals are achieved. Some of the
better plans, such as DOT's and SBA's, provided strategies to mitigate the
negative factors or take advantage of positive factors, as appropriate.
For example, for its transportation accessibility goals, DOT's plan states
that as the population ages, more people will require accessible public
transit, for which states and local agencies decide how best to allocate
federally provided resources. One of the strategies DOT intends to employ
to address this external factor is the "Special Needs of Elderly
Individuals and Individuals with Disabilities" grant program. The plan
states the grant program will help meet transportation needs of the
elderly and persons
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
with disabilities when regular transportation services are unavailable,
insufficient, or inappropriate to meet their needs.
Agencies' 2004 plans did not consistently describe all the resources
needed and how they would be used to achieve agency goals. Our review of
agencies' fiscal year 1999 plans found that most did not adequately
describe-or reference other appropriate documents that describe-the
capital, human, information, and financial resources needed to achieve
their agencies' performance goals. The 2004 plans we reviewed generally
described the funding levels needed to achieve their performance goals
overall and in some cases broke out funding needs by high-level
performance goal. For example, SSA's plan provides a general perspective
on the financial resources needed to achieve its performance goals because
it provides budget information by account and program activity. However,
the plan is neither structured by budget program activity or account, nor
does it provide a crosswalk between the strategic goals and budget program
accounts. In contrast, HUD's plan presented its requested funding and
staffing levels at the strategic goal level, but did not present budget
information at the level of its annual goals. In addition, although the
plans make brief mention of nonfinancial resources, such as human capital,
information technology, or other capital investments, little information
is provided on how such resources would be used to achieve performance
goals.
Plans Continue to Provide Less Than Full Confidence That Performance Data
Will Be Credible
Credible performance information is essential for accurately assessing
agencies' progress towards the achievement of their goals and, in cases
where goals are not met, identifying opportunities for improvement or
whether goals need to be adjusted. Under GPRA, agencies' annual
performance plans are to describe the means that will be used to verify
and validate performance data. To help improve the quality of agencies'
performance data, Congress amended GPRA through the Reports Consolidation
Act of 2000 to require that agencies assess the completeness and
reliability of the performance data in their performance reports. Agencies
were also required to discuss in their report any material inadequacies in
the completeness and reliability of their performance data and discuss
actions to address these inadequacies. Meeting these new requirements
suggests the need for careful planning to ensure that agencies can comment
accurately on the quality of the performance data they report to the
public.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
As shown in table 2, although five of the six agencies we reviewed
improved in reporting how they plan to ensure that performance data will
be credible, only one agency-DOT-improved enough over its 1999 plan to
provide a full level of confidence in the credibility of its performance
data. Four agencies-Education, HUD, SBA, and SSA-improved enough to
provide a general level of confidence. However, DOE provided the same
limited level of confidence in the credibility of the performance data as
in its 1999 plan. Regarding all 24 of the fiscal year 1999 performance
plans we reviewed, we found most provided only superficial descriptions of
procedures that agencies intended to use to verify and validate
performance data. Moreover, in general, agencies' performance plans did
not include discussions of documented limitations in financial and other
information systems that may undermine efforts to produce high-quality
data. As we have previously noted, without such information, and
strategies to address those limitations, Congress and other decision
makers cannot assess the validity and reliability of performance
information.
We found that each of the 2004 plans we reviewed contained some discussion
of the procedures the agencies would use to verify and validate
performance information, although in some cases the discussion was
inconsistent or limited. For example, the discussions of SBA's
verification and validation processes for its indicators in the 2004 plan
were generally one- or two-sentence statements. SBA also noted that it
does not independently verify some of the external data it gathers or that
it does not have access to the data for this purpose. In contrast, the DOT
plan referred to a separate compendium available on-line that provides
source and accuracy statements, which give more detail on the methods used
to collect performance data, sources of variation and bias in the data,
and methods used to verify and validate the data.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
In addition, all of the agencies except DOE discussed known limitations to
performance data in their plans. These agencies' plans generally provided
information about the quality of each performance measure, including any
limitations. According to DOE officials, DOE's plan generally does not
discuss data limitations because the department selected goals for which
data are expected to be available and therefore did not anticipate finding
any limitations. However, in our 2003 Performance and Accountability
Series report on DOE, we identified several performance and accountability
challenges where data were a concern, such as the need for additional
information on the results of contractors' performance to keep projects on
schedule and within budget.8 DOE's contract management continues to be a
significant challenge for the department and remains at high risk.
Finally, the remaining five agencies also discussed plans to address
limitations to the performance data. For example, DOT's plan provided a
general discussion of the limitations to the internal and external sources
of data used to measure performance. Detailed discussions were contained
in an appendix to the plan and separate source and accuracy statements.
This information had been lacking in its 1999 plan. Education, HUD, SBA,
and SSA also provided information on limitations to their performance data
and plans for improvement.
Strengths and Weaknesses of Selected Agencies' Fiscal Year 2002 Annual
Performance Reports
Key to improving accountability for results as Congress intended under
GPRA, annual performance reports are to document the results agencies have
achieved compared to the goals they established. To be useful for
oversight and accountability purposes, the reports should clearly
communicate performance results, provide explanations for any unmet goals
as well as actions needed to address them, and discuss known data
limitations as well as how the limitations are to be addressed in the
future. Compared to our reviews of the six agencies' fiscal year 1999
performance reports, we identified a number of strengths and improvements
as well as areas that continued to need improvement. Because the scope of
our review of the fiscal year 2002 reports was broader than that for the
fiscal year 1999 reports we previously reviewed, we were unable to make
specific comparisons for the three characteristics we used to assess the
fiscal year 2002 reports. However, we discuss comparative information on
aspects of
8U.S. General Accounting Office, Major Management Challenges and Program
Risks: Department of Energy, GAO-03-100 (Washington, D.C.: January 2003).
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
the reports where available. Table 3 shows the results of our assessment
of the six agencies' annual performance reports for fiscal year 2002.
Appendix V provides a more detailed discussion of (1) the required and
other useful elements we reviewed to assess the clarity of the picture of
performance, the clarity of the linkage between costs and performance, and
the level of confidence in the performance data and (2) changes in the
quality of the six agencies' annual performance plans we reviewed.
Table 3: Characterizations of Agencies' 2002 Annual Performance Reports
Picture of Resources
linked
to results Data
performance credible
(unclear, (no, limited, (no,
limited, limited,
Agency general, clear) general, general,
clear) full)
Department of Limited Clear General
Education
Department of Energy General Limited Limited
Department of Housing General No General
and Urban Development
Small Business Limited General General
Administration
Social Security General Limited General
Administration
Department of General No Full
Transportation
Sources: GAO analysis of agencies' fiscal year 2002 annual performance
reports and U.S. General Accounting Office, Observations on the Department
of Education's Fiscal Year 1999 Performance Report and Fiscal Year 2001
Performance Plan, GAO/HEHS-00-128R (Washington, D.C.: June 30, 2000);
Observations on the Department of Energy's Fiscal Year 1999 Accountability
Report and Fiscal Years 2000 and 2001 Performance Plans, GAO/RCED-00-209R
(Washington, D.C.: June 30, 2000); Observations on the Department of
Housing and Urban Development's Fiscal Year 1999 Performance Report and
Fiscal Year 2001 Performance Plan, GAO/RCED-00-211R (Washington, D.C.:
June 30, 2000); Observations on the Small Business Administration's Fiscal
Year 1999 Performance Report and Fiscal Year 2001 Performance Plan,
GAO/RCED-00-207R (Washington, D.C.: June 30, 2000); Observations on the
Social Security Administration's Fiscal Year 1999 Performance Report and
Fiscal Year 2001 Performance Plan, GAO/HEHS-00-126R (Washington, D.C.:
June 30, 2000); and Observations on the Department of Transportation's
Fiscal Year 1999 Performance Report and Fiscal Year 2001 Performance Plan,
GAO/RCED-00-201R (Washington, D.C.: June 30, 2000).
Progress in Providing a Clear Picture of Performance
The six agency reports that we reviewed contained a number of strengths,
some of which we can describe as improvements over the reports on fiscal
year 1999 performance. A key strength of four of the 2002 reports
(Education, HUD, DOT, SSA) was a discussion of the relationship between
the strategic plan, performance plan, and performance report. For example,
SSA's report identified relevant results that were linked to its strategic
objective to deliver "citizen-centered, world-class service," such as
maintaining the accuracy, timeliness, and efficiency of service to people
applying for its benefit programs. The clarity of the DOE and SBA reports
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
was limited by not clearly relating agency performance results to
strategic and annual performance goals. For example, the structure of
SBA's report reflected the objectives in its draft 2003 to 2008 strategic
plan rather than those in its 2002 performance plan, making it difficult
to assess progress against the original 2002 objectives. Furthermore,
although there is no "right" number of performance measures to be used to
assess progress, a number of the plans allowed for an easier review of
results by limiting the overall number of measures presented or by
highlighting key performance measures of greatest significance to their
programs. For example, SBA discussed a total of 19 performance goals and
DOT discussed a total of 40. Although SSA discussed a total of 69
performance goals, the report highlighted its progress in achieving 14 key
goals. In contrast, Education, HUD, and DOE presented a total of 120, 184,
and 260 measures, respectively. Furthermore, while Education and SSA each
provided a table showing progress across all its measures, the other
agencies did not provide such summary information.
As we found in our earlier reviews, the six agencies' fiscal year 2002
reports generally allowed for an assessment of progress made in achieving
agency goals. Some of the reports made this assessment easier than others
by providing easy-to-read summary information. For example, SSA provided a
table at the beginning of the report that summarized the results for each
of its 69 indicators with the following dispositions: met, not met, almost
met, and data not yet available. Other reports, such as HUD's, required an
extensive review to make this assessment. In addition, to place current
performance in context, each of the agencies' reports contained trend
information, as required by GPRA, which allowed for comparisons between
current year and prior year performance.
In addition, the majority of agencies maintained, or demonstrated
improvements over, the quality of their 1999 reports in discussing the
progress achieved in addressing performance and accountability challenges
identified by agency IGs and GAO. For example, SBA's report contained two
broad overviews and an appendix describing the status of GAO audits and
recommendations, as well as a description of the most serious management
challenges SBA faces as identified by the agency's IG.
Unfortunately, many of the weaknesses we identified in the agencies'
fiscal year 2002 reports were similar to those we found in their fiscal
year 1999 reports related to the significant number of performance goals
(1) which were not achieved and lacked explanations or plans for achieving
the goal in the future and (2) for which performance data were
unavailable. Three
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
of the six agencies we reviewed-HUD, SSA, and Transportation-did not
consistently report the reasons for not meeting their goals. For example,
Transportation provided explanations for only 5 of the 14 goals it did not
meet. In addition, similar to our 1999 report findings, three of the six
agencies we reviewed-HUD, SBA, and DOT-did not discuss their plans or
strategies to achieve unmet goals in the future. For example, HUD reported
"substantially meeting" only 47 percent of the performance targets in
fiscal year 2002. However, although HUD provides various reasons for not
meeting all its targets, it offers no information on plans or time frames
to achieve the goals in the future. Finally, we continued to observe a
significant number of goals for which performance data were unavailable.
For example, performance data for 10 of SBA's 19 performance goals were
unavailable.
In addition, the majority of the reports we reviewed did not include other
GPRA requirements. The reports generally did not evaluate the performance
plan for the current year relative to the performance achieved toward the
performance goals in the fiscal year covered by the report. The reports
also did not discuss the use or effectiveness of any waivers in achieving
performance goals. In addition, for two of the agencies-DOE and
SBA-program evaluation findings completed during the fiscal year were not
summarized. As we have previously noted, such evaluations could help
agencies understand the relationship between their activities and the
results they hope to achieve.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
Progress in Linking Resources to Results
Although linking costs to performance goals is not a requirement of GPRA,
both GPRA and the CFO Act emphasized the importance of linking program
performance information with financial information as a key feature of
sound management and an important element in presenting to the public a
useful and informative perspective on federal spending. The committee
report for GPRA suggested that developing the capacity to relate the level
of program activity with program costs, such as cost per unit of result,
cost per unit of service, or cost per unit of output, should be a high
priority. In our survey of federal managers, this year we asked for the
first time the extent to which federal managers had measures of
costeffectiveness for the programs they were involved with. Only 31
percent of federal managers we surveyed reported having such measures to a
great or very great extent, lower than any of the other types of measures
associated with GPRA we asked about by at least 12 percent (see fig. 3 in
ch. 2). Under the PMA, the current administration has set an ambitious
agenda for performance budgeting, calling for agencies to better align
budgets with performance goals and focus on capturing full budgetary costs
and matching those costs with output and outcome goals. All this suggests
that agencies will need to develop integrated financial and performance
management systems that will enable the reporting of the actual costs
associated with performance goals and objectives along with presentations
designed to meet other budgetary or financial purposes, such as the
accounts and program activities found in the President's Budget and
responsibility segments found in financial statements.9
9According to OMB's Statement of Federal Financial Accounting Standards
No. 4- Managerial Cost Accounting Standards, July 31, 1995, a
responsibility segment is a component of a reporting entity that is
responsible for carrying out a mission, conducting a major line of
activity, or producing one or a group of related products or services. In
addition, responsibility segments usually possess the following
characteristics: (1) their managers report to the entity's top management
directly and (2) their resources and results of operations can be clearly
distinguished from those of other segments of the entity. Managerial cost
accounting should be performed to measure and report the costs of each
segment's outputs.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
Of the six agencies we reviewed, only Education's report clearly linked
its budgetary information to the achievement of its performance goals or
objectives. Education's report laid out, using both graphics and text, the
estimated appropriations associated with achieving each of its 24
objectives. In addition the report provided the staffing in full-time
equivalent employment (FTEs) and an estimate of the funds from salaries
and expenses contributing to the support of each of these objectives.
SBA's report contained crosswalks that showed the relationship between
SBA's strategic goals, outcome goals, performance goals, and programs.
Because SBA shows the resources for each program, a reader can infer a
relationship between SBA's resources and performance goals. However, the
linkage between resources and results would be clearer if results and
resources were presented by performance goal as well. SSA provided a
limited view of the costs of achieving its performance goals by providing
the costs associated with four out of five of its strategic goals.10
However, as reported by the IG, SSA needs to further develop its cost
accounting system, which would help link costs to performance.11 DOE also
provided a limited view of the costs of achieving its performance goals by
organizing its performance information by budget program activity and
associated net costs. According to DOE officials, the department plans to
link its individual performance measures to the costs of program
activities in future reports. Neither HUD nor DOT provided information on
the cost of achieving individual performance goals or objectives.
Progress in Providing Confidence in the Credibility of Performance Data
To assess the degree to which an agency's report provided full confidence
that the agency's performance information was credible, we examined the
extent to which the reports discussed the quality of the data presented.
As shown in table 3, only DOT's report provided a full level of confidence
in the quality of the data. The other agencies provided general or limited
confidence in their data.
10SSA noted that its fifth strategic goal, "Valued Employees," supports
the accomplishment of all its basic functions, so its resources are
inherently included in the other four goals.
11According to the IG, SSA began to implement an improved cost accounting
system in fiscal year 2002, which was to be phased in over the next 3 to 4
years.
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
All six agencies in our current review complied with the Reports
Consolidation Act of 2000 by including assessments of the completeness and
reliability of their performance data in their transmittal letters. In
contrast, we found that only 5 of the 24 CFO Act agencies included this
information in their fiscal year 2000 performance reports.12 Of the six
agencies in our current review, only DOE provided this assessment in its
fiscal year 2000 report. For example, the Secretary of DOT stated in the
transmittal letter that the 2002 report "contains performance and
financial data that are substantially complete and reliable." However,
only two of the six agencies also disclosed material inadequacies in the
completeness and reliability of their performance data and discussed
actions to address the inadequacies in their transmittal letters. For
example, SBA stated in its transmittal letter that it is "working to
improve the completeness and reliability of the performance data for the
advice provided to small business through SBA's resource partners." SBA
explained that data for this aspect of its performance are collected
through surveys, which are inconsistent and not comparable, and for which
client responses are difficult to obtain. SBA stated that it is working to
improve the survey instruments it uses to obtain performance data.
In addition to the requirements of the Reports Consolidation Act, we have
previously reported on other practices that enhance the credibility of
performance data that are not specifically required by GPRA. For instance,
discussions of standards and methods used by agencies to assess the
quality of their performance data in their performance reports provide
decision makers greater insight into the quality and value of the
performance data. None of the reports explicitly referred to a specific
standard they used, however, DOE described its method for assuring data
quality. The report states that the heads of DOE's organizational elements
certified the accuracy of their performance data. DOE subsequently
reviewed the data for quality and completeness.
Other useful practices that help foster transparency to the public and
assist decision makers in understanding the quality of an agency's data
include: (1) discussion of data quality, including known data limitations
and actions to address the limitations, and (2) discussion of data
verification and
12U.S. General Accounting Office, Performance Reporting: Few Agencies
Reported on the Completeness and Reliability of Performance Data,
GAO-02-372 (Washington, D.C.: Apr. 26, 2002).
Chapter 3
Agencies Have Addressed Many Critical
Performance Planning and Reporting
Challenges, but Weaknesses Persist
validation procedures, including proposals to review data collection and
verification and validation procedures.
All six agencies' reports described data limitations, although discussions
were mostly brief and very high level. One exception was DOT, which
directed readers to the DOT Web site to obtain an assessment of the
completeness and reliability of its performance data and detailed
information on the source, scope, and limitations of the performance data.
HUD and SBA also discussed plans for addressing the limitations. For
example, HUD stated that to address problems with its indicator on the
number of homeowners who have been assisted with the Home Investment
Partnership Program (HOME), HUD has established a team of managers,
technical staff, and contractors to make a series of improvements to the
Integrated Disbursement and Information System beginning in fiscal year
2003 that should reduce the need to clean up the data.
Each of the six agencies' reports also discussed the procedures they used
to verify and validate their performance data. However, these discussions
ranged from the very general description of the DOE method (noted
previously), to the very detailed discussions provided by DOT. DOT
provides an on-line compendium that discusses the source and accuracy of
its data. Furthermore, DOT's 2002 report also describes strategies being
undertaken to address the quality of its data. The report states that a
DOT intermodal working group addressed data quality issues by developing
departmental statistical standards and by updating source and accuracy
statements for all of DOT's data programs. The working group also worked
to improve quality assurance procedures, evaluate sampling and nonsampling
errors, and develop common definitions for data across modes.
Chapter 4
Challenges to GPRA Implementation Persist
While a great deal of progress has been made in making federal agencies
more results oriented, numerous challenges still exist to effective
implementation of GPRA. The success of GPRA depends on the commitment of
top leadership within agencies, OMB, and Congress. However, according to
federal managers surveyed, top leadership commitment to achieving results
has not grown significantly since our 1997 survey. Furthermore, although
OMB has recently shown an increased commitment to management issues, it
significantly reduced its guidance to agencies on GPRA implementation
compared to prior years, and it is not clear how the program goals
developed through its PART initiative will complement and integrate with
the long-term, strategic focus of GPRA. Obtaining leadership commitment to
implement a strategic plan depends in part on the usefulness and relevance
of agency goals and strategies to agency leaders, Congress, and OMB.
However, GPRA's requirement to update agency strategic plans every 3 years
is out of sync with presidential and congressional terms and can result in
updated plans that do not have the support of top administration
leadership and key congressional stakeholders.
As noted in chapter 2, more federal managers surveyed reported having
results-oriented performance measures for their programs and we would
expect to have seen similar increases in the use of this information for
program management. However, we did not observe any growth in their
reported use of this information for key management activities, such as
adopting new program approaches or changing work processes. Additionally,
managers noted human capital-related challenges that impede
results-oriented management, including a lack of authority and training to
carry out GPRA requirements, as well as a lack of recognition for the
results achieved.
Consistent with our previous work, federal managers in our focus groups
reported that significant challenges persist in setting outcome-oriented
goals, measuring performance, and collecting useful data. However, our
survey data suggested that federal managers do not perceive issues, such
as "difficulty distinguishing between the results produced by the program
and results caused by other factors" and "difficulty obtaining data in
time to be useful," to be substantial hindrances to measuring performance
or using performance information.
Additionally, mission fragmentation and overlap contribute to difficulties
in addressing crosscutting issues, particularly when those issues require
a national focus, such as homeland security, drug control, and the
Chapter 4
Challenges to GPRA Implementation Persist
environment. GAO has previously reported on a variety of barriers to
interagency cooperation, such as conflicting agency missions, jurisdiction
issues, and incompatible procedures, data, and processes. We have also
reported that OMB could use the provision of GPRA that calls for OMB to
develop a governmentwide performance plan to integrate expected
agencylevel performance. Unfortunately, this provision has not been fully
implemented and the federal government lacks a tool, such as a strategic
plan, that could provide a framework for a governmentwide reexamination of
existing programs, as well as proposals for new programs. Finally, federal
managers in our focus groups and political appointees we interviewed
believed that Congress does not use performance information to the fullest
extent to conduct oversight and to inform appropriations decisions. While
there is concern regarding Congress' use of performance information, it is
important to make sure that this information is initially useful. As a key
user of performance information, Congress needs to be considered a partner
in shaping agency goals at the outset. GPRA provides Congress
opportunities to influence agency performance goals through the
consultation requirement for strategic plans and through Congress'
traditional oversight role.
Top Leadership Does Not Consistently Show Commitment to Achieving Results
We have previously testified that perhaps the single most important
element of successful management improvement initiatives is the
demonstrated commitment of top leaders to change.1 This commitment is most
prominently shown through the personal involvement of top leaders in
developing and directing reform efforts. Organizations that successfully
address their long-standing management weaknesses do not "staff out"
responsibility for leading change. Top leadership involvement and clear
lines of accountability for making management improvements are critical to
overcoming organizations' natural resistance to change, marshalling the
resources needed in many cases to improve management, and building and
maintaining the organizationwide commitment to new ways of doing business.
Results from our surveys show that while the majority of managers continue
to indicate top leadership demonstrates a strong commitment to achieving
results, we have not seen a noteworthy improvement in the percentage of
managers expressing this view. From our 1997 survey, we
1U.S. General Accounting Office, Management Reform: Elements of Successful
Improvement Initiatives, GAO/T-GGD-00-26 (Washington, D.C.: Oct. 15,
1999).
Chapter 4
Challenges to GPRA Implementation Persist
estimated about 57 percent of managers overall reported such commitment to
a great or very great extent. On our 2003 survey, 62 percent of managers
expressed a comparable view-a higher but not statistically significant
increase. (See fig. 10.)
Figure 10: Percentage of Federal Managers Who Reported to a Great or Very
Great Extent Their Top Leadership Has a Strong Commitment to Achieving
Results Percent
100
90
80
70
62
60
50
40
30
20
10
0 1997 2000 2003 Year
Source: GAO.
As shown in figure 11, however, we continued to see a significant
difference between the perceptions of SES and non-SES managers on this
issue. That is, the percentage of SES managers reporting that top
leadership demonstrated strong commitment to a great or very great extent
in 2003 was 22 percent higher than for non-SES managers.
Chapter 4
Challenges to GPRA Implementation Persist
Figure 11: Percentage of SES and Non-SES Managers Who Reported to a Great
or
Very Great Extent Their Agency Top Leadership Demonstrated Strong
Commitment
to Achieving Results
Percent
100
90
82
80
70
60
60
50
40
30
20
10
0
1997a 2000a 2003a
Year
SES
Non-SES
Source: GAO.
aThere was a statistically significant difference between SES and non-SES.
We observed in our 1997 and 2000 reports on governmentwide implementation
of GPRA that we would expect to see managers' positive perceptions on
items, such as the extent to which top leadership is committed to
achieving results, become more prevalent and the gap between SES and
non-SES managers begin to narrow as GPRA and related reforms are
implemented; however, these changes do not appear to be happening as
expected.2
Demonstrating the willingness and ability to make decisions and manage
programs based on results and the ability to inspire others to embrace
such
2U.S. General Accounting Office, The Government Performance and Results
Act: 1997 Governmentwide Implementation Will Be Uneven, GAO/GGD-97-109
(Washington, D.C.: June 2, 1997) and Managing for Results: Federal
Managers' Views Show Need for Ensuring Top Leadership Skills, GAO-01-127
(Washington, D.C.: Oct. 20, 2000).
Chapter 4
Challenges to GPRA Implementation Persist
a model are important indicators of leadership commitment to
resultsoriented management. However, in both our 1997 and 2000 surveys,
only about 16 percent of managers reported that changes by management
above their levels to the programs for which they were responsible were
based on results or outcome-oriented performance information to a great or
very great extent. In our 2003 survey, this indicator increased to 23
percent, a statistically significant increase from prior surveys.
Twenty-eight percent of federal managers surveyed who expressed an opinion
reported that the lack of ongoing top executive commitment or support for
using performance information to make program/funding decisions hindered
measuring performance or using performance information to a great or very
great extent.
Our interviews with 10 top political appointees from the Clinton and
current Bush administrations indicated a high level of support and
enthusiasm for effectively implementing the principles embodied in GPRA.
For example, one appointee noted that GPRA focused senior management on a
set of goals and objectives to allow the organization to understand what
is important and how to deal with accomplishment at a macro-level, as well
as provided a structure for problem solving. Another political appointee
noted that GPRA has made it important to look at what you get out of the
budget, not just what you put into it, while another concluded that GPRA
brought about a fundamental rethinking of how they managed their programs
and processes. Such indications of support for GPRA are promising.
However, to support the transition to more results-oriented agency
cultures, top agency management will need to make a more concerted effort
to translate their enthusiasm for GPRA into actions that communicate to
employees that top management cares about performance results and uses the
information in its decision making.
The need for strong, committed leadership extends to OMB as well. OMB has
shown a commitment to improving the management of federal programs, both
through its leadership in reviewing agency program performance using the
PART tool as well as through the PMA, which calls for improved financial
performance, strategic human capital management, competitive sourcing,
expanded electronic government, and performance budget integration. Using
the foundation of information generated by agencies in their strategic
plans, annual performance plans, and program performance reports, OMB has
used the PART tool to exercise oversight of selected federal programs by
assessing program purpose and design, the quality of strategic planning,
the quality of program management, and the extent to which programs can
demonstrate results. PART provides OMB a
Chapter 4
Challenges to GPRA Implementation Persist
lens through which to view performance information for use in the budget
formulation process. PART, and OMB's use of performance data in the budget
formulation process, potentially can complement GPRA's focus on increasing
the supply of credible performance information by promoting the demand for
this information in the budget formulation process. As we reported in
chapter 2, more federal managers noted that OMB was paying attention to
their agencies' efforts under GPRA. (See fig. 6.) Additionally, OMB
convened a performance measurement workshop in April 2003 to identify
practical strategies for addressing common performance measurement
challenges. As a result of this workshop, it produced a paper in June 2003
that included basic performance measurement definitions and concepts and
common performance measurement problems that were discussed at the
workshop. This was part of OMB's continuing efforts to improve PART as an
evaluation tool.
However, there are areas where OMB could further enhance its leadership.
OMB has stated that the PART exercise presents an opportunity to inform
and improve on agency GPRA plans and reports and establish a meaningful,
systematic link between GPRA and the budget process. OMB has instructed
agencies that, in lieu of a performance plan, they are to submit a
performance budget that includes information from the PART assessments,
including all performance goals used in the assessment of program
performance done under the PART process. The result is that
program-specific performance measures developed through the PART review
are to substitute for other measures developed by the agency through its
strategic planning process. GPRA is a broad legislative framework that was
designed to be consultative with Congress and other stakeholders and
address the needs of many users of performance information-Congress to
provide oversight and inform funding decisions, agency managers to manage
programs and make internal resource decisions, and the public to provide
greater accountability. Changing agency plans and reports for use in the
budget formulation process may not satisfy the needs of these other users.
Users other than OMB are not likely to find the information useful unless
it is credible and valid for their purposes. PART's program-specific focus
may fit with OMB's agency-byagency budget reviews, but it is not well
suited to achieving one of the key purposes of strategic plans-to convey
agencywide, long-term goals and objectives for all major functions and
operations. PART's focus on program-specific measures does not substitute
for the strategic, long-term focus of GPRA on thematic goals and
department-and governmentwide crosscutting comparisons.
Chapter 4
Challenges to GPRA Implementation Persist
To reach the full potential of performance management, agency planning and
reporting documents need to reflect the full array of uses of performance
information, which may extend beyond those needed for formulating the
President's Budget. However, it is not yet clear whether the results of
those reviews, such as changes to agencies' program performance measures,
will complement and be integrated with the longterm, strategic goals and
objectives agencies have established in consultation with Congress and
other stakeholders under GPRA. OMB has not yet clearly articulated how
PART is to complement GPRA. Focus group participants suggested that the
administration and OMB needed to reinforce GPRA's usefulness as a
management tool for agencies. They also emphasized the need for OMB to
help agencies understand how to integrate GPRA with other management
initiatives, such as PART.
As we noted in chapter 3, agencies' plans and reports still suffer from
persistent weaknesses and could improve in a number of areas, such as
attention to issues that cut across agency lines, and better information
about the quality of the data that underlie agency performance goals.
However, OMB's July 2003 guidance for the preparation and submission of
strategic plans, annual performance plans, and annual performance reports
is significantly shorter and less detailed than its 2002 guidance. For
example, OMB no longer provides detailed guidance to agencies for the
development of performance plan components. OMB's 2002 guidance on the
preparation and submission of annual performance plans is approximately 39
pages long; in its 2003 guidance, that discussion spans only 2 pages. The
2003 guidance in this area does not include entire sections found in the
2002 guidance, such as principles for choosing performance goals and
indicators for inclusion in the annual plan, types of performance goals,
crosscutting programs, and requirements for verifying and validating data.
OMB needs to maintain and strengthen its leadership role in working with
agencies to help them produce the highest quality GPRA documents through
its formal guidance and reviews of strategic plan and report submissions.
Focus group participants discussed the need for consistent guidance on how
to implement GPRA. Furthermore, there is no evidence that agencies have
institutional knowledge of GPRA requirements that would obviate the need
for OMB's guidance. New managers will need a consistent resource that
provides practical guidance on what agencies need to include in their
planning and reporting documents to comply with GPRA and reflect best
practices. Consistent, explicit OMB guidance on preparing GPRA documents
can help ensure that gains in the quality of
Chapter 4
Challenges to GPRA Implementation Persist
GPRA documents are maintained and provide a resource for agencies to make
further improvements in those documents. For example, guidance on how to
discuss coordination of crosscutting programs or improvements to the
credibility of performance data in agency performance plans goes
hand-in-hand with OMB's enhanced oversight of agency performance through
the PART exercise.
The success of GPRA depends on the commitment of top leadership within
agencies, OMB, and Congress. Obtaining such leadership commitment depends
in part on the usefulness and relevance of agency goals and strategies to
these parties. GPRA requires an agency to develop a strategic plan at
least every 3 years to cover the following 5-year period. Thus, there have
been two required updates of strategic plans since the initial strategic
plans were submitted for fiscal year 1997-fiscal year 2000 and fiscal year
2003. The fiscal year 2000 update occurred the year before a new
presidential term began. According to our focus group participants-both
the experts and federal managers-it makes little sense to require an
update of a strategic plan shortly before a new administration is
scheduled to take office. For example, changes in political leadership
generally result in a new agenda with new objectives. Such changes force
agencies to revise their plans, management initiatives, and strategies,
which translates into additional GPRA-related work. A strategic plan that
does not reflect the participation and buy-in of top administration
leadership and key congressional stakeholders is unlikely to be
successfully implemented. Therefore, GPRA's requirement to update agency
strategic plans according to a schedule that is out of sync with
presidential and congressional terms means that effort may be wasted on
plans that lack the support of top leadership.
Managers Report Mixed Results in Use of Performance Information
GPRA's usefulness to agency leaders and managers as a tool for management
and accountability was cited as a key accomplishment numerous times by
focus group participants. However, a number of alternative views indicated
use of performance information for key management decisions has been
mixed. For example, one participant said they did not believe GPRA has
been used as a tool yet, while another participant commented that only
better managers take advantage of GPRA as a management tool. According to
focus group participants, although many federal managers understand and
use results-oriented management concepts in their day-to-day activities,
such as strategic planning and performance measurement, they do not always
connect these concepts to the requirements of GPRA.
Chapter 4
Challenges to GPRA Implementation Persist
This view was strongly supported by our survey results. Prior to
mentioning GPRA in our survey, we asked federal managers the extent to
which they consider their agency's strategic goals when engaging in key
management tasks such as setting program activities, allocating resources,
or considering changes in their programs. A relatively high percentage of
managers-ranging from 66 to 79 percent-responded to a great or very great
extent. However, when we asked similar questions about the extent to which
they considered their agency's annual performance goals as set forth in
the agency's GPRA annual performance plan for the same activities, the
comparable responses were considerably lower, ranging from 22 to 27
percent.
Because the benefit of collecting performance information is only fully
realized when this information is actually used by managers, we asked them
about the extent to which they used the information obtained from
measuring performance for various program management activities. As shown
in figure 12, for seven of the nine activities we asked about, the
majority of managers who expressed an opinion reported using performance
information to a great or very great extent in 2003. Across all nine
activities, the percentage of managers saying they used performance
information to a great or very great extent ranged from 41 percent for
developing and managing contracts to 60 percent for allocating resources,
setting individual job expectations, and rewarding staff. While we had
observed a decline in the reported use of performance information to this
extent for many of these activities between 1997 and 2000, our 2003
results increased to levels not significantly different from 1997 for all
but one category-adopting new program approaches or changing work
processes. This category of use continued to be significantly lower at 56
percent in 2003 than it was in 1997 at 66 percent. Although another
category, coordinating program efforts with other internal or external
organizations, shows a similar pattern of limited recovery, the difference
between the 1997 and 2003 results is not statistically significant.
Chapter 4
Challenges to GPRA Implementation Persist
Figure 12: Percentage of Federal Managers Who Reported Using Information
Obtained from Performance Measurement to a
Great or Very Great Extent for Various Management Activities
Percent
100
90
80
70 66 66 60
50
40
30
20
10
0
rk processesapproaches or c
owAdopting new program
am
cesAllocatingresour
ts
amogr
performance measures
w or revising isting performance goals Setting individual job
ge
or supervise
ogrpriorities
Setting pr
a
Coordinating prwith other internal or external y staff
r mo
pectations fxe
veDemanaging contractsor
ganizations Refining pr
warding staff I manaRe
am effogror
xeKey management activities
1997
2000
2003
Source: GAO. Note: Percentages are based on those respondents answering on
the extent scale. aThere was a statistically significant difference
between the 1997 and 2003 surveys. bThis question was not asked in 1997.
Chapter 4
Challenges to GPRA Implementation Persist
We have reported that involving program managers in the development of
performance goals and measures is critical to increasing the relevance and
usefulness of this information to their day-to-day activities.3 Yet, our
survey data indicate that participation in activities related to the
development and use of performance information has also been mixed. In
2003, only 14 percent of managers believed to a great or very great extent
that their agencies considered their contributions to or comments on their
agency's GPRA plans or reports. However, significantly more SES managers
(43 percent) than non-SES managers (12 percent) expressed this view. Also,
when compared to our 2000 survey when we first asked this question, the
percentage of SES managers expressing this view in 2003 was significantly
higher than in 2000 (32 percent). The percentage of non-SES managers was
essentially unchanged from 2000 (10 percent).
Furthermore, as shown in figure 13, overall around half or fewer of
managers responded "yes" on our 2003 survey to questions about being
involved in developing ways to measure whether program performance goals
are being achieved (46 percent), gathering and analyzing data to measure
whether programs were meeting their specific performance goals (51
percent), or using measures for program performance goals to determine if
the agency's strategic goals were being achieved (43 percent). None of
these overall results were significantly different from our 1997 results.
We did find, however, that significantly more SES managers responded "yes"
on the 2003 survey (72 percent) than the 1997 survey (55 percent) with
regard to being involved in using performance measurement information to
determine if the agency's strategic goals were being achieved when
compared to our 1997 results.
3GAO/GGD-97-109 and GAO-01-127.
Chapter 4
Challenges to GPRA Implementation Persist
Figure 13: Percentage of Federal Managers Responding "Yes" about Being
Involved in the Following Activities
Percent 100
90
80
70
60 55
50 40
30 35
20
10
0 1997 2003
Year
Developing ways to measure whether program performance goals are being
achieved
Gathering and analyzing data to measure whether programs are meeting their
specific performance goals
Using measurements for program performance goals to determine if the
agency's strategic goals are being achieved
Source: GAO.
Managers Continue to Managing people strategically and maintaining a
highly skilled and
energized workforce that is empowered to focus on results are
criticallyConfront a Range of important. Such human capital management
practices are essential to the Human Capital success of the federal
government in the 21st century and to maximizing Management the value of
its greatest asset-its people. Our survey results showed
continuing challenges related to the adequacy of managerial
decisionChallenges making authority, training, and incentives.
Chapter 4
Challenges to GPRA Implementation Persist
Federal Managers Report That They Are Held Accountable for Program Results
but Do Not Have the Decision-Making Authority They Need to Accomplish
Agency Goals
High-performing organizations seek to shift the focus of management and
accountability from activities and processes to contributions and
achieving results. In each of our three surveys, we asked managers about
the amount of decision-making authority they had and the degree to which
they were held accountable for results.
As shown in figure 14, for 2003, an estimated 40 percent of federal
managers overall reported that they had the decision-making authority they
needed to help the agency accomplish its strategic goals to a great or
very great extent. This was a statistically significant increase over our
1997 estimate of 31 percent. While there were more SES and non-SES
managers expressing this view on our 2003 survey than the 1997 survey, it
was the non-SES managers that showed the significant increase. Despite
this promising trend, however, there continued to be substantial
differences in 2003, as well as on the two previous surveys, between the
responses of SES and lower-level managers on this question. Compared to
the 57 percent of SES managers who reported having such authority to a
great or very great extent in 2003, only 38 percent of non-SES managers
reported having such authority to a great or very great extent.
Chapter 4
Challenges to GPRA Implementation Persist
Figure 14: Percentage of Federal Managers Reporting to a Great or Very
Great Extent That Managers/Supervisors at Their Levels Had the
Decision-Making Authority They Needed to Help the Agency Accomplish Its
Strategic Goals
Percent
100
90
80
70
60 56 57
50
40
40 30
29
20 10
1997 2000 2003 Year
Federal managersa
SESb
Non-SESa,b
Source: GAO.
aThere was a statistically significant difference between the 1997 and
2003 surveys. bThere was a statistically significant difference between
SES compared to non-SES for each survey.
However, when asked the extent to which managers or supervisors at their
levels were held accountable for the accomplishment of agency strategic
goals, 57 percent responded to a great or very great extent in 2003.
Unlike in other areas, where SES managers had significantly different
views from non-SES managers, there was little difference in the area of
accountability. (See fig. 15.)
Chapter 4
Challenges to GPRA Implementation Persist
Figure 15: Percentage of Federal Managers, SES, and Non-SES in 2003
Reporting to
a Great or Very Great Extent That They Were Held Accountable for the
Accomplishment of Agency Strategic Goals Percent
100
90
80
70
61
60
50
40
30
20
10
0 Federal Non-SES SES managers
Source: GAO.
This 57 percent is significantly higher than the 40 percent of managers
overall who indicated that they had comparable decision-making authority.
However, in contrast to the question on authority, as shown in figure 14,
where more SES managers than non-SES managers expressed the view that they
had the authority, there was little difference, as shown in figure 15,
between the two groups in their views about being held accountable for
achieving agency strategic goals to a great or very great extent. As
figures 14 and 15 further illustrate, roughly the same percentage of SES
managers perceived to a great or very great extent that managers at their
level had decision-making authority and accountability for achieving
agency strategic goals. This result suggests that their authority was
perceived to be on par with their accountability. In contrast, only 38
percent of non-SES managers perceived that managers at their levels had
the decision-making authority they needed to a great or very great extent,
while 57 percent perceived that they were held accountable to a comparable
extent.
Managers are hard-pressed to achieve results when they do not have
sufficient authority to act. In our report containing the results of our
1997 survey, we noted that agencies needed to concentrate their efforts on
areas
Chapter 4
Challenges to GPRA Implementation Persist
where managers were not perceiving or experiencing progress, such as that
concerning devolving decision-making authority to managers throughout
their organizations. While authority for achieving results appears to be
in a modestly upward trend, the balance between authority and
accountability that fosters decision making to achieve results could be
further improved, particularly among non-SES managers.
Fewer Than Half of Managers Reported Training on Key Tasks
We previously reported on the need for agencies to expend resources on
effective training and professional development to equip federal employees
to work effectively.4 Among the resources focus group participants cited
as lacking included federal managers and staff with competencies and
skills needed to plan strategically, develop robust measures of
performance, and analyze what the performance data mean. Our 2003 Guide
calls for training and development efforts to be strategically focused on
improving performance toward the agency's goals and put forward with the
agency's organizational culture firmly in mind.5 Throughout this process
it is important that top leaders in the agencies communicate that
investments in training and development are expected to produce clearly
identified results. By incorporating valid measures of effectiveness into
the training and development programs they offer, agencies can better
ensure that they will adequately address training objectives and thereby
increase the likelihood that desired changes will occur in the target
population's skills, knowledge, abilities, attitudes, or behaviors.
Furthermore, if managers understand and support the objectives of training
and development efforts, they can provide opportunities to successfully
use the new skills and competencies on the job and model the behavior they
expect to see in their employees.
In response to our 2003 survey, fewer than half of managers answered "yes"
when we asked them whether, during the past 3 years, their agencies had
provided, arranged, or paid for training that would help them accomplish
any of seven critical results-oriented management-related tasks. However,
progress is indicated in our survey results. As shown in figure 16, more
managers answered "yes" in 2003 on all seven training areas than in
4GAO-01-127.
5U.S. General Accounting Office, Human Capital: A Guide for Assessing
Strategic Training and Development Efforts in the Federal Government
(Exposure Draft) GAO-03893G (Washington, D.C.: July 1, 2003).
Chapter 4
Challenges to GPRA Implementation Persist
previous surveys. These increases were statistically significant for five
of the tasks-setting program performance goals, using program performance
information to make management decisions, linking program performance to
the achievement of agency strategic goals, and implementing the
requirements of GPRA.
Figure 16: Percentage of Federal Managers in Each Survey Year Who Reported
That during the Past 3 Years Their Agencies Provided, Arranged, or Paid
for Training That Would Help Them Accomplish Specific Tasks
50 Percent 49
40
30
20
10
Assess the quality of performance datab
Source: GAO.
Develop program Conduct Implement the Link the Use program Set program
performance strategic requirements performance of performance performance
measures planning of GPRAc programs/operations/ information to goalsc
projects to the make decisionsc achievement of agency strategic goalsc
1997
2000
2003
aThis question was not asked in the 1997 survey.
bThere was a statistically significant difference between the 2000 and
2003 surveys.
cThere was a statistically significant difference between the 1997 and
2003 surveys.
As with our 2000 survey results, the 2003 survey results continued to
demonstrate that there is a positive relationship between agencies
providing training and development on setting program performance goals
and the use of performance information when setting or revising
Chapter 4
Challenges to GPRA Implementation Persist
performance goals. For those managers who responded "yes" to training on
setting performance goals, 60 percent also reported that they used
information obtained from performance measurement when setting new or
revising existing performance goals to a great or very great extent. In
contrast, for those managers who responded "no" to training on setting
performance goals, only 38 percent reported that they used information
obtained from performance measurement for setting new or revising existing
performance goals to a great or very great extent. The difference between
these percentages is statistically significant. Effective training and
development programs are an integral part of a learning environment that
can enhance the federal government's ability to attract and retain
employees with the skills and competencies needed to achieve results.
Training and developing new and current staff to fill new roles and work
in different ways will be a crucial part of the federal government's
endeavors to meet its transformation challenges. Ways that employees learn
and achieve results will also continue to transform how agencies do
business and engage employees in further innovation and improvements.
Managers Perceive a Lack of Positive Recognition for Helping Agencies
Achieve Results
Another fundamental aspect of the human capital management challenge
agencies face is providing the incentives to their employees to encourage
results-oriented management. Monetary and nonmonetary incentives can be
used as a method for federal agencies to reward employees and to motivate
them to focus on results.
Overall, an increasing but still small percentage of managers reported in
1997, 2000, and 2003 that employees in their agencies received positive
recognition to a great or very great extent for helping agencies
accomplish their strategic goals. In 1997, 26 percent of federal managers
reported such an extent of positive recognition as compared to 37 percent
in 2003, a statistically significant increase. Interestingly, this
improvement is seen in the responses of non-SES managers. As shown in
figure 17, the percentage of SES managers expressing this view stayed at
about the same level over the three surveys, while the percentage of
non-SES managers holding this view was significantly higher in 2003 than
in 1997. Even with this improvement on the part of the responses from
non-SES managers, significantly more SES managers (47 percent) than
non-SES managers (36 percent) expressed this perception to a comparable
extent in 2003.
Chapter 4
Challenges to GPRA Implementation Persist
Figure 17: Percentage of Federal Managers Who Reported to a Great or Very
Great Extent That Employees in Their Agencies Received Positive
Recognition for Helping Their Agencies Accomplish Their Strategic Goals
Percent
100
90
80
70
60
52
50
37
40
30
20 25
10
0
1997 2000 2003
Year
Federal managersa
SES
Non-SESa
Source: GAO.
aThere was a statistically significant difference between the 1997 and
2003 surveys.
Unfortunately, most existing federal performance appraisal systems are not
designed to support a meaningful performance-based pay system in that they
fail to link institutional, program, unit, and individual performance
measurement and reward systems. In our view, one key need is to modernize
performance management systems in executive agencies so that they link to
the agency's strategic plan, related goals, and desired outcomes and are
therefore capable of adequately supporting more performancebased pay and
other personnel decisions.
We have reported federal agencies can develop effective performance
management systems by implementing a selected, generally consistent set of
key practices. These key practices helped public sector organizations both
in the United States and abroad create a clear linkage-"line of
sight"-between individual performance and organizational success and,
Chapter 4
Challenges to GPRA Implementation Persist
thus, transform their cultures to be more results oriented,
customerfocused, and collaborative in nature. Examples of such practices
include
o aligning individual performance expectations with organizational goals,
o connecting performance expectations to crosscutting goals,
o linking pay to individual and organizational performance, and
o making meaningful distinctions in performance.6
Beyond implementing these key practices, high-performing organizations
understand that their employees are assets whose value to the organization
must be recognized, understood, and enhanced. They view an effective
performance management system as an investment to maximize the
effectiveness of people by developing individual potential to contribute
to organizational goals. To maximize this investment, an organization's
performance management system is designed, implemented, and continuously
assessed by the standard of how well it helps the employees help the
organization achieve results and pursue its mission.
6For a complete list and discussion of the practices, see U.S. General
Accounting Office, Results oriented Cultures: Creating a Clear Linkage
between Individual Performance and Organizational Success, GAO-03-488
(Washington, D.C.: Mar. 14, 2003).
Chapter 4
Challenges to GPRA Implementation Persist
Persistent Challenges in Setting Outcome-Oriented Goals, Measuring
Performance, and Collecting Useful Data
In prior reports, we have described difficulties faced by federal managers
in developing useful, outcome-oriented measures of performance and
collecting data indicating progress achieved.7 One of the most persistent
challenges has been the development of outcome-oriented performance
measures. Additionally, it is difficult to distinguish the impact of a
particular federal program from the impact of other programs and factors,
thus making it difficult to attribute specific program performance to
results. The lack of timely and useful performance information can also
hinder GPRA implementation.
Meaningful, Outcome-Oriented Performance Measures Are Sometimes Hard to
Develop
In the past, we have noted that federal managers found meaningful
performance measures difficult to develop. Focus group participants and
survey respondents noted that outcome-oriented performance measures were
especially difficult to establish when the program or line of effort was
not easily quantifiable. The challenge of the "complexity of establishing
outcome-oriented goals and measuring performance" was cited by six of the
eight focus groups as one of the key challenges that managers face in
implementing GPRA. Focus group participants agreed that they often felt as
if they were trying to measure the immeasurable, not having a clear
understanding of which performance indicators could accurately inform the
agency how it is carrying out a specific activity. Managers from agencies
engaged in basic science research and development and grantmaking
functions noted that this effort was particularly difficult for them
because federal programs, especially those that are research-based, often
take years to achieve the full scope of their goals. On our most recent
survey, we estimated that 36 percent of federal managers who had an
opinion indicated that the determination of meaningful measures hindered
the use of performance information or performance measurement to a great
or very great extent. While this number was significantly lower than the
percentage of managers expressing the comparable view on the 1997 or 2000
survey and may reflect some lessening of this as a hindrance to some
7See for example, U.S. General Accounting Office, Managing for Results:
Analytic Challenges in Measuring Performance, GAO/HEHS/GGD-97-138
(Washington, D.C.: May 30, 1997); Program Evaluation: Agencies Challenged
by New Demand for Information on Program Results, GAO/GGD-98-53
(Washington, D.C.: Apr. 24, 1998); Managing for Results: Measuring Program
Results That Are Under Limited Federal Control, GAO/GGD-99-16 (Washington,
D.C.: Dec. 11, 1998); and Managing for Results: Challenges Agencies Face
in Producing Credible Performance Information, GAO/GGD-00-52 (Washington,
D.C.: Feb. 4, 2000).
Chapter 4
Challenges to GPRA Implementation Persist
managers, it nonetheless continues to be among those items having the
largest percentage of managers citing it as a substantial hindrance.
Impact of Federal Programs Difficult to Discern
In our June 1997 report on GPRA, we noted that "the often limited or
indirect influence that the federal government has in determining whether
a desired result is achieved complicates the effort to identify and
measure the discrete contribution of the federal initiative to a specific
program result."8 This occurs primarily because many federal programs'
objectives are the result of complex systems or phenomena outside the
program's control. In such cases, it is particularly challenging for
agencies to confidently attribute changes in outcomes to their program-the
central task of program impact evaluation. This is particularly
challenging for regulatory programs, scientific research programs, and
programs that deliver services to taxpayers through third parties, such as
state and local governments.
We have reported that determining the specific outcomes resulting from
federal research and development has been a challenge that will not be
easily resolved.9 Due to the difficulties in identifying outcomes,
research and development agencies typically have chosen to measure a
variety of proxies for outcomes, such as the number of patents resulting
from federally funded research, expert review and judgments of the quality
and importance of research findings, the number of project-related
publications or citations, and contributions to expanding the number of
research scientists.
8GAO/GGD-97-109, 6.
9U.S. General Accounting Office, Managing for Results: Key Steps and
Challenges in Implementing GPRA in Science Agencies, GAO/T-GGD/RCED-96-214
(Washington, D.C.: July 10, 1996).
Chapter 4
Challenges to GPRA Implementation Persist
We have also reported that implementing GPRA in a regulatory environment
is particularly challenging.10 Although federal agencies are generally
required to assess the potential benefits and costs of proposed major
regulatory actions, they generally do not monitor the benefits and costs
of how these and other federal programs have actually performed. For
example, in the case of the Environmental Protection Agency (EPA), to
determine if existing environmental regulations need to be retained or
improved, we previously recommended that EPA study the actual costs and
benefits of such regulations.11
In the past, regulatory agencies have cited numerous barriers to their
efforts to establish results-oriented goals and measures. These barriers
included problems in obtaining data to demonstrate results, accounting for
factors outside of the agency's control that affect results, and dealing
with the long time periods often needed to see results. Our prior work
discussed best practices for addressing challenges to measuring the
results of regulatory programs. In particular, to address the challenge of
discerning the impact of a federal program, when other factors also affect
results, we suggested agencies "establish a rationale of how the program
delivers results." Establishing such a rationale involves three related
practices: (1) taking a holistic or "systems" approach to the problem
being addressed, (2) building a program logic model that described how
activities translated to outcomes, and (3) expanding program assessments
and evaluations to validate the model linkages and rationale.
We have also reported on the difficulties encountered in meeting GPRA
reporting requirements for intergovernmental grant programs.12 Programs
that do not deliver a readily measurable product or service are likely to
have difficulty meeting GPRA performance measurement and reporting
requirements. Intergovernmental grant programs, particularly those with
the flexibility inherent in classic block grant design, may be more likely
to
10U.S. General Accounting Office, Managing for Results: Strengthening
Regulatory Agencies' Performance Management Practices, GAO/GGD-00-10
(Washington, D.C.: Oct. 28, 1999).
11U.S. General Accounting Office, Environmental Protection: Assessing the
Impacts of EPA's Regulations Through Retrospective Studies,
GAO/RCED-99-250 (Washington, D.C.: Sept. 14, 1999).
12U.S. General Accounting Office, Grant Programs: Design Features Shape
Flexibility, Accountability, and Performance Information, GAO/GGD-98-137
(Washington, D.C.: June 22, 1998).
Chapter 4
Challenges to GPRA Implementation Persist
have difficulty producing performance measures at the national level and
raise delicate issues of accountability. Although most flexible grant
programs we reviewed reported simple activity or client counts, relatively
few of them collected uniform data on the outcomes of state or local
service activities. Collecting such data requires conditions (such as
uniformity of activities, objectives, and measures) that do not exist
under many flexible program designs, and even where overall performance of
a state or local program can be measured, the amount attributable to
federal funding often cannot be separated out.
Focus group participants also suggested that they faced challenges in
obtaining timely performance data from relevant partner organizations and
in identifying what the federal government's contribution has been to a
specific outcome. Furthermore, survey respondents provided some
corroboration for these views. Across all three of our surveys, we
estimate that roughly a quarter of all federal managers reported this
difficulty- distinguishing between the results produced by the program
they were involved with and results caused by other factors-as a
substantial hindrance. In response to a survey question about what the
federal government could do to improve its overall focus on managing for
results, one respondent noted: "Defining meaningful measures for the work
we do is extremely difficult; and even if they could be defined,
performance and accomplishment is (sic) dependent on so many factors
outside our control that it is difficult, if not impossible, to make valid
conclusions."
Timely, Useful Performance Information Not Always Available
In February 2000, we reported that intergovernmental programs pose
potential difficulties in collecting timely and consistent national
data.13 We also noted that agencies had limited program evaluation
capabilities and weaknesses in agencies' financial management capabilities
make it difficult for decision makers to effectively assess and improve
many agencies' financial performance. On the basis of our current
findings, these issues still exist. Federal managers who participated in
our focus groups cited difficulties in gathering data from state or local
entities, as well as statutory limitations regarding the nature and
breadth of data that they were permitted to collect. However, in our 2003
survey, only 27 percent of federal managers indicated that obtaining data
in time to be useful was a
13GAO/GGD-00-52.
Chapter 4
Challenges to GPRA Implementation Persist
substantial hindrance; 31 percent expressed a comparable view with regard
to obtaining valid or reliable data.
Focus group participants also noted that OMB's accelerated time frames for
reporting performance information will contribute to the challenge of
producing complete, timely information in their agencies' performance
reports. Over the past 2 fiscal years, OMB has moved the deadline for
submission of agencies' performance reports (now performance and
accountability reports) back from the statutory requirement of March 31;
for fiscal year 2003 data, the deadline is January 30, 2004. In fiscal
year 2004, these reports will be due on November 15, 2004. According to
the managers, individual agencies may work on different time frames based
partially on the population they serve or the stakeholders they must work
with, such as state or local agencies. This "one size fits all" approach
does not take such differences into account.
Additionally, OMB requires agencies to report on their performance data
quarterly; managers noted that this was particularly difficult for
outcomes that may be achieved over extended periods of time, such as
outcomes associated with basic science. As we have previously reported,
measuring the performance of science-related projects can be difficult
because a wide range of factors determine if and how a particular research
and development project will result in a commercial application or have
other benefits. Efforts to cure diseases or pursue space exploration are
difficult to quantify and break down into meaningful quarterly performance
measures.
Crosscutting Issues Hinder Successful GPRA Implementation
Crosscutting issues continue to be a challenge to GPRA implementation.
Mission fragmentation and program overlap are widespread across the
federal government. Moreover, addressing this challenge is essential to
the success of national strategies in areas such as homeland security,
drug control, and the environment.
We have reported that agencies could use the annual performance planning
cycle and subsequent annual performance reports to highlight crosscutting
program efforts and to provide evidence of the coordination of those
efforts. Our review of six agencies' strategic and annual performance
plans showed some improvement in addressing their crosscutting program
efforts, but a great deal of improvement is still necessary. Few of the
plans we reviewed attempted the more challenging task of discussing
planned strategies for coordination and establishing complementary
performance
Chapter 4
Challenges to GPRA Implementation Persist
goals and complementary or common performance measures. For example, SSA's
2004 performance plan makes some mention of the agency's efforts to
coordinate with other agencies to preserve the integrity of the Social
Security number as a personal identifier, but there are very few details
about this important component of its mission.
Previous GAO reports and agency managers identified several barriers to
interagency coordination. First, missions may not be mutually reinforcing
or may even conflict, making reaching a consensus on strategies and
priorities difficult. In 1998 and 1999, we found that mission
fragmentation and program overlap existed in 12 federal mission areas,
ranging from agriculture to natural resources and the environment.
Implementation of federal crosscutting programs is often characterized by
numerous individual agency efforts that are implemented with little
apparent regard for the presence of related activities. Second, we
reported on agencies' interest in protecting jurisdiction over missions
and control over resources.14 Focus group participants echoed this
concern, noting that there can be "turf battles" between agencies, where
jurisdictional boundaries, as well as control over resources, are hotly
contested. Finally, incompatible procedures, processes, data, and computer
systems pose difficulties for agencies to work across agency boundaries.
For example, we reported how the lack of consistent data on federal
wetlands programs implemented by different agencies prevented the
government from measuring progress toward achieving the governmentwide
goal of no net loss of the nation's wetlands.15
14U.S. General Accounting Office, Managing for Results: Barriers to
Interagency Coordination, GAO/GGD-00-106 (Washington, D.C.: Mar. 29,
2000).
15U.S. General Accounting Office, Wetlands Overview: Problems With Acreage
Data Persist, GAO/RCED-98-150 (Washington, D.C.: July 1, 1998) and
Results-Oriented Management: Agency Crosscutting Actions and Plans in
Border Control, Flood Mitigation and Insurance, Wetlands, and Wildland
Fire Management, GAO-03-321 (Washington, D.C.: Dec. 20, 2002).
Chapter 4
Challenges to GPRA Implementation Persist
We have previously reported and testified that GPRA could provide OMB,
agencies, and Congress with a structured framework for addressing
crosscutting program efforts.16 OMB, for example, could use the provision
of GPRA that calls for OMB to develop a governmentwide performance plan to
integrate expected agency-level performance. Unfortunately, this provision
has not been fully implemented. OMB issued the first and only such plan in
February 1998 for fiscal year 1999. In our review of the plan,17 we found
that it included a broad range of governmentwide management objectives and
a mission-based presentation of key performance goals based on agency
performance plans and the plan's framework should ultimately allow for a
cohesive presentation of governmentwide performance. However, the specific
contents of this initial plan did not always deliver an integrated,
consistent, and results-oriented picture of fiscal year 1999 federal
government performance goals.
OMB officials we interviewed at the time stressed that developing the
governmentwide plan was viewed as an essential and integral component of
the President's budget and planning process. From OMB's perspective, both
the plan and the budget submission were intended to serve as communication
tools for a range of possible users. In their opinion, the plan added
value by reflecting a governmentwide perspective on policy choices made
throughout the budget formulation process. OMB acknowledged that the plan
itself did not serve to change the process through which decisions on
government priorities were made, but enhanced it by placing a greater
emphasis on results. As one official described it, the governmentwide
performance plan was a derivative document, reflecting the budget and
management decisions made throughout the process of formulating the
President's budget submission. However, we found that focusing broadly on
governmentwide outcomes should be a central and distinguishing feature of
the federal government performance plan. To be most effective and
supportive of the purposes of GPRA, the governmentwide plan must be more
than a compilation of agency-level plans; integration, rather than
repetition, must be its guiding principle.
16GAO/GGD-00-106 and U.S. General Accounting Office, Results-Oriented
Government: Using GPRA to Address 21st Century Challenges, GAO-03-1166T
(Washington, D.C.: Sept. 18, 2003).
17U.S. General Accounting Office, The Results Act: Assessment of the
Governmentwide Performance Plan for Fiscal Year 1999, GAO/AIMD/GGD-98-159
(Washington, D.C.: Sept. 8, 1998).
Chapter 4
Challenges to GPRA Implementation Persist
OMB has not issued a distinct governmentwide performance plan since fiscal
year 1999. Most recently, the President's fiscal year 2004 budget focused
on describing agencies' progress in addressing the PMA and the results of
PART reviews of agency programs. Although such information is important
and useful, it does not provide a broader and more integrated perspective
of planned performance on governmentwide outcomes. Additionally, the
fiscal year 2004 budget identified budget requests and performance
objectives by agency, such as the U.S. Department of Defense, as opposed
to crosscutting governmentwide themes. From this presentation, one could
assume that the only activities the U.S. government planned to carry out
in support of national defense were those listed under the chapter
"Department of Defense." However, the chapter of the fiscal year 2004
budget discussing "the Department of State and International Assistance
Programs," contains a heading titled, "Countering the Threat from Weapons
of Mass Destruction." And while OMB may have a technical reason for not
classifying this task as being related to national defense or homeland
security, it is unclear that a lay reader could make that distinction. The
fiscal year 2005 budget also identified budget requests by agency, not by
crosscutting theme. Without such a governmentwide focus, OMB is missing an
opportunity to assess and communicate the relationship between individual
agency goals and outcomes that cut across federal agencies and more
clearly relate and address the contributions of alternative federal
strategies. The governmentwide performance plan also could help Congress
and the executive branch address critical federal performance and
management issues, including redundancy and other inefficiencies in how we
do business. It could also provide a framework for any restructuring
efforts.
A strategic plan for the federal government, supported by key national
indicators to assess the government's performance, position, and progress,
could provide an additional tool for governmentwide reexamination of
existing programs, as well as proposals for new programs. If fully
developed, a governmentwide strategic plan could potentially provide a
cohesive perspective on the long-term goals of the federal government and
provide a much needed basis for fully integrating, rather than merely
coordinating, a wide array of federal activities. Successful strategic
planning requires the involvement of key stakeholders. Thus, it could
serve as a mechanism for building consensus. Further, it could provide a
vehicle for the President to articulate long-term goals and a road map for
achieving them. In addition, a strategic plan could provide a more
comprehensive framework for considering organizational changes and making
resource decisions.
Chapter 4
Challenges to GPRA Implementation Persist
Developing a strategic plan for the federal government would be an
important first step in articulating the role, goals, and objectives of
the federal government. It could help provide critical horizontal and
vertical linkages. Horizontally, it could integrate and foster synergies
among components of the federal government as well as help to clarify the
role of the federal government vis-`a-vis other sectors of our society.
Vertically, it could provide a framework of federal missions and goals
within which individual federal agencies could align their own missions
and goals that would cascade down to individual employees. The development
of a set of key national indicators could be used as a basis to inform the
development of the governmentwide strategic and annual performance plans.
The indicators could also link to and provide information to support
outcomeoriented goals and objectives in agency-level strategic and annual
performance plans.
Managers View Congress' Use of Performance Information as Limited
Focus group members believed that one of the main challenges to GPRA
implementation was the reluctance of Congress to use that information when
making decisions, especially appropriations decisions. This concern was
cited as a significant challenge in each of the focus groups, and was one
of the top three "challenges" in five of the eight focus groups. In some
cases, managers in our focus groups noted that this lack of usage was a
significant disincentive to doing a good job in preparing GPRA plans and
reports. Agency managers made the following criticisms regarding the
perceived lack of congressional use of performance information:
o appropriators have not bought into GPRA, so there is no incentive to do
this well,
o failure of congressional leadership in developing and using performance
measures,
o appropriators do not use performance data or tools to make decisions,
and
o GPRA does not drive public policy decisions.
Results from our survey provide some further information in support of
this view. On our 2003 survey, when we asked federal managers about the
extent to which they thought congressional committees paid attention to
agency efforts under GPRA, only 22 percent of federal managers responded
in the great to very great categories. This result was not significantly
Chapter 4
Challenges to GPRA Implementation Persist
different from the results we observed on our 2000 survey when we asked
this question about three specific types of congressional committees-
authorization, appropriation, and oversight. On the 2000 survey, only 18
percent of federal managers held a similar view concerning authorizing
committees, 19 percent for appropriations committees, and 20 percent for
oversight committees. As we noted earlier, when this item was asked in
relation to OMB, there was a significant increase in the percentage of
managers responding to a great or very great extent from 2000 to 2003. The
31 percent of managers who viewed OMB as paying attention to a great or
very great extent in 2003 was significantly higher than the 22 percent
holding a comparable view of congressional committees.
Although managers expressed these concerns about the use of this
information, a recent review by the CRS suggested that Congress uses
performance information to some extent, as evidenced by citations in
legislation and committee reports.18 For example, in the 106th Congress
(1999-2000), 42 public laws contained statutory language relating to GPRA
and performance measures, and 118 legislative reports19 contained
GPRAassociated passages. As shown in figure 18, across all three of our
surveys, only a minority of federal managers governmentwide viewed the
lack of ongoing congressional commitment for using performance information
as a hindrance to a great or very great extent.
18Congressional Research Service, Government Performance and Results Act:
Overview of Associated Provisions in the 106th Congress, (Washington,
D.C.: 2002).
19This included reports that accompanied bills passed by both the House
and Senate that were either enacted into law or vetoed by the President.
Chapter 4
Challenges to GPRA Implementation Persist
Figure 18: Percentage of Federal Managers Reporting to a Great or Very
Great Extent That a Lack of Ongoing Congressional Commitment or Support
for Using Performance Information in Making Program/Funding Decisions Is a
Hindrance
Percent
100
90
80
70
60
50
40
33 34 31
30
20
10
0 1997 2000 2003 Year
Source: GAO.
Note: Percentages are based on those respondents answering on the extent
scale.
While there is concern regarding Congress' use of performance information,
it is important to make sure that this information is initially useful.
One of GPRA's purposes is to respond to a need for accurate, reliable
information for congressional decision making. In 2000, we reported that
congressional staffs stated that they were looking for recurring
information on spending priorities within programs; the quality, quantity,
and efficiency of program operations; the populations served or regulated;
as well as programs' progress in meeting their objectives.20 For example,
learning who benefits from a program can help in addressing questions
about how well services are targeted to those most in need. Some of these
recurring needs were met through formal agency documents, such as annual
performance plans. However, some information the agencies provided did not
fully meet the congressional
20U.S. General Accounting Office, Managing for Results: Views on Ensuring
the Usefulness of Agency Performance Information to Congress, GGD-00-35
(Washington, D.C.: Jan. 26, 2000).
Chapter 4
Challenges to GPRA Implementation Persist
staffs' needs because the presentation was not clear, directly relevant,
or sufficiently detailed. For example, congressional staffs wanted to see
more direct linkages among the agencies' resources, strategies, and goals.
In other cases, the information was not readily available to the
congressional staffs, either because it had not been requested or
reported, or because staff were not informed that it was available.
As a key user of performance information, Congress also needs to be
considered a partner in shaping agency goals at the outset. For example,
through the strategic planning requirement, GPRA requires federal agencies
to consult with Congress and key stakeholders to reassess their missions
and long-term goals as well as the strategies and resources they will need
to achieve their goals. GPRA also provides a vehicle for Congress to
explicitly state its performance expectations in outcome-oriented terms
when establishing new programs or in exercising oversight of existing
programs that are not achieving desired results. Congress could use
authorizing and appropriations hearings to determine if agency programs
have clear performance goals, measures, and data with which to track
progress and whether the programs are achieving their goals. If goals and
objectives are unclear or not results oriented, Congress could use
legislation to articulate the program outcomes it expects agencies to
achieve. This would provide important guidance to agencies that could then
be incorporated in agency strategic and annual performance plans.
Chapter 5
Conclusions and Recommendations
Agenda for Achieving a Sustainable, Governmentwide Focus on Results
As we have shown in this report, in the 10 years since the enactment of
GPRA, significant progress has been made in instilling a focus on results
in the federal government. First, GPRA statutory requirements laid a
foundation for results-oriented management in federal agencies. Expert and
agency focus group participants cited the creation of this statutory
foundation as one of the key accomplishments of GPRA. Since GPRA began to
be implemented governmentwide in fiscal year 1997, we have observed
significant increases in the percentage of federal managers who reported
having results-oriented performance measures for their programs. Focus
group participants' views on whether GPRA has had a positive effect on the
federal government's ability to deliver results to the American public
were mixed. For example, the information gathered and reported for GPRA
allows agencies to make better-informed decisions, which improves their
ability to achieve results. In addition, GPRA has made the results of
federal programs more transparent to the public. Other participants stated
that while certain aspects of GPRA-related work have been positive,
agencies' ability to deliver results and public awareness of their
activities cannot be exclusively attributed to GPRA.
Second, GPRA has increased the connection between resources and results by
creating more formal linkages between agency performance goals and
objectives and the program activities in the budget. Over the first 4
years of agency efforts to implement GPRA, we observed that agencies
continued to tighten the required linkage between their performance plans
and budget requests. However, much remains to be done in this area. For
example, we have not observed notable increases in federal managers'
perceptions about their personal use of plans or performance information
when allocating resources, or about the use of performance information
when funding decisions are made about their programs. However, it should
be noted that we estimate a majority have positive perceptions about the
use of performance information to allocate resources.
Third, GPRA has provided a foundation for examining agency missions,
performance goals and objectives, and the results achieved. We have seen
improvements in the quality of agency strategic plans, annual performance
plans, and performance reports since initial efforts. However, few of the
six agencies we reviewed in this report produced GPRA planning and
reporting documents that met all of our criteria for the highest level of
quality. Most of these agencies continued to miss opportunities to present
clear pictures of their intended and actual performance results in their
GPRA plans and reports and to show how resources are aligned with actual
Chapter 5 Conclusions and Recommendations
performance results. Furthermore, most of the agencies we reviewed did not
provide a full level of confidence in the credibility of their performance
data.
Performance-based management, as envisioned by GPRA, requires transforming
organizational cultures to improve decision making, maximize performance,
and assure accountability. This transformation is not an easy one and
requires investments of time and resources as well as sustained leadership
commitment and attention. Challenges to successful implementation of GPRA
include inconsistent top leadership commitment to creating a focus on
results; an approach to setting goals and developing strategies for
achieving critical outcomes that creates individual agency stovepipes
rather than an integrated, holistic governmentwide approach; getting
federal managers to make greater use of performance information to manage
their programs and providing them authority to act that is commensurate
with their accountability for results; difficulty in establishing
meaningful measures of outcomes and assessing results of federal programs
that are carried out by nonfederal entities; and untimely performance
data.
The challenges identified in this report are not new-most have not changed
significantly since we first reported on governmentwide implementation of
GPRA. However, we have frequently reported on approaches that agencies,
OMB, and Congress could use to address the challenges. These approaches
include strengthening the commitment of top leadership to creating and
sustaining a focus on results; taking a governmentwide approach to
achieving outcomes that are crosscutting in nature; improving the
usefulness of performance information to managers, Congress, and the
public; and improving the quality of performance measures and data.
Collectively, these approaches form the agenda that federal agencies, OMB,
and Congress will need to follow to bring about a more sustainable,
governmentwide focus on results.
Strengthening Top Successfully addressing the challenges that federal
agencies face requires Leadership Commitment to leaders who are committed
to achieving results, who recognize the Creating and Sustaining importance
of using results-oriented goals and quantifiable measures, and
who integrate performance-based management into the culture and day-to-
Results-Oriented Cultures day activities of their organizations. Top
leadership must play a critical role in creating and sustaining
high-performing organizations. Without the clear and demonstrated
commitment of agency top leadership-both
Chapter 5 Conclusions and Recommendations
political and career-organizational cultures will not be transformed, and
new visions and ways of doing business will not take root.
To be positioned to address the array of challenges faced by our national
government, federal agencies will need to transform their organizational
cultures so that they are more results oriented, customer-focused, and
collaborative. Leading public organizations here in the United States and
abroad have found that strategic human capital management must be the
centerpiece of any serious change management initiative and efforts to
transform the cultures of government agencies. Performance management
systems are integral to strategic human capital management. Such systems
can be key tools to maximizing performance by aligning institutional
performance measures with individual performance and creating a "line of
sight" between individual and organizational goals. Leading organizations
use their performance management systems as a key tool for aligning
institutional, unit, and employee performance; achieving results;
accelerating change; managing the organization day to day; and
facilitating communication throughout the year so that discussions about
individual and organizational performance are integrated and ongoing.1
Furthermore, achieving this cultural transformation requires people to
have the knowledge and skills to develop and use performance information
to improve program performance. Our survey data indicated a significant
relationship between those managers who reported they received training on
setting performance goals and those who used performance information when
setting or revising performance goals. However, federal agencies have not
consistently showed a commitment to investing in needed training and
development opportunities to help ensure that managers and employees have
the requisite skills and competencies to achieve agency goals.
The commitment to focusing on and using performance information needs to
extend to OMB and Congress as well. Through the administration's PMA and
PART initiatives, OMB has clearly placed greater emphasis on management
issues over the past several years. However, the focus of such oversight
needs to extend beyond the emphasis on formulating the President's Budget
to include an examination of the many challenges agencies face that may be
contributing to poor performance. In spite of the
1U.S. General Accounting Office, Human Capital: Key Principles From Nine
Private Sector Organizations, GAO/GGD-00-28 (Washington, D.C.: Jan. 31,
2000).
Chapter 5 Conclusions and Recommendations
persistent weaknesses we found in agencies' strategic plans and annual
performance plans and reports, OMB significantly reduced the scope of its
guidance to agencies on how to prepare these documents. By emphasizing a
focus on resource allocation through its PART exercise and providing less
information on how to comply with GPRA, OMB may be sending a message to
agencies that compliance with GPRA is not important. Without strong
leadership from OMB, the foundation of performance information that has
been built could deteriorate.
OMB leadership is critical to addressing the continuing challenges
presented in GPRA implementation and the transformation of the federal
government to an increasingly results-oriented culture. OMB, as the
primary focal point for overall management in the federal government, can
provide the needed impetus by providing guidance, fostering communication
among agencies, and forming intragovernmental councils and work groups
tasked with identifying potential approaches and solutions to overcoming
the persistent challenges to results-oriented management.
Congress can also play a decisive role in fostering results-oriented
cultures in the federal government by using information on agency goals
and results at confirmation, oversight, authorization, and appropriation
hearings. Consistent congressional interest in the status of an agency's
GPRA efforts, performance measures, and uses of performance information to
make decisions, will send an unmistakable message to agencies that
Congress expects GPRA to be thoroughly implemented.
We also found that timing issues may affect the development of agency
strategic plans that are meaningful and useful to top leadership. The
commitment of top leadership within agencies, OMB, and Congress is
critical to the success of strategic planning efforts. A strategic plan
should reflect the policy priorities of an organization's leaders and the
input of key stakeholders if it is to be an effective management tool.
However, GPRA specifies time frames for updating strategic plans that do
not correspond to presidential or congressional terms. As a result, an
agency may be required to update its strategic plan a year before a
presidential election and without input from a new Congress. If a new
president is elected, the updated plan is essentially moot and agencies
must spend additional time and effort revising it to reflect new
priorities. Our focus group participants, including GPRA experts, strongly
agreed that this timing issue should be addressed by adjusting time frames
to correspond better with presidential and congressional terms.
Chapter 5 Conclusions and Recommendations
Addressing Governmentwide Needs
Mission fragmentation and program overlap are widespread throughout the
federal government.2 We have noted that interagency coordination is
important for ensuring that crosscutting program efforts are mutually
reinforcing and efficiently implemented. Our review of six agencies'
strategic and annual performance plans along with our previous work on
crosscutting issues has demonstrated that agencies' still present their
goals and strategies in a mostly stovepiped manner. They have generally
not used their plans to communicate the nature of their coordination with
other agencies, in terms of the development of common or complementary
goals and objectives or strategies jointly undertaken to achieve those
goals.
We have also reported that GPRA could provide a tool to reexamine federal
government roles and structures governmentwide. GPRA requires the
President to include in his annual budget submission a federal government
performance plan. Congress intended that this plan provide a "single
cohesive picture of the annual performance goals for the fiscal year." The
governmentwide performance plan could help Congress and the executive
branch address critical federal performance and management issues,
including redundancy and other inefficiencies in how we do business. It
could also provide a framework for any restructuring efforts.
Unfortunately, this provision has not been fully implemented. Instead, OMB
has used the President's Budget to present high-level information about
agencies and certain program performance issues. The agency-byagency focus
of the budget does not provide the integrated perspective of government
performance envisioned by GPRA.
If the governmentwide performance plan were fully implemented, it could
provide a framework for such congressional oversight. For example, in
recent years, OMB has begun to develop common measures for similar
programs, such as job training. By focusing on broad goals and objectives,
oversight could more effectively cut across organization, program, and
other traditional boundaries. Such oversight might also cut across
existing committee boundaries, which suggests that Congress may benefit
from using specialized mechanisms to perform oversight (i.e., joint
hearings and special committees).
A strategic plan for the federal government, supported by key national
indicators to assess the government's performance, position, and progress,
2GAO/AIMD-97-146.
Chapter 5 Conclusions and Recommendations
could provide an additional tool for governmentwide reexamination of
existing programs, as well as proposals for new programs. If fully
developed, a governmentwide strategic plan can potentially provide a
cohesive perspective on the long-term goals of the federal government and
provide a much needed basis for fully integrating, rather than merely
coordinating, a wide array of federal activities. Successful strategic
planning requires the involvement of key stakeholders. Thus, it could
serve as a mechanism for building consensus. Further, it could provide a
vehicle for the President to articulate long-term goals and a road map for
achieving them. In addition, a strategic plan can provide a more
comprehensive framework for considering organizational changes and making
resource decisions.
Developing a strategic plan for the federal government would be an
important first step in articulating the role, goals, and objectives of
the federal government. It could help provide critical horizontal and
vertical linkages. Horizontally, it could integrate and foster synergies
among components of the federal government as well as help to clarify the
role of the federal government vis-`a-vis other sectors of our society.
Vertically, it could provide a framework of federal missions and goals
within which individual federal agencies could align their own missions
and goals that would cascade down to individual employees. The development
of a set of key national indicators could be used as a basis to inform the
development of governmentwide strategic and annual performance plans. The
indicators could also link to and provide information to support
outcomeoriented goals and objectives in agency-level strategic and annual
performance plans.
Improving Usefulness of We have found that leading organizations that
progressed the farthest to
Performance Information results-oriented management did not stop after
strategic planning and performance measurement. They applied their
acquired knowledge and data to identify gaps in their performance, report
on that performance, and finally use that information to improve their
performance to better support their missions.
Chapter 5 Conclusions and Recommendations
Performance data can have real value only if they are used to identify the
gap between an organization's actual performance level and the performance
level it has identified as its goal. Once the performance gaps are
identified for different program areas, managers can determine where to
target their resources to improve overall mission accomplishment. When
managers are forced to reduce their resources, the same analysis can help
them target reductions to keep to a minimum the impact on their
organization's overall mission.3
Under GPRA, agencies produce a single strategic plan, annual performance
plan, and annual performance report. However, there are many potential
consumers of agencies' performance information-Congress, the public, and
the agency itself. One size need not fit all. Clearly, an agency will need
more detailed information on its programs for operational purposes than
would be suitable for external audiences. Of the six agencies' performance
reports we reviewed, some of them provided useful summary tables or
information that provided overall snapshots of performance or highlighted
progress in achieving key goals. Other reports that lacked such a summary
made it difficult to assess the progress achieved.
To improve the prospect that agency performance information will be useful
to and used by these different users, agencies need to consider the
different information needs and how to best tailor their performance
information to meet those needs. For example, we have reported that,
although many information needs were met, congressional staff also
identified gaps in meeting their information needs.4 Key to addressing
these information gaps was improving communication between congressional
staff and agency officials to help ensure that congressional information
needs are understood, and that, where feasible, arrangements are made to
meet them. Improved two-way communication might also make clear what
information is and is not available, as well as what is needed and what is
not needed. This might entail the preparation of simplified and
streamlined plans and reports for Congress and other external users.
3U.S. General Accounting Office, Executive Guide: Effectively Implementing
the Government Performance and Results Act, GAO/GGD-96-118 (Washington,
D.C.: June 1, 1996).
4GAO/GGD-00-35.
Chapter 5 Conclusions and Recommendations
Another challenge that limits the usefulness of agency performance reports
is the lack of timely data on performance. For the six performance reports
we reviewed we continued to observe a significant number of goals for
which performance data were unavailable. Policy decisions made when
designing federal programs, particularly intergovernmental programs, may
make it difficult to collect timely and consistent national data. In
administering programs that are the joint responsibility of state and
local governments, Congress and the executive branch continually balance
the competing objectives of collecting uniform program information to
assess performance with giving states and localities the flexibility
needed to effectively implement intergovernmental programs.
Improving Performance Measures and Data Quality
Another key challenge to achieving a governmentwide focus on results is
that of developing meaningful, outcome-oriented performance goals and
collecting performance data that can be used to assess results.
Performance measurement under GPRA is the ongoing monitoring and reporting
of program accomplishments, particularly progress toward preestablished
goals. It tends to focus on regularly collected data on the level and type
of program activities, the direct products and services delivered by the
programs, and the results of those activities. For programs that have
readily observable results or outcomes, performance measurement may
provide sufficient information to demonstrate program results. In some
programs, however, outcomes are not quickly achieved or readily observed,
or their relationship to the program is uncertain. In such cases, more
in-depth program evaluations may be needed, in addition to performance
measurement, to examine the extent to which a program is achieving its
objectives.
Chapter 5 Conclusions and Recommendations
Given the difficult measurement challenges we have identified, it is all
the more important that agency strategic planning efforts include the
identification of the most critical evaluations that need to take place to
address those challenges. However, our previous work has raised concerns
about the capacity of federal agencies to produce evaluations of program
effectiveness.5 Few of the agencies we reviewed deployed the rigorous
research methods required to attribute changes underlying outcomes to
program activities. Yet we have also seen how some agencies have
profitably drawn on systematic program evaluations to improve their
measurement of program performance or understanding of performance and how
it might be improved.6 Our review of six agencies' strategic plans and
performance reports in this report revealed weaknesses in their
discussions of program evaluation. Most of the strategic plans lacked
critical information required by GPRA, such as a discussion of how
evaluations were used to establish strategic goals or a schedule of future
evaluations. Furthermore, two of the six performance reports did not
summarize the results of program evaluations completed that year, as
required.
Our work has also identified substantial, long-standing limitations in
agencies' abilities to produce credible data and identify performance
improvement opportunities that will not be quickly or easily resolved.7
According to our review, five of six agencies' annual performance plans
showed meaningful improvements in how they discussed the quality of
performance data. However, only DOT's performance plan and report
contained information that provided a full level of confidence in the
credibility of its performance data. In particular, the plans and reports
did not always provide detailed information on how the agencies verified
and validated their performance data.
5U.S. General Accounting Office, Program Evaluation: Agencies Challenged
by New Demand for Information on Program Results, GAO/GGD-98-53
(Washington, D.C.: Apr. 24, 1998).
6U.S. General Accounting Office, Program Evaluation: Studies Helped
Agencies Measure or Explain Program Performance, GAO/GGD-00-204
(Washington, D.C.: Sept. 28, 2000).
7GAO/GGD-00-52.
Chapter 5 Conclusions and Recommendations
Recommendations for Executive Action
To provide a broader perspective and more cohesive picture of the federal
government's goals and strategies to address issues that cut across
executive branch agencies, we recommend that the Director of OMB fully
implement GPRA's requirement to develop a governmentwide performance plan.
To achieve the greatest benefit from both GPRA and PART, we recommend that
the Director of OMB articulate and implement an integrated and
complementary relationship between the two. GPRA is a broad legislative
framework that was designed to be consultative with Congress and other
stakeholders, and allows for varying uses of performance information. PART
looks through a particular lens for a particular use-the executive budget
formulation process.
To improve the quality of agencies' strategic plans, annual performance
plans, and performance reports and help agencies meet the requirements of
GPRA, we recommend that the Director of OMB provide clearer and more
consistent guidance to executive branch agencies on how to implement GPRA.
Such guidance should include standards for communicating key performance
information in concise as well as longer formats to better meet the needs
of external users who lack the time or expertise to analyze lengthy,
detailed documents.
To help address agencies' performance measurement challenges, we recommend
that the Director of OMB engage in a continuing dialogue with agencies
about their performance measurement practices with a particular focus on
grant-making, research and development, and regulatory functions to
identify and replicate successful approaches agencies are using to measure
and report on their outcomes, including the use of program evaluation
tools. Additionally, we recommend that the Director of OMB work with
executive branch agencies to identify the barriers to obtaining timely
data to show progress against performance goals and the best ways to
report information where there are unavoidable lags in data availability.
Interagency councils, such as the President's Management Council and the
Chief Financial Officers' Council, may be effective vehicles for working
on these issues.
To facilitate the transformation of agencies' management cultures to be
more results-oriented, we recommend that the Director of OMB work with
agencies to ensure they are making adequate investments in training on
Chapter 5 Conclusions and Recommendations
performance planning and measurement, with a particular emphasis on how to
use performance information to improve program performance.
Matters for To ensure that agency strategic plans more closely align with
changes in the federal government leadership, Congress should consider
amending
Congressional GPRA to require that updates to agency strategic plans be
submitted at
Consideration least once every 4 years, 12-18 months after a new
administration begins its term. Additionally, consultations with
congressional stakeholders should be held at least once every new Congress
and interim updates made to strategic and performance plans as warranted.
Congress should consider using these consultations along with its
traditional oversight role and legislation as opportunities to clarify its
performance expectations for agencies. This process may provide an
opportunity for Congress to develop a more structured oversight agenda.
To provide a framework to identify long-term goals and strategies to
address issues that cut across federal agencies, Congress also should
consider amending GPRA to require the President to develop a
governmentwide strategic plan.
Agency Comments We provided a copy of the draft report to OMB for comment.
OMB's written comments are reprinted in appendix VIII. In general, OMB
agreed with our findings and conclusions. OMB agreed to implement most of
our recommendations, noting that these recommendations will enhance its
efforts to make the government more results oriented. OMB agreed to (1)
work with agencies to ensure they are provided adequate training in
performance management, (2) revise its guidance to clarify the integrated
and complementary relationship between GPRA and PART, and (3) continue to
use PART to improve agency performance measurement practices and share
those practices across government.
In response to our recommendation that OMB fully implement GPRA's
requirement to develop a governmentwide performance plan, OMB stated that
the President's Budget represents the executive branch's governmentwide
performance plan. However, the agency-by-agency focus of the budget over
the past few years does not provide an integrated perspective of
government performance, and thus does not meet GPRA's requirement to
provide a "single cohesive picture of the annual performance goals for the
fiscal year." In response to our matter for
Chapter 5 Conclusions and Recommendations
congressional consideration that Congress should consider amending GPRA to
require the President to develop a governmentwide strategic plan, OMB
noted that the budget serves as the governmentwide strategic plan.
However, in our opinion, the President's Budget focuses on establishing
agency budgets for the upcoming fiscal year. Unlike a strategic plan, it
provides neither a long-term nor an integrated perspective on the federal
government's activities. A governmentwide strategic plan should provide a
cohesive perspective on the long-term goals of the federal government and
provide a basis for fully integrating, rather than primarily coordinating,
a wide array of federal activities.
We provided relevant sections of the draft report to Education, DOE, HUD,
SBA, SSA, and DOT. Education and SBA did not provide any comments, while
DOT provided minor technical comments. Written comments from DOE, HUD, and
SSA are reprinted in appendixes IX, X, and XI, respectively, along with
our responses.
DOE disagreed with portions of our analyses of its 2004 Annual Performance
Plan and its 2002 Performance and Accountability Report. Our analysis of
DOE's documents was based on specific criteria (see appendixes IV and V
for details) and was characterized in relation to our reviews of the other
five agencies' documents. We modified or clarified certain
characterizations in response to DOE comments, but for the most part found
that our characterizations were appropriate.
SSA generally agreed with our observations and agreed to incorporate them
in its future planning efforts. SSA made several points of clarification
and disagreed with our observation that its performance and accountability
report does not clearly state how program evaluations were used to answer
questions about program performance and results and how they can be
improved. SSA noted that its evaluations rely on surveys, and these
surveys form the basis for its efforts to deliver high-quality service.
SSA also noted that it lists other evaluations that are of great
importance to its ongoing operations. We do not discount the usefulness of
SSA's surveys in assessing its day-to-day management of programs. Rather,
we believe that it would be helpful for SSA to clearly identify the range
of evaluations conducted and how each of them contributed to improved
program performance.
HUD noted that all of the areas we suggested for further improvement are
under consideration for improvement. However, they disagreed with us on
two observations related to the strategic plan: (1) that the link between
its
Chapter 5 Conclusions and Recommendations
long-term and intermediate goals is difficult to discern and (2) that it
did not explain how it used the results of program evaluations to update
the current plan and did not include a schedule for future evaluations. On
the basis of OMB guidance for preparing strategic plans and the criteria
we used to evaluate all six agencies' strategic plans (see app. III for
more detail), we maintain that these two observations are valid and
require further attention. HUD also disagreed with how we presented the
performance information in its summary report cards (see fig. 22). HUD
noted that many of the results were explained in the individual indicator
write-ups that followed the summary information. Our analysis of HUD's
information included qualitative aspects of how the information was
presented, such as its usefulness to inform the average reader with little
or no exposure to the subject matter, and the extent to which HUD
presented a complete summary of performance information in a user-friendly
format.
Technical comments from DOE, HUD, and SSA were incorporated, as
appropriate.
Appendix I
Objectives, Scope, and Methodology
As agreed with your offices, our objectives for this report were to
determine (1) the effect of the Government Performance and Results Act
(GPRA) over the last 10 years in creating a governmentwide focus on
results and the government's ability to deliver results to the American
public, including an assessment of changes in the overall quality of
agencies' strategic plans, annual performance plans, and annual
performance reports; (2) the challenges that agencies face in measuring
performance and using performance information in management decisions; and
(3) how the federal government can continue to shift toward a more
results-oriented focus. To meet our objectives, we collected
governmentwide data to assess the government's overall focus on results.
We conducted a governmentwide survey of federal managers, focus groups
with federal managers and GPRA experts, and interviews with top appointed
officials. We identified and reviewed previously published reports on
GPRA. Finally, we selected a sample of agencies to review for changes in
the quality of their strategic plans, performance plans, and performance
reports since their initial efforts.
We conducted our work from January through November 2003 in Washington,
D.C. in accordance with generally accepted government auditing standards.
We provided drafts of the relevant sections of this report to officials
from each of the agencies whose GPRA reports we reviewed. We also provided
a draft of this report to OMB.
Methodology for A Web-based questionnaire on performance and management
issues was
administered to a stratified random probability sample of 800 persons
fromGovernmentwide a population of approximately 98,000 mid-level and
upper-level civilian Survey managers and supervisors working in the 24
executive branch agencies
covered by the Chief Financial Officers Act of 1990 (CFO). The sample was
drawn from the Office of Personnel Management's (OPM) Civilian Personnel
Data File as of December 31, 2002, using file designators indicating
performance of managerial and supervisory functions.
Appendix I
Objectives, Scope, and Methodology
The questionnaire was designed to obtain the observations and perceptions
of respondents on various aspects of GPRA as well as such results-oriented
management topics as the presence, use, and usefulness of performance
measures, hindrances to measuring and using performance information, and
agency climate. Most of the items on the questionnaire were closedended,
meaning that depending on the particular item, respondents could choose
one or more response categories or rate the strength of their perception
on a 5-point extent scale. Almost all the items on this questionnaire were
asked in two earlier mail-out surveys. One survey was conducted between
November 1996 and January 1997 as part of the work we did in response to a
GPRA requirement that we report on implementation of the act. The other
survey was conducted between January and August 2000.1
This survey covered the same CFO Act agencies and was designed to update
the results from the two earlier surveys. Similar to the two earlier
surveys, the sample was stratified by whether the manager or supervisor
was Senior Executive Service (SES) or non-SES. The management levels
covered general schedule (GS), general management (GM), or equivalent
schedules at levels comparable to GS/GM-13 through career SES or
equivalent levels of executive service. The sample also included the same
or equivalent special pay plans that were covered in our 2000 survey,
e.g., Senior Foreign Service executives.
We sent an e-mail to members of the sample that notified them of the
survey's availability on the GAO Web site and included instructions on how
to access and complete the survey. Members of the sample who did not
respond to the initial notice were sent up to two subsequent reminders
asking them to participate in the survey. The survey was administered from
June through August 2003.
During the course of the survey, we deleted 26 persons from our sample who
had either retired, separated, died, or otherwise left the agency or had
some other reason that excluded them from the population of interest. We
received useable questionnaires from 503 sample respondents, about 65
percent of the eligible sample. The eligible sample includes 39 persons
that
1For information on the design and administration of the two earlier
surveys, see GAO/GGD97-109, GAO-01-127, and U.S. General Accounting
Office, Managing For Results: Federal Managers' Views on Key Management
Issues Vary Widely Across Agencies, GAO-01-592 (Washington, D.C.: May 25,
2001).
Appendix I
Objectives, Scope, and Methodology
we were unable to locate and therefore unable to request that they
participate in the survey.
To assess whether the views of those individuals who chose not to
participate in our survey might be different than those who did, we made
an effort to administer a brief survey over the telephone to those
individuals who still had not responded about a month or more after the
survey had been available to them despite being contacted twice after the
initial notification e-mail had been sent out. This telephone survey
consisted of four items from the full survey. There were 58 persons who
agreed to answer these questions over the telephone. This was 41 percent
of those individuals who had not responded at the time we attempted to
contact them for the purpose of asking these four questions.
We analyzed the responses of this group on the four selected items
compared to the responses received from all other respondents. Our
analyses of the items showed very few differences between nonresponders
and responders. There was no sufficient or consistent pattern of
responding that would warrant a conclusion that the views of nonresponders
were notably different than responders. The responses of each eligible
sample member who provided a useable questionnaire were subsequently
weighted in the analysis to account statistically for all the members of
the population.
The overall survey results are generalizable to the population of managers
as described above at the CFO Act agencies. All results are subject to
some uncertainty or sampling error as well as nonsampling error. As part
of our effort to reduce nonsampling sources of error in survey results, we
checked and edited (1) the survey data for responses that failed to follow
instructions and (2) the programs used in our analyses. In general,
percentage estimates in this report for the entire sample have confidence
intervals ranging from about +- 4 to +-11 percentage points at the 95
percent confidence interval. In other words, if all CFO Act agency
managers and supervisors in our population had been surveyed, the chances
are 95 out of 100 that the result obtained would not differ from our
sample estimate in the more extreme cases by more than +-11 percent.
Appendix VI shows the questions asked with the weighted percentage of
managers responding to each item.
Because a complex sample design was used in the current survey as well as
the two previous surveys and different types of statistical analyses are
being done, the magnitude of sampling error will vary across the
particular
Appendix I
Objectives, Scope, and Methodology
surveys, groups, or items being compared due to differences in the
underlying sample sizes and associated variances. The number of
participants in the current survey is only about one fifth of the number
in the 2000 survey (2,510) and slightly more than half of those in the
first survey (905). The 2000 survey was designed with a larger sample than
the other two surveys in order to provide estimates for each individual
agency as well as all the CFO Act agencies collectively. Consequently, in
some instances, a difference of a certain magnitude may be statistically
significant. In other instances, depending on the nature of the comparison
being made, a difference of equal or even greater magnitude may not
achieve statistical significance. For example, when comparing a result
from the current survey to the larger 2000 survey with its relatively
smaller confidence interval, a difference of a certain magnitude may be
significant. However, when comparing the current survey with the first
survey, that difference may not be significant given the greater
imprecision in the estimates due to both surveys' smaller sample sizes. We
note throughout the report when differences are significant at the .05
probability level.
Methodology for Focus Groups
We held a series of focus groups as one of our methods for obtaining
information about the accomplishments and challenges agencies face in
implementing and overseeing GPRA-related activities. Focus groups are a
form of qualitative research in which a specially trained leader, a
moderator, meets with a small group of people (usually 8 to 10) who are
knowledgeable about the topics to be discussed.
In all, we conducted eight focus groups-one with experts on GPRA and
performance management and seven with federal managers. For our focus
group with experts, we invited individuals from the private sector,
academia, the National Academy of Public Administration, and OMB. These
individuals were involved either in drafting GPRA, overseeing its
implementation, or studying and critiquing implementation, from outside
government. Out of the 14 experts we invited, a total of 11 attended the
focus group.
Appendix I
Objectives, Scope, and Methodology
For our focus groups with agency managers, we obtained a list of potential
participants for our focus groups by contacting all 24 CFO Act agencies
and requesting that they submit a list of candidates and their profiles
based on the following criteria: federal managers (1) in the GS-13 pay
grade and above, including members of the SES; (2) having at least 3 years
of managerial experience; (3) currently located in the Washington, D.C.,
metropolitan area; (4) having hands-on experience with GPRA or performance
management;2 and (5) representing both departments and their component
bureaus. We received profiles of candidates from all agencies; however, no
managers from the OPM chose to participate in the focus groups.
To select focus group participants, we reviewed the profiles submitted by
agencies and selected candidates with diverse experience who held a
variety of different positions within the agency in order to capture a
broad range of perspectives. For example, we invited a comptroller; a
deputy director of management, administration, and planning; budget
directors; budget officers; management analysts; and program managers;
among others. We contacted the candidates and provided them with the list
of questions to be discussed at the focus group in advance so they would
be aware of our interests and be better able to provide us, where
possible, with examples to illustrate their responses to our questions.
Out of 104 agency officials we invited, 70 participated in the focus
groups.3
2For example, candidates could be operations managers with hands-on
experience managing a federal program or agency officials directly
involved in carrying out the activities required under GPRA, such as
developing a strategic or annual performance plan or annual performance
report.
3Due to last minute circumstances, a federal manager participated via
teleconference from an agency's field office and another was unable to
attend the focus group, but mailed his answers to questions we sent to all
participants in advance of the focus groups.
Appendix I
Objectives, Scope, and Methodology
During each session, the moderator explained the scope of our work and
elaborated on how the focus groups were one of several methods we were
using to collect information relevant to our objectives. As part of the
focus group process, the moderator asked participants at each session to
identify the main accomplishments and challenges that, in their view,
could be attributed to GPRA and to mention possible solutions to these
challenges. During the sessions, we created lists of the accomplishments,
challenges, and solutions identified by group participants and posted
these lists around the meeting room. Participants were then asked to
review the lists and vote on the three most important accomplishments and
the top three challenges.4
To organize the information collected during the focus groups, we reviewed
the statements made by participants in response to our questions. We
identified related sets of statements and summarized them as a general
theme. We noted how often a theme was expressed both within and across
each focus group. We also examined the number of votes each posted
statement obtained. Our analysis focused on those themes that were
supported by statements that obtained a high number of votes and were
mentioned frequently within and across the majority of focus groups.
The focus group results discussed in this report are summary descriptions
reflecting the range of views and perceptions held by employees,
supervisors, or project managers who participated in the focus groups.
Although we cannot assume all federal managers share these views, the
extent to which certain opinions or perceptions were repeatedly expressed
or endorsed by many participants from multiple focus groups provides a
rough gauge of the significance of these views.
Methodology for Interviews with Political Appointees
To obtain an additional perspective from top political managers of federal
agencies on GPRA, we held telephone or in-person interviews with 10
highlevel officials serving under political appointments with CFO Act
agencies. Five former officials from the Clinton administration and five
serving under the current Bush administration were interviewed. For
example, we interviewed deputy secretaries, chief financial officers, and
deputy assistant secretaries for management. We asked them to provide
their perspective on the main accomplishments or other effects of GPRA,
the
4We read the list of comments to the manager who participated via
teleconference and voted on his behalf based on his preferences.
Appendix I
Objectives, Scope, and Methodology
key challenges to implementation, and possible improvements to GPRA. We
summarized the interviewees' answers and identified recurring themes or
observations for our analysis.
Methodology for Selecting Agencies to Review for Changes in the Quality of
Their Strategic Plans, Annual Performance Plans, and Annual Performance
Reports
To address how the quality of agency strategic plans, performance plans,
and performance reports have changed since their initial efforts, we
reviewed a sample of six agencies' current strategic plans, annual
performance plans, and annual performance reports and compared the results
to our findings from prior reviews of the agencies' initial efforts in
producing these documents. We did not independently verify or assess the
information we obtained from agency plans and reports. If an agency chose
not to discuss its efforts concerning elements in the plans and reports,
it does not necessarily mean that the agency is not implementing those
elements.
We selected the departments and agencies to review based on the extent to
which they collectively represented the full range of characteristics in
the following four areas: (1) agency size (small, medium, large); (2)
primary program types (direct service, research, regulatory, transfer
payments, and contracts or grants); (3) quality of fiscal year 2000
performance plan based on our previous review (low, medium, high);5 and
(4) type of agency (cabinet department and independent agency).
Based on these characteristics, we selected the following departments and
agencies:
o Department of Education (Education),
o Department of Energy (DOE),
o Department of Housing and Urban Development (HUD),
o Small Business Administration (SBA),
5GAO/GGD/AIMD-99-215. Based on how we had rated agencies' annual
performance plans on their picture of performance, specificity of
strategies and resources, and the degree of confidence that performance
information will be credible, we assigned numeric values to each agencies'
rating (e.g., clear=3, general=2, limited=1, unclear=0) and added them up
to determine overall quality of high, medium, or low. An agency's plan was
considered high quality if its score was between 7-9, a score of 5-6 was
considered medium quality, and a score of 3-4 was low. No agencies
received a score lower than 3.
Appendix I
Objectives, Scope, and Methodology
o Social Security Administration (SSA), and
o Department of Transportation (DOT).
Table 4 shows the characteristics represented by each of these agencies.
Table 4: Summary of Characteristics of Agencies Selected for Review of
Strategic Plans, Annual Performance Plans, and Annual Performance Reports
Quality of fiscal
year 2000
performance
Sizea Functions plans
Full time Low
Small (S), (L),
equivalent medium
medium Direct Transfer Grants/ (M),
(M),
Departments positions Research service Regulatory payments contracts high
large (L) (H)
Education 4,756 S X X X X X M
DOE 16,067 M X X X L
HUD 10,752 M X X X X M
DOT 135,022b L X X X X H
Agencies
SBA 4,005S XX X L
SSA 64,418L X X H
Source: GAO.
aThe size of the agencies is based on data from December 2002.
bIn March 2003, the U.S. Coast Guard and the Transportation Security
Administration (TSA) were transferred from DOT to the Department of
Homeland Security. According to the fiscal year 2005 President's Budget,
in fiscal year 2003, the Coast Guard and TSA had 43,702 and 57,324
full-time equivalent positions, respectively.
A more detailed discussion of the criteria we used to assess the quality
of the agencies' planning and reporting documents and the results of our
review is contained in appendixes III, IV, and V.
Appendix II
Focus Group Participants Agreed GPRA Provides a Framework for Federal
Agencies to Become More Results Oriented
While GPRA's goal is to make the federal government more results oriented,
work carried out in support of this effort, such as planning activities,
implementing programs, reporting on outcomes, and evaluating performance,
generally lies in the hands of federal managers. To get a better
appreciation for the main accomplishments and challenges agencies face in
implementing and overseeing GPRA-related activities, we conducted a total
of seven focus groups comprised of federal managers from 23 CFO Act
agencies, and an eighth focus group comprised of 11 experts on GPRA and
performance management and budgeting.
For our focus groups, we asked participants to discuss (1) the key
accomplishments of GPRA to date, (2) whether GPRA has created a focus on
achieving results across the federal government, (3) the effect of GPRA on
the government's ability to deliver results to the American public, (4)
the persistent and prevalent challenges agencies face in implementing and
overseeing GPRA-related activities, and (5) suggestions to address these
challenges.
We recorded the views expressed by participants and categorized them into
themes that were most commonly expressed or endorsed both within and
across the groups. The focus group results discussed in this report are
organized according to the themes we identified and are summary
descriptions reflecting the range of views and perceptions expressed by
the experts, supervisors, and project managers. A more complete discussion
of our scope and methodology can be found in appendix I.
Focus group participants indicated that GPRA has helped to make the
federal government more results oriented. However, they noted that a
number of obstacles have made GPRA implementation challenging, such as
difficulty in establishing results-oriented goals and measuring
performance, addressing frequently changing priorities resulting from
changes in administration, and lack of top leadership support for GPRA. In
all, participants generally perceive the information contained in GPRA
reports to be important and useful; however, they do not believe that
lawmakers use this information to make resource decisions or conduct
oversight. To address these problems and concerns, focus group
participants stated that, among other things, Congress should provide
guidance to agencies on how to make GPRA reports more useful, OMB should
reinforce its value as a management tool, and agencies need to commit the
resources necessary to carry out GPRA-related activities.
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
GPRA Accomplishments
Overall, focus group participants stated that GPRA has had a positive
effect on federal agencies' efforts to become more results oriented. Based
in statute, GPRA has created a framework for agencies to focus on
achieving results by requiring them to establish program goals and
objectives, develop performance indicators, and measure the extent to
which they have made progress towards achieving program goals. As a
result, federal managers noted that they have been increasingly able to
view their programs in terms of outcomes, not outputs, and have been
generally learning how to use this framework as a management tool.
Participants also attributed a series of cultural changes within federal
agencies to GPRA, where problem solving, creative thinking, and agencywide
discussions on budget and performance have become more common. The
strategic and annual performance plan and performance reports that federal
agencies are required to submit to OMB and Congress under GPRA have
increased the transparency of government activities. These documents have
also helped agencies justify their budget requests based on their
performance.
Creating a Framework in Statute and a Management Tool for Agency
Leadership
Participants agreed that GPRA created a framework in statute for federal
agencies to plan their activities in order to become more results oriented
and provided a managerial tool for program accountability. Using this
framework, agencies can develop and focus on strategies to carry out the
programs they administer; set goals and identify performance indicators
that will inform them whether or not they achieved the performance they
expected; and determine what impact, if any, their programs have had on
the American public. According to the experts in one of our focus groups,
comparing federal agencies' current mission statements contained in their
strategic plans to what they were in the past demonstrates that agencies
have done some "soul searching" to get a better sense of what their role
is (or should be) and how they can achieve it. Given that GPRA is in
statute, the use of this planning framework is likely to be sustained
within agencies.
Participants also mentioned that GPRA has encouraged federal managers to
view their programs in terms of results, not just inputs and outputs. Such
a change is important, as it has encouraged federal managers to reflect on
the statutory intent of their programs and use this framework as a
management tool for establishing accountability within their programs.
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
Cultural Change within Federal Agencies
Participants in the majority of focus groups agreed that GPRA has been a
driving force behind many cultural changes that have occurred within
federal agencies. Highlighting the focus on results, participants stated
that GPRA has stimulated a problem-solving approach within federal
agencies and encouraged them to think creatively when developing
performance indicators for their programs. GPRA has also changed the
dialogue within federal agencies; front-line managers and staff at lower
levels of the organization now discuss budget issues in connection with
performance. Similarly, experts noted that performance management and
resource investments are more frequently communicated between agency
officials and Congress than in the past. Within agencies, GPRA documents
can provide a context of missions, goals, and strategies that political
appointees can use to articulate agencies' priorities.
Increased Transparency of Government Results
Some participants agreed that GPRA has increased federal agencies' ability
to present their results to the American public, benefiting both
stakeholders and agency staff. On the one hand, GPRA reports enable
federal agencies to communicate the results of government programs and
activities to a broad public. For example, GPRA reports are available on
agencies' Web sites and provide information to OMB, Congress, and the
American public on what agencies plan to do, how they plan to do it, and,
as summarized in the performance and accountability reports, what agencies
have accomplished and how much money it cost them to do it.
Similarly, some participants agreed that GPRA allows federal employees to
see exactly how their work can produce a positive outcome, increasing
employee morale. Using information contained in GPRA reports, agency
employees can compare department goals to the results of their activities,
and see how their work contributes to these goals. For example, a focus
group participant from the Indian Health Service in California stated that
he was pleased to learn that health care indicators of some Native
American tribes had already exceeded levels originally projected by his
agency to be reached by the year 2010.
Link between Budget and Participants also agreed that the GPRA framework
has had a positive effect
Performance on agencies' ability to link their performance to their
budget. By focusing on their mission and outcomes, agencies are learning
to prioritize activities and align their resources to ensure that they
will be able to achieve results. For example, a participant stated that
the National Wild Horse and Burro
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
Program, managed by the Bureau of Land Management in the U.S. Department
of the Interior, recently developed a model for its GPRArelated work which
divided their program into discrete components, identifying the results it
could accomplish in the short- and long-term and specifying what
additional resources were needed. According to the participant, the
program received additional funding based on this needs assessment.
In addition, a few managers noted that the link between budget and
performance has given agencies an incentive to commit the resources
necessary to modernize information systems. Having the right information
on time enables agencies to integrate budget requests and performance
information in ways that are more meaningful. This information also
increases Congress's ability to make informed budget decisions based on
agency performance.
Views on Delivering Results to the American Public Were Mixed
Participants' views on whether GPRA has helped agencies deliver results to
the American public were generally mixed. Some federal managers in our
focus groups agreed that GPRA has had a positive effect on raising
awareness on many issues, and that in and of itself is a way of delivering
results. The information gathered and reported for GPRA allows agencies to
make better-informed decisions, which improves their ability to achieve
results. Of note, GPRA has allowed agencies to move towards
performance-based budgeting, which helps agencies identify resources
available to use in programs where outcomes can be achieved. For example,
programs and expenses that do not add value to the agency's mission could
be eliminated. Having performance data readily available is another key
area where GPRA has enabled agencies to deliver results to the American
public.
Other participants stated that while certain aspects of GPRA-related work
have been positive, agencies' ability to deliver results and public
awareness of their activities cannot always be exclusively attributed to
GPRA. For example, while measuring performance is a move in the right
direction, GPRA reports provide too many indicators and it is unclear how
this has led to better performance among federal agencies. A few
participants also stated that agencies deliver results in ways that the
American public does not fully recognize. For example, a participant
stated that agencies working in the area of international relations
generally lack recognition for the results of their activities, as the
American public is generally unfamiliar with the nuances of foreign policy
and all the programs the U.S.
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
government finances overseas. And while GPRA has helped these agencies
prioritize their work to achieve results, it is unclear that GPRA has
improved the visibility and understanding in the public eye of what these
agencies do because the American public does not see the results of their
work. A participant stated that research-based agencies have also used
GPRA to plan activities that benefit the American public in ways they are
not fully aware of. For example, National Aeronautics and Space
Administration's (NASA) space program includes, among other things,
predicting whether or not an asteroid will strike the earth, although on a
daily basis the American public is probably not worried about such a
rarity.
For other participants, the link between GPRA and agencies' service
delivery was not clear. Participants characterized GPRA-related activities
as time consuming, and there is no real evidence that this work has
improved their ability to deliver results to the American public. Other
participants stated that many agencies rely on grant recipients to carry
out their work, and delivering results to the American public depends, to
a large extent, on the diligence of these organizations to implement their
programs. Overall, they held the view that performance would not change
dramatically if GPRA were no longer a requirement for federal agencies.
Alternate Views on GPRA's Effect
Participants in one of our focus groups stated that GPRA, per se, had not
led federal agencies to achieve specific accomplishments. These
participants believed that managers' initiative, not the framework
established by GPRA, has been key to improving agencies' planning efforts
and focus on results. A few participants also mentioned that the results
framework established by GPRA is somewhat intangible unless managers can
use it effectively; without the adequate infrastructure to implement GPRA,
an agency's compliance with the law is simply paperwork, as GPRA does not
allow for a systematic and thorough approach to performance management.
For example, while agencies can develop performance indicators to gauge
progress towards a program goal, they often encounter problems in
collecting relevant data and measuring the agencies' contribution to a
specific outcome. In addition, agencies are not able to make changes to
the programs they manage, limiting their ability to deliver results.
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
Challenges in Implementing and Overseeing GPRA Activities
Participants stated that agencies face significant challenges in complying
with GPRA-related activities. Focus group participants also agreed
Congress does not appear to use agencies' performance information when
making budget decisions. In carrying out GPRA-related activities, managers
find it difficult to identify performance indicators and establish program
goals that are results oriented, as required by GPRA. In addition, a few
participants stated that they lack the support from senior management to
work on GPRA activities. Changes in administration also tend to disrupt
progress made by agencies in support of GPRA.
Performance Information Participants strongly believed that Congress does
not take into account
Not Used agencies' performance information when overseeing agency
activities and making budget decisions. As a result, there is a negative
effect on how agencies view their efforts in producing GPRA plans and
reports-many believe that they are less worthwhile than the effort and
resources invested. On the other hand, participants perceive the budget
process strictly as a political exercise, and it is unclear how useful
performance information can be in this context.
Complexity of Establishing Results-Oriented Goals and Measuring
Performance
Participants stated that establishing results-oriented goals and
identifying performance indicators are generally complex undertakings.
Participants agreed that they often feel as if they were trying to measure
the immeasurable, not having a clear understanding of which performance
indicators could accurately inform the agency how it is carrying out a
specific activity. And while agencies generally try to improve the
indicators from one year to another, in doing so, they generally lose
their ability to develop trend information to track progress made over
time.
Participants also mentioned that there appears to be a disconnect between
some federal programs that generally produce results over a longer time
period and GPRA's requirement that agencies report annually on their
progress towards achieving their goals. Participants stated that federal
programs, especially those that are research-based, often take years to
achieve the full scope of their work. Consequently, and for reasons that
range from lack of performance data to an absence of short-term outcomes,
it could appear as though resources invested in carrying out their
activities led to no results in the short run.
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
Focus group participants generally agreed that in cases where third
parties, such as states or localities, implement federal programs, some
agencies face challenges in obtaining timely performance data from
relevant partner organizations. In some instances, federal managers have
trouble identifying what the government's contribution has been to a
specific outcome. While the experts generally attributed this to agencies
not doing a good job at setting appropriate goals and the corresponding
measures and objectives not being clear enough, managers stated that their
lack of control over grantees and the strict reporting deadlines imposed
by GPRA were factors that worked against their efforts to deliver results.
Managing Strategically with Frequently Changing Priorities
Some participants stated that it is difficult for them to manage
strategically, given the frequently changing priorities that come with
changes in administrations. While GPRA requires an agency to develop a
strategic plan at least every 3 years to cover the following 5-year
period, participants agreed that it makes little sense to update it
shortly before a new administration is scheduled to take office. In
addition, changes in political leadership generally result in a new agenda
with new objectives. These changes force agencies to revise their plans,
management initiatives, and strategies, which translate into additional
GPRA-related work, generally characterized by focus group participants as
a burden agency staff must add to their normal work load.
Lack of Top Leadership Some participants stated that they often encounter
resistance from agency
Support leadership to endorse GPRA-related activities. In some instances,
seniorlevel support for GPRA falters or is nonexistent. Participants
attributed this to the reluctance within some federal agencies to think
about outcomes and performance. Some focus group participants stated that
in some instances high-level managers are somewhat averse to being held
accountable for the results of programs they run.
Suggestions to Address To address these challenges, focus group
participants made the following GPRA Challenges suggestions.
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
Congress Should Clarify Congressional staff should provide guidance on
agencies' GPRA
How Performance submissions-specifically, how information could be
presented in the
Information Could Be More reports to make it more useful in the
decision-making process. They could also communicate to agencies how the
performance information is being
Useful used, so that agencies do not perceive their data gathering
efforts as inconsequential. Additionally, in using performance information
to make budget decisions, Congress should consider the results achieved by
agencies in addition to results not achieved.
The Administration and Agencies should embrace GPRA as a management tool,
not just an external OMB Need to Reinforce the requirement that is
separate from their day-to-day activities. To this end, Value of GPRA as a
the administration and OMB need make sure agency officials understand
Management Tool how GPRA can be further integrated with other management
initiatives.
Agency Guidance on GPRA OMB's guidance to agencies on how to implement
GPRA should recognize Should Recognize Diversity that one-size-fits-all
approaches are unlikely to be useful. OMB should also of Federal Agencies
afford agencies some flexibility by simplifying the reporting process. For
example, some participants believed that not everything needed to be
measured. OMB should make exceptions for unique situations and programs,
e.g., science programs, and it could consider providing multiyear funding
for them.
OMB Should Publish a OMB should commit to regularly publishing a
governmentwide Governmentwide performance report that would articulate the
government's Performance Report accomplishments to the public. It would
also be useful if it singled out
higher-performing programs so agencies could use them as models to guide
their planning efforts.
Agencies Need to Obtain Agency leadership needs to ensure that staff
members have the necessary and Commit Resources to resources to work on
GPRA-related activities. In addition, they need to Carry Out GPRA-Related
invest resources to train staff, including political appointees, on GPRA
and Activities the benefits of linking budget to performance.
Appendix II
Focus Group Participants Agreed GPRA
Provides a Framework for Federal Agencies
to Become More Results Oriented
Timing of GPRA Reports Under GPRA, agencies are to update their strategic
plans every 3 years. Should Take into Account However, this effort can be
wasted. Given that federal administrations Changes of Administration
generally span at least 4 years, participants suggested that the required
update be changed to every 4 years to correspond with new presidential
terms.
Agencies Should Share Agencies should collaborate more to develop
strategies to address difficult Experiences on How to issues, such as how
to identify performance indicators and measure agency Address Common
Problems contributions to specific outcomes. It would be useful if more
agencies
created structured forums for managers to share experiences, talk about
effective practices, and share solutions.
Appendix III
Observations on Agencies' Strategic Plans
Under GPRA, strategic plans are the starting point and basic underpinning
for results-oriented management. One of our objectives was to assess the
changes in the overall quality of agencies' goals, strategies, and data
articulated in their strategic plans. To meet this objective, we
judgmentally selected six agencies-Education, DOE, HUD, SBA, SSA, and
DOT-using criteria such as agency size, primary program types, and
previous GAO reviews. To assess the overall quality and improvements made
to the agencies' strategic plans, we relied on requirements contained in
GPRA and accompanying committee report language,1 guidance to agencies
from OMB for developing strategic plans,2 previous GAO reports and
evaluations,3 and our knowledge of agencies' operations and programs. In
conducting our reviews, we compared our assessments of agencies' current
strategic plans to our assessments of draft plans from fiscal year 1997.4
A more detailed discussion of our scope and methodology can be found in
appendix I.
Required Elements of Agency Strategic Plans
GPRA requires an agency's strategic plan to contain six key elements:
1. A comprehensive agency mission statement. The agency mission statement
should concisely summarize what the agency does, as required by law,
presenting the main purposes for all its major functions and operations.
According to OMB guidance issued in 2002, a mission statement is brief,
defining the basic purpose of the agency, and corresponds directly with an
agency's core programs and activities. The program goals should flow from
the mission statement as well. The federal government's adaptive responses
over time to new needs and problems have contributed to fragmentation and
overlap in a host of program areas, such as food safety, employment
training, early childhood development, and rural development, which could
limit the
1Government Performance and Results Act of 1993, Committee on Governmental
Affairs, United States Senate, S. Rpt. No. 58, 103d Cong. 1st Sess.
(1993).
2Office of Management and Budget, Circular No. A-11, Part 6, Preparation
and Submission of Strategic Plans, Annual Performance Plans, and Annual
Program Performance Reports
(Washington, D.C.: June 2002).
3U.S. General Accounting Office, Agencies' Strategic Plans Under GPRA: Key
Questions to Facilitate Congressional Review, GAO/GGD-10.1.16 (Washington,
D.C.: May 1, 1997).
4GAO/HEHS-97-176R; GAO/RCED-97-199R; GAO/RCED-97-208R; GAO/HEHS-97-179R;
GAO/RCED-97-205R; and GAO/RCED-97-224R.
Appendix III Observations on Agencies' Strategic Plans
overall effectiveness of the federal effort. The mission statement helps
to distinguish agencies' roles from one another and reduce the overlap and
identify areas needing coordination and collaboration.
2. Agencywide long-term goals and objectives. General goals and
objectives-or strategic goals-explain what results are expected from the
agency's major functions and when to expect those results. Thus, such
goals are an outgrowth of the mission and are very often results oriented.
OMB guidance states that the goals should be defined in a manner that
allows a future assessment to be made on whether the goal was or is being
achieved. General goals should predominately be outcomes, and are
long-term in nature.
3. Approaches or strategies to achieve goals and objectives. Strategies
help in aligning an agency's activities, core processes, and resources to
support achievement of the agency's strategic goals and mission. Under
GPRA, strategies are to briefly describe the operational processes, staff
skills, and technologies, as well as the human, capital, information, and
other resources needed. According to OMB guidance, descriptions should be
brief, but more detailed data should be provided if a significant change
in a particular means or strategy would be essential for goal achievement.
In addition, the plan should summarize agencies' efforts to provide
high-quality and efficient training and skill improvement opportunities
for employees. As we have reported previously, agencies' planning
processes should support making intelligent resource allocation decisions
that minimize, to the extent possible, the effect of funding reductions on
mission accomplishment.
4. A description of the relationship between long-term and annual goals.
Under GPRA, agencies' long-term strategic goals and objectives are to be
linked to their annual performance plans and the day-to-day activities of
their managers and staff. Without this linkage, Congress may not be able
to judge whether an agency is making progress toward achieving its
long-term goals. OMB guidance states that an updated and revised strategic
plan should briefly outline (1) the type, nature, and scope of the
performance goals being included in annual performance plans and (2) how
these annual performance goals relate to the longterm, general goals and
their use in helping determine the achievement of the general goals.
5. An identification of key external factors. Identification of key
factors, external to the agency and beyond its control that could
Appendix III Observations on Agencies' Strategic Plans
significantly affect the achievement of the strategic goals, are important
for Congress or the agencies to judge the likelihood of achieving the
strategic goals and actions needed to better meet those goals. Such
external factors could include economic, demographic, social,
technological, or environmental factors. Information on these factors can
be useful for goal setting and also for explaining results in the agency's
annual performance reports, including, when applicable, the reasons annual
performance goals were not met. According to OMB guidance, if key factors
cannot be identified, a statement of explanation should be included in the
plan.
6. A description of program evaluations. Finally, strategic plans should
include a description of completed program evaluations that were used in
developing the strategic plan, and a schedule for future program
evaluations. Program evaluations can be a potentially critical source of
information for Congress and others in ensuring the validity and
reasonableness of goals and strategies, as well as for identifying factors
likely to affect performance. Such information can also be useful in
explaining results in an agency's annual performance report, including,
when applicable, the reasons annual performance goals were not met, and
identifying appropriate strategies to meet unmet goals. Program
evaluations are defined in the act as objective and formal assessments of
the results, impact, or effects of a program or policy. The evaluations
include assessments of the implementation and results of programs,
operating policies, and practices.
In addition to the six key elements, OMB guidance also states that
agencies participating in crosscutting programs should describe in their
strategic plans how the programs are related and how coordination will
occur to support common efforts. Uncoordinated program efforts can waste
scarce funds, confuse and frustrate program customers, and limit the
overall effectiveness of the federal effort. OMB guidance also states that
the strategic plan should include a brief description of any steps being
taken to resolve mission-critical management problems. One purpose of GPRA
is to improve the management of federal agencies. Therefore, it is
particularly important that agencies develop strategies to address
management challenges that threaten their ability to meet long-term
strategic goals as well as this purpose of GPRA.
As shown in table 5, the majority of agencies have made progress in
addressing the required elements of strategic planning under GPRA.
Appendix III Observations on Agencies' Strategic Plans
Table 5: Agencies' Progress in Addressing Required Elements of Strategic
Planning under GPRA Element included in agency strategic plan? Agency strategic
plans Plan year
Mission statement Long-term goals Strategies Relationship between longterm
goals and annual goals External factors Evaluations
Department of Education 1997 X X X X X
2002XXX X X
Department of Energy 1997 X X X
2003aX XX X X
Department of Housing and 1997 X
Urban Development
2003XXX X X
Small Business Administration 1997 X X X X
2001bX XX X X
Social Security Administration 1997 X X X X X X
2003X XX X XX
Department of Transportation 1997 X X X
2003aX XX X X
Sources: GAO/GGD-10.1.16; GAO/HEHS-97-176R; GAO/RCED-97-199R;
GAO/RCED-97-208R; GAO/HEHS-97-179R; GAO/RCED-97-205R; GAO/RCED-97-224R;
and GAO analysis of U.S. Department of
Education, Office of the Deputy Secretary, Planning and Performance
Management Service, U.S. Department of Education Strategic Plan 2002-2007
(Washington, D.C.: 2002); Department of Energy, 2003
Strategic Plan (Draft) (Washington, D.C.: 2003); Department of
Transportation, U.S. Department of Transportation Draft Strategic Plan for
Fiscal Years 2003-2008 (Washington, D.C.: 2003); Social Security
Administration, Social Security Administration Strategic Plan 2003-2008
(Washington, D.C.: 2003); U.S. Department of Housing and Urban
Development, Strategic Plan FY 2003-FY 2008 (Washington, D.C.:
2003); and Small Business Administration, SBA Strategic Plan, FY 2001-FY
2006 (Washington, D.C.: 2000).
aThe 2003 plans for DOE and DOT were in draft form during the time of our
review.
bAt the time of our review, the most recent SBA strategic plan was for
fiscal years 2001-2008. SBA released a new strategic plan for fiscal years
2003-2008 in October 2003.
The remainder of this appendix discusses our observations on how the
quality of each of the agencies' strategic plans we reviewed has changed
since the agencies submitted their first draft strategic plans in 1997. We
did not independently verify or assess the information we obtained from
agency strategic plans. If an agency chose not to discuss its efforts
concerning elements in the strategic plan, it does not necessarily mean
that the agency is not implementing those elements.
Appendix III Observations on Agencies' Strategic Plans
Observations on Changes in the Quality of Education's Strategic Plan
In our review of Education's June 1997 draft strategic plan,5 we found
that the plan generally complied with GPRA and included all but one of the
six elements required by GPRA; it did not discuss how Education's
long-term goals and objectives would be related to its annual performance
goals. Also, we observed that the plan presented a logical, fairly
complete description of how Education intended to achieve its mission, but
a few areas could have been improved. In comparison, Education's 2002-2007
strategic plan has improved on several areas we identified in our 1997
review. However, we still found areas where Education could improve. Table
6 summarizes these findings.
Table 6: Education's Progress in Addressing Required Elements of Strategic
Planning under GPRA
Element of strategic planning Included in initial draft strategic plan
Included in current strategic plan
Mission statement Yes. Mission statement Yes. Mission statement is
clearly and briefly outcome
explained why the agency oriented, comprehensive, covers
exists, what the all of the
agency does, and how it agency's functions and
performs its work. activities, and is the
same as in the 1997 draft plan.
Long-term goals Yes. These goals were related to the Yes. The goals have
changed, but are mission and were results oriented, but did related to
each other and the mission and not appear to reflect civil rights
enforcement are results oriented. They now additionally and monitoring
responsibilities. reflect civil rights enforcement and monitoring
responsibilities.
Strategies Yes. The plan outlined strategies to achieve Yes. The plan
includes strategies that are goals overall and to hold managers linked to
the goals. Several strategies relate accountable for achieving objectives,
and to resource alignment to achieve outcomes, generally described some of
its resource but the actual resources required are not requirements
throughout the plan. always specified.
Relationship between long-term goals and No. Education did not discuss the
Yes. Plan contains the annual performance
annual goals relationship between its strategic plan goals measures and
targets, which represent the and those to be included in its annual plan,
annual performance goals, and aligns them but indicated this would be done
once its with long-term goals, with which they have a annual plan was
prepared. logical relationship.
External factors Yes. The plan generally described factors Yes. The plan
adequately describes outside program scope and responsibilities external
factors that could affect achieving that could negatively affect
Education's its goals and they are linked to the particular ability to
achieve goals, but factors were not goals. The plan also briefly discusses
directly linked to particular goals. strategies to ameliorate the effects
of a number of these factors.
5GAO/HEHS-97-176R.
Appendix III Observations on Agencies' Strategic Plans
(Continued From Previous Page)
Element of strategic planning Included in initial draft strategic plan
Included in current strategic plan
Evaluations Yes. Education said it would provide No. Because of the
comprehensive detailed descriptions of supporting revamping of Education's
strategic plan in evaluations once it consulted with Congress, 2002, its
program evaluation plan was completed the strategic plan, and agreed on
completely restructured and was not performance indicators. The plan
indicated released until it was included in the 2004 a commitment to using
evaluations, listing annual plan,a which contains information on
evaluations it intended to use to develop its new directions for program
evaluation. sound measures, but did not describe the evaluations.
Sources: GAO/HEHS-97-176R; U.S. Department of Education, Office of the
Deputy Secretary, Planning and Performance Management Service, U.S.
Department of Education Strategic Plan 2002-2007 (Washington, D.C.: 2002).
aAccording to OMB Circular No. A-11, Part 6, Preparation and Submission of
Strategic Plans, Annual Performance Plans, and Annual Program Performance
Reports, June 2002, general goals, which are multiyear and long term, are
synonymous with general objectives, and either term can be used. The
objectives in Education's strategic plan are such multiyear, long-term
objectives, and are referred to in our report as the agency's "long-term
goals." OMB's Circular A-11 also indicates that some agencies include
strategic goals in their strategic plan, which represent overarching
statements of aim or purpose whose achievement cannot be determined and
which can be used to group outcome or output goals. Education's strategic
goals meet this description.
Current Strategic Plan Strengths and Improvements from Fiscal Year 1997
Draft Strategic Plan
Education's current mission, "to ensure equal access to education and to
promote educational excellence throughout the nation," is the same
comprehensive, outcome-oriented mission that was included in its 1997
draft plan. All of Education's functions and activities are covered by it.
The plan's long-term goals6 are expressed so as to allow Education and
Congress to assess whether they are being achieved. Moreover, in contrast
to findings in our review of Education's 1997 Draft Strategic Plan,
Education's civil rights responsibilities-including enforcing five civil
right statutes that ensure equal educational opportunity for all students,
regardless of race, color, national origin, sex, disability, or age-appear
to be reflected, at least in part, in the current plan's goals. For
example, one long-term goal is to reduce the gaps in college access and
completion among student populations differing by race/ethnicity,
socioeconomic status, and disability while increasing the educational
attainment of all. Another is to enhance the literacy and employment
skills of American adults. Under the latter, the plan includes a strategy
to work with state vocational rehabilitation agencies to ensure
implementation of standards that will assist individuals with disabilities
in obtaining high-quality
6U.S. Department of Education, Office of the Deputy Secretary, Strategic
Accountability Service, U.S. Department of Education FY 2004 Annual Plan
(Washington, D.C.: March 2003). This document represents Education's
performance plan for GPRA and will be referred to henceforth as the
"annual plan."
Appendix III Observations on Agencies' Strategic Plans
employment outcomes. As in the past, some goals are targeted at results
for which Education has limited direct control and are, instead, greatly
influenced by third parties. However, Education recognizes this situation
and shows its intent to work closely with its partners.
Education's current plan provides some information on linking results and
day-to-day activities within the department. For example, the plan says
employees will be held accountable for implementation and success from top
to bottom and senior officers will have performance contracts linked to
the plan and be recognized for achieving results. In addition, the
strategy to foster a customer service orientation by ensuring that states,
districts, and other partners receive timely responses to inquiries; to
assign senior officers to develop relationships with individual states;
and to create a customer support team to respond to issues, seems
logically linked to the day-to-day activities of managers and staff.
However, the link between results and day-to-day activities is not
apparent for most of the goals and their strategies.
The current plan includes several annual performance measures that are at
least related to how well information technology is supporting strategic
and program goals, as required by the Clinger-Cohen Act. For example, for
its long-term goal to modernize the Federal Student Assistance (FSA)
Programs and reduce their high-risk status, the plan contains a measure on
the integration of FSA processes and systems that work together to support
FSA program delivery functions. Commendably, Education includes some goals
related to reducing agency program unintended negative effects, such as a
goal and related measure to reduce the data collection and reporting
burden.
Education's current plan shows great improvement on recognizing and
addressing external factors. Beyond adequately describing external factors
that could affect achieving its goals and directly linking these factors
to particular goals, the plan also briefly describes strategies to
ameliorate the effects of a number of factors. For example, for an
external factor on teacher certification under a goal on reading, the plan
says that Education "will work with the states and national accreditation
bodies to encourage the incorporation of research-based reading
instruction into teacher certification requirements." In addition, the
plan includes strategies indicating that Education monitors some internal
factors that could affect achievement of long term goals. Moreover, the
plan recognizes the overarching critical external factor for
Education-that it depends greatly on third parties who often control the
results the department is trying to
Appendix III Observations on Agencies' Strategic Plans
achieve-and its strategies related to the goal-specific external factors
often reflect this.
In our 1997 assessment of Education's draft plan, we commented that
although the plan identified several management challenges the department
would face in the coming years, it provided little detail about them and
how they would be addressed. In our January 2001 performance and
accountability series, we identified four department-specific and two
governmentwide challenges that we said Education needed to meet.7
Education's current strategic plan includes some goals, measures, and
strategies that could be used to address these challenges. For example,
Education's goal to "develop and maintain financial integrity and
management and internal controls" and this goal's strategies and measures
are directly related to the management challenge we identified on
improving financial management. One of the measures under this goal is
"the achievement of an unqualified audit opinion," an important issue in
our identification of financial management weaknesses as a challenge in
2001, and Education received an unqualified audit opinion in early 2003.
Moreover, the current strategic plan includes goals and strategies meant
to improve Education's strategic human capital management and strengthen
information technology security, two important governmentwide high-risk
areas we identified in 2001.
Critical Strategic Planning In our report on Education's 1997 draft
strategic plan, we found that, Issues Needing Further although the
department had numerous crosscutting programs and Improvement activities,
the plan had identified key interagency activities for some
programs but not for others. For example, we said that the plan did not
identify or discuss activities for postsecondary programs for which the
Higher Education Act of 1965 required coordination. Education's current
plan includes an appendix, which highlights collaborative initiatives
under each of the department's strategic goals, with some activities
related to
7U.S. General Accounting Office, Performance and Accountability Series:
Major Management Challenges and Program Risks, Department of Education,
GAO-01-245 (Washington, D.C.: January 2001). The January 2001 assessment
was the last time we assessed the Department of Education under our
Performance and Accountability Series before the release of the agency's
2002-2007 Strategic Plan. We further reported in October 2002 on how
Education and other agencies reported responding to their management
challenges and program risks. (U.S. General Accounting Office, Performance
and Accountability: Reported Agency Actions and Plans to Address 2001
Management Challenges and Program Risks, GAO-03-225 (Washington, D.C.:
Oct. 31, 2002).
Appendix III Observations on Agencies' Strategic Plans
postsecondary education, including most of those mentioned in our 1997
report. However, as the plan states, the appendix presents a brief
overview of the highlights of some of its collaborative initiatives with
other agencies.
In our 1997 review, we stated that some resource requirements were
described throughout the plan. In the current plan, while a number of
strategies under the department's strategic goal to establish management
excellence are related to aligning resources to achieve outcomes, the
actual resources required-such as human, capital, and information-are not
always specifically indicated. The exception is for information resources,
for which a number of strategies discuss the information resources that
will be required to address the related goal. For example, under the goal
to develop and maintain financial integrity and management and internal
controls, the plan says that Education will "implement a new financial
system capable of producing timely and reliable financial data and
reconcile systems to the general ledger." Moreover, while the plan
stresses accountability throughout, it only refers to providing the
authority needed to achieve results once.8 In addition, consideration of
alternate strategies for achieving goals is not discussed.
In our 1997 review, we reported that Education said it would provide
detailed descriptions of supporting evaluations once it consulted with
Congress, completed the strategic plan, and agreed on performance
indicators. The draft plan indicated Education's commitment to using
evaluations and listed evaluations and strategies it intended to use to
develop sound measures, but did not describe the evaluations. The current
plan does not include descriptions of supporting evaluations or a future
program evaluation schedule. According to Education officials, this was
not done because, with the complete revamping of the strategic plan based
on passage of the No Child Left Behind Act of 2001,9 Education was set on
a course to totally restructure its evaluation program, but could not do
so in time to include it in the strategic plan. Consequently, Education
instead included information about its new directions for program
evaluation studies and a schedule of evaluations in its 2004 annual plan.
The schedule, however, lacked a timetable, except for stating whether the
evaluations
8Within one of its strategies, the plan states that "managers will be
given the freedom to manage and will be held accountable for results."
9Pub. L. No. 107-110, January 8, 2002. The No Child Left Behind Act of
2001 is a reauthorization of the Elementary and Secondary Education Act,
one of the major pieces of authorizing legislation for the Department of
Education.
Appendix III Observations on Agencies' Strategic Plans
were new or continuing. The current strategic plan indicates, in some
cases, the use or planned use of program evaluation findings to develop or
revise components of the plan. For example, for long-term goals on various
types of academic achievement, data from the National Assessment of
Educational Progress were identified as having been used to set related
targets.
Observations on Changes in the Quality of DOE's Strategic Plan
Overall, DOE's draft 2003 strategic plan meets the required elements of
GPRA and has improved greatly over its 1997 draft strategic plan, as shown
in table 7. DOE made improvements to its plan by establishing
resultsoriented and measurable objectives, and addressing elements that
were not included in the department's 1997 plan, such as reporting on
external factors. Although DOE has shown improvement, a few elements in
the plan could still be enhanced. For instance, further improvement could
be made to the mission statement so that it better addresses the
department's major activities, and additional information could be
included in DOE's strategies to achieve its goals, such as management
accountability.
Table 7: DOE's Progress in Addressing Required Elements of Strategic
Planning under GPRA
Element of strategic planning Included in initial draft strategic plan
Included in current draft strategic plan
Mission statement Yes. DOE's mission was results oriented, Yes. DOE's
mission is results oriented and met a public need, and covered the
agency's meets a public need, but it does not address major activities.
the department's major activities related to energy supply and
conservation.
Long-term goals Yes. Long-term goals Yes. Long-term goals cover the
covered mission and mission and
major functions of the major functions of the agency.
agency. Goals and Objectives,
objectives were results referred to as intermediate goals,
oriented but not are results
measurable. oriented and measurable.
Strategies Yes. The plan included strategies and Yes. Strategies to
achieve each goal are measures to evaluate the results of the included in
the plan, along with intermediate strategies, but was missing information
on goals to measure success, but information linkages to day-to-day
activities, and the on linkages to day-to-day activities, and the extent
to which managers have the extent to which managers have the knowledge,
skills, and abilities to implement knowledge, skills, and abilities to
implement the strategies. the strategies, is not included.
Relationship between long-term goals and No. Relationship between the
long-term Yes. The draft plan provides a brief annual goals goals and
annual performance goals was description of the relationship between the
missing in the draft plan. long-term strategic goals and the annual
performance goals.
Appendix III Observations on Agencies' Strategic Plans
(Continued From Previous Page)
Element of strategic Included in initial draft Included in current draft
planning strategic plan strategic plan
External factors No. Key external factors Yes. Key external factors
were not and how they
addressed in the draft could affect the ability to
plan. achieve each goal
are identified in the draft
plan.
Evaluations No. The impact of program No. The impact of program
evaluations on evaluations on
the development of the development of
strategic goals was not strategic goals is not
included in the draft
plan. discussed thoroughly.
Sources: GAO/RCED-97-199R and Department of Energy, 2003 Strategic Plan (Draft),
(Washington, D.C.: 2003).
Strategic Plan Strengths and Improvements from Fiscal Year 1997 Plan
In its 2003 draft strategic plan, DOE has made significant improvements in
developing measurable objectives, referred to in its plan as intermediate
goals. GAO's review of DOE's 1997 draft strategic plan found that
objectives related to DOE's long-term goals were stated in ways that would
make it difficult to measure whether they were being achieved. In the 2003
draft plan, the objectives are stated in ways that will enable DOE to
measure its progress in achieving goals. For example, to meet the goal of
enhancing energy security through various means, one of DOE's affiliated
objectives is to ensure that throughout DOE's 25-year planning period, the
Strategic Petroleum Reserve is ready to supply oil at a sustained rate of
4.3 million barrels per day for 90 days within 15 days notice by the
President.
In addition, DOE has improved its draft 2003 strategic plan by including
elements that were not included in its 1997 draft plan. These elements
consisted of (1) identifying external factors and (2) describing the
relationship between long-term and annual goals. For each of its long-term
goals, DOE identified external factors, such as reduced funding, lack of
scientific talent, and unpredictable technological developments, that
could affect its ability to achieve its goals. The strategic plan also
included a description of the relationship between the long-term strategic
goals and the annual performance goals. The plan included a diagram that
showed an example of a strategic goal, its associated objectives, and how
they are related to the annual performance goals and targets. However, the
plan could be improved if all annual performance goals were discussed in
the plan and linked to each strategic goal so that it would be clear how
annual performance goals would be used to gauge performance. DOE staff
stated that a description of all actual annual performance goals was not
something that they thought should be included in the strategic plan
because the annual goals differ each year.
Finally, our past and current reviews of DOE's 1997 and 2003 draft
strategic plans found that DOE addressed performance and accountability
Appendix III Observations on Agencies' Strategic Plans
challenges that we had previously identified. In January 2003, we
identified six areas where DOE's management attention was needed: (1)
addressing security concerns, (2) revitalizing infrastructure, (3)
improving contract management, (4) managing the nuclear weapons stockpile,
(5) cleaning up radioactive and hazardous wastes, and (6) enhancing
leadership in meeting energy needs. All areas were addressed in the 2003
draft strategic plan, with the exception of improving contract management.
For example, for the challenge of enhancing leadership in meeting energy
needs, one of the intermediate goals requires DOE to develop and
demonstrate technologies that can reduce emissions by more than 70 metric
tons of carbon and equivalent greenhouse gases by 2012.
Critical Strategic Planning Issues Needing Further Improvement
There are three elements in DOE's draft 2003 strategic plan that requires
further improvement. To begin with, although DOE's mission is results
oriented, it was revised from the 1997 draft strategic plan and does not
address the department's major activities related to energy supply and
conservation. These activities account for approximately 10 percent of
DOE's $23.4 billion fiscal year 2004 budget request. Our review of the
1997 draft strategic plan found that DOE's mission addressed all of its
major activities.
In addition, the impact of program evaluations on the development of
strategic goals could be discussed in greater detail. The strategic plan
stated that internal, GAO, and the Inspector General (IG) evaluations were
used as resources to develop the draft strategic plan, but specific
program evaluations were not identified. A schedule of future program
evaluations was also not discussed in the strategic plan. According to
DOE, there is no plan to include a table of reports and evaluations that
were used to develop the goals for the strategic plan because the
evaluations were from a prior administration, and when the administration
changes, it usually does not use evaluations from past administrations.
Appendix III Observations on Agencies' Strategic Plans
DOE could also enhance the strategies included in its plan by providing
information on how goals are linked to the department's day-to-day
activities, and the extent to which managers have the knowledge, skills,
and abilities to implement the strategies. None of this information was
included in the 1997 and 2003 draft strategic plans. According to DOE
officials, in drafting the plan, their goal was to keep the plan at a high
level and this additional information would require more detail than what
is needed for a strategic plan. For example, one official stated that a
description of linkages to day-to-day activities was not discussed in the
strategic plan because it would conflict with the performance plan and
budget justification, which is where the information can be found. As we
have stated previously, without this information, it is difficult to judge
DOE's likelihood of success in achieving the goals or the appropriateness
of the strategies.10
Finally, DOE's draft strategic plan could be improved by identifying
programs and activities that are crosscutting or similar to those of other
agencies. In the 2003 draft plan, crosscutting activities are only
identified for one goal related to science. According to one DOE official,
as the strategic goals were being developed, DOE staff took crosscutting
activities into consideration. As we stated in 1997, unless DOE addresses
crosscutting issues in its plan, Congress cannot be assured that federal
programs are working effectively.
Observations on Changes in the Quality of HUD's Strategic Plan
Overall, HUD has made progress in developing its strategic plan for fiscal
years 2003 through 2008 as required under GPRA. In 1997, we stated that
HUD's draft strategic plan did not cover the six components required under
GPRA.11 HUD's fiscal year 2003-2008 strategic plan addressed several
issues we had previously identified, such as making sure the mission
statement is linked to the department's major operations and functions;
including outcome-related goals and objectives for the department's
functions and operations; generally providing a description of how it will
achieve its goals and objectives; identifying key external factors
affecting the achievement of departmental goals; and explaining how HUD
will coordinate with other agencies to address crosscutting issues.
However, HUD could improve its strategic plan by explaining the
relationship
10GAO/RCED-97-199R. 11GAO/RCED-97-224R.
Appendix III Observations on Agencies' Strategic Plans
between the long-term and intermediate performance measures listed for
each strategic goal and discussing how it used program evaluations to
develop the plan. Table 8 summarizes these findings.
Table 8: HUD's Progress in Addressing Required Elements of Strategic
Planning under GPRA
Element of strategic planning Included in initial draft strategic plan
Included in current strategic plan
Mission statement No. HUD's mission statement did not cover Yes. HUD's
mission statement covers the the major program functions and operations
agency's major program functions and of the agency. operations and
relevant statutes.
Long-term goals Yes. The plan included eight Yes. The plan includes
strategic long-term goals that
objectives, and they generally cover the major programs
covered the and functions of
department's major functions and the agency.
operations.
No. The strategic plan lacked an
Strategies adequate
description of how its strategic
objectives
would be achieved.
Yes. The strategic plan discusses the means and strategies to address each
strategic goal, including plans to address human capital issues critical
to carrying out its mission.
Relationship between long- term goals and No. HUD's strategic plan provided
limited Yes. However, some long-term performance
annual goals examples of annual performance goals under each of its
strategic objectives, but it did not describe the relationship between
them. measures do not appear to have corresponding intermediate measures,
and in other instances it is not clear how HUD will measure progress
towards its goals
External factors No. HUD briefly discussed the external Yes. HUD describes
the external factors factors in its draft strategic plan without that
could affect achieving its strategic linking them to specific strategic
objectives. objectives.
Evaluations No. HUD's strategic plan did No. HUD does not describe how
not include program
information on program evaluations were used to prepare
evaluations. its
strategic plan. Although it
mentions that
program evaluations will be used
to advance
key policy objectives, it states
that there are
no fixed timetables for when
these
evaluations will take place.
Sources: GAO/RCED-97-224R and U.S. Department of Housing and Urban
Development, Strategic Plan FY 2003-FY 2008. (Washington, D.C.: 2003).
Strategic Plan Strengths and In its most recent strategic plan for fiscal
years 2003-2008, HUD has made Improvements from Fiscal progress in
crafting a mission statement that generally covers its major Year 1997
Draft Plan program functions, operations, and relevant statutes that
authorize its
programs. In addition, the current plan builds upon the strength of its
first draft strategic plan by including the strategic objectives that
cover the department's major functions and operations.
Appendix III Observations on Agencies' Strategic Plans
In contrast to its draft strategic plan of 1997, HUD's most recent plan
provides a general description of how its strategic objectives will be
achieved. For example, HUD lists the means and strategies following each
strategic goal and supporting objectives to describe how it will carry out
its activities. In addition, the strategic plan identifies a few
legislative and regulatory changes HUD will pursue to meet its objectives,
such as a new tax credit for developers of affordable housing and expanded
eligibility for the Assisted Living Conversion Program. To carry out its
work, HUD acknowledges in its plan that it needs to recruit and retain its
current workforce to ensure the proper skills and abilities needed to
carry out its mission.
HUD's most recent strategic plan also describes key factors external to
the department and beyond its control that could significantly affect the
achievement of its objectives. For example, for its goal of Promoting
Decent and Affordable Housing, HUD identifies the impact of broad economic
factors on opportunities for low-income workers as a factor that will
affect the department's ability to assist renters that depend on HUD's
programs to make progress towards self sufficiency. HUD's strategic plan
also provides a general description of current program evaluations and
mentions that the results of these evaluations will support key policy
objectives within the department.
Addressing a shortcoming of its draft strategic plan in 1997, HUD's
current plan generally explains how it will coordinate with other agencies
to address crosscutting problems. For example, the Interagency Council on
the Homeless (ICH), created by the Secretaries of HUD, the Department of
Health and Human Services (HHS), and Veterans Affairs (VA), will continue
to work to identify the obstacles homeless people face to enroll in the
main service programs and recommend specific changes-legislative, policy,
and procedural-that would make federal supportive service programs more
accessible to them.
Appendix III Observations on Agencies' Strategic Plans
Lastly, HUD improved its discussion of how it plans to address performance
and accountability challenges we have raised. In January 2003, we reported
that programmatic and financial management information systems and human
capital issues were HUD's performance and accountability challenges.12
While in the past HUD acknowledged some of these problems and described
how they would be addressed, its plans for management reform were not
fully integrated into its draft strategic plan. In contrast, one of HUD's
strategic goals in its current strategic plan, "Embrace High Standards of
Ethics, Management, and Accountability," lists specific actions the
department will take to address these challenges.
Critical Strategic Planning Issues Needing Further Improvement
HUD has made progress in linking its strategic objectives to both
long-term goals and intermediate measures in its strategic plan; however,
long-term performance measures are not consistently linked to a
corresponding intermediate measure, making it difficult for the reader to
understand how the department will measure progress towards its strategic
goals. For example, HUD projects that the percentage of architects and
builders indicating awareness of the design and construction requirements
of the Fair Housing Act will increase through fiscal year 2008. However,
HUD does not mention in the plan how many architects or engineers it will
survey to establish a baseline of awareness or to gauge progress made
towards this goal in subsequent years. HUD officials explained that only
those long-term goals that are critical to being achieved within the next
1-2 years have corresponding intermediate performance measures. Therefore,
it is somewhat unclear how HUD plans to assess progress made towards its
broader goal of "Ensure Equal Opportunity in Housing."
Similarly, the strategic plan does not always provide a clear picture of
how HUD will be able to measure progress towards its strategic goals. For
example, the plan states that HUD's Community Development Block Grants
(CDBG) Program will create or retain 400,000 jobs by fiscal year 2008;
HUD's intermediate measure for fiscal year 2004 is 84,000 jobs. These
estimates are based on the average jobs created per grant dollar reported
by grantees. However, HUD does not mention how it will be able to discern
between those jobs created by CDBG and those created by other means.
12U.S. General Accounting Office, Major Management Challenges and Program
Risks: Department of Housing and Urban Development, GAO-03-103
(Washington, D.C.: January 2003).
Appendix III Observations on Agencies' Strategic Plans
Similar to our previous finding, HUD's current strategic plan does not
describe how program evaluations were used to develop its strategic goals
or other components of its strategic plan, and does not include a schedule
for future evaluations. While the plan mentions that program evaluations
will be used to advance key policy objectives, it states that there are no
fixed timetables for when these evaluations will take place.
Observations on Changes in the Quality of SBA's Strategic Plan
In our review of SBA's March 1997 draft strategic plan,13 we found that
the plan did not meet two of the six requirements set forth by GPRA: (1) a
discussion of the relationship between the long-term goals and objectives
and the annual performance goals and (2) a description of how program
evaluations were used to establish or revise strategic goals and a
schedule for future program evaluations. In addition, the four elements
the plan contained could have better conformed to GPRA's requirements and
OMB guidance. In comparison, SBA's 2001-2006 strategic plan has improved
on several areas we identified in our 1997 review.14 However, we still
found areas that SBA could improve. Table 9 summarizes these findings.
Table 9: SBA's Progress in Addressing Required Elements of Strategic
Planning under GPRA
Element of strategic planning Initial draft strategic plan Current
strategic plan
Mission statement Yes. The mission statement was results Yes. SBA's
mission statement now oriented. However, it did not directly specifically
mentions its mandate and incorporate key aspects of SBA's legislative
encompasses SBA's disaster loan program mandate and it did not encompass
one of to families and businesses. SBA's significant activities.
Long-term goals Yes. Generally the strategic Yes. The strategic goals are
goals covered outcome
the major functions and oriented and cover the major
operations of SBA. functions and
Most of SBA's strategic goals operations of SBA.
were
expressed as processes, not
as outcomes.
Strategies Yes. However, the strategies were too Yes. The strategies will
generally help SBA vague or general to assess whether or not achieve its
goals. The strategies are now they would help achieve SBA's goals. Also,
listed by strategic goal and related objective. the strategies and goals
could have been The plan discusses some of the resources linked more
explicitly. SBA needs to achieve its goals.
13GAO/RCED-97-205R.
14At the time of our review, the most recent SBA strategic plan was for
fiscal years 20012008. SBA released a new strategic plan for fiscal years
2003-2008 in October 2003.
Appendix III Observations on Agencies' Strategic Plans
(Continued From Previous Page)
Element of strategic Initial draft strategic Current strategic plan
planning plan
Relationship between No. The linkage between Yes. The plan lists
long-term goals and proposed performance measures
annual goals performance measures, by objectives for each
strategies, and strategic goal. Logic
objectives was unclear. models show the relationship
between
measures and outcomes.
External factors Yes. The plan listed a Yes. The plan lists several
number of external external factors.
factors. A discussion Generally, strategies to
of how the external ameliorate the
factors would be taken effects of these factors are
into account when included.
assessing progress However, SBA does not discuss
toward goals was not how
included in the plan. external factors will be
taken into account
when assessing progress
toward goals.
Evaluations No. The plan did not describe how program No. The plan states
that lessons learned evaluations were used to establish or revise from
SBA's program evaluations have strategic goals or include a schedule for
influenced its strategic direction. Examples future program evaluations.
of future program evaluations are given, but a specific schedule of when
these evaluations are to occur is not included.
Sources: GAO/RCED-97-205R and Small Business Administration, SBA Strategic Plan,
FY 2001 - FY 2006 (Washington, D.C.: 2000).
Strategic Plan Strengths and Improvements from Fiscal Year 1997 Plan
SBA's current mission statement is an improvement from the one contained
in its 1997 draft strategic plan. In our July 1997 report on SBA's draft
strategic plan, we noted that the mission statement could be improved by
more directly incorporating key aspects of SBA's legislative mandate to
aid, counsel, assist, and protect the interests of small businesses. In
addition, the mission statement did not encompass one of SBA's significant
activities-making loans to individuals. The mission statement in the
20012006 strategic plan now includes both of these items.
The long-term, or strategic, goals of SBA's 2001-2006 strategic plan are
(1) helping small businesses succeed, (2) helping Americans recover from
disasters, and (3) modernizing SBA. These three strategic goals are
outcome oriented. We stated in our previous report on SBA's draft
strategic plan that only two of SBA's seven strategic goals described
outcomes. The rest of the goals were expressed as processes.
SBA's 2001-2006 strategic plan shows significant improvement in its
strategies. In 1997 we stated that the plan could be improved by making
the linkage between the strategies and goals/objectives more explicit.
Objectives were listed as a group, followed by the strategies, which were
also listed as a group. The 2001-2006 plan describes the strategies,
objective by objective, making the linkage clear. The strategies contained
in SBA's 1997 plan consisted entirely of one-line statements and most were
too vague or general to enable an assessment of whether or not they would
Appendix III Observations on Agencies' Strategic Plans
help achieve the goals and objectives in the plan. While many of the
strategies listed in SBA's 2001-2006 plan are only one- or two-sentence
statements, all of the strategies seem to aid in achieving SBA's goals and
objectives. For example, one strategy for the objective of providing
entrepreneurial development assistance is to train SBA personnel in
outreach and business development approaches. This strategy would
presumably aid in making staff more competent at helping small businesses
develop. The current plan also includes some of the resources needed to
achieve SBA's goals. These resources include a table showing SBA's fiscal
year 2001 budget request and its plans to modernize its information
technology systems.
The 2001-2006 strategic plan also makes a clearer connection between
strategic goals and annual performance measures that will be used to gauge
progress in achieving strategic goals. SBA shows this relationship by
linking its performance output measures with the intended outcomes of
SBA's programs for two of the strategic goals. For the other strategic
goal, "Modernizing SBA," SBA lists the performance measures it will use by
objective. For example, the performance measure "personnel trained or
retrained" will gauge progress in achieving SBA's human capital
investments objective. This shows improvement over the draft strategic
plan, which listed the performance measures as a group without showing how
they were related to SBA's strategic goals.
Similar to its 1997 draft plan, SBA's 2001-2006 plan specifies external
factors that could affect the achievement of its strategic goals. The plan
lists eight external factors associated with two of its three strategic
goals; no external factors were associated with the third goal. The plan
includes strategies that seem to be intended to mitigate the effect of six
of these external factors, while two of the external factors,
congressional support and public support, do not seem to be addressed. An
example of an external factor that seems to be addressed by a strategy is
economic conditions, listed for the strategic goal "helping small
businesses succeed." One of the strategies for this strategic goal is to
determine economic trends and conditions. SBA states that it tracks
economic trends that affect small businesses and the contribution small
businesses make to the economy. SBA then brings these data to the
attention of policymakers.
We noted in our 1997 report that SBA did not explicitly address the
relationship of SBA's activities to similar activities in other agencies
and provided no evidence of coordination. In contrast, SBA's current
strategic plan includes a section on crosscutting issues. This section
contains
Appendix III Observations on Agencies' Strategic Plans
discussions of innovation and research assistance, international trade
assistance, business development assistance, veterans affairs, and
disaster assistance. One example of coordination is the U.S. Export
Assistance centers, which combine the trade-promotion and export-finance
resources of SBA, the Department of Commerce, the Export-Import Bank, and
in some locations, the Agency for International Development and the
Department of Agriculture.
Finally, SBA's 2001-2006 strategic plan generally addresses performance
and accountability challenges that we have previously identified. For
example, we noted in our January 2001 report on major management
challenges and program risks15 that SBA needed to continue to improve
oversight of its lending partners to correct oversight weaknesses. The
plan identifies lender oversight as a management challenge and states that
SBA has developed and implemented a safety and soundness oversight program
for Small Business Lending companies, institutionalizing the process
through the Office of Lender Oversight. This is an improvement over SBA's
draft strategic plan, which we reported did not clearly address previously
identified management problems.
Critical Strategic Planning Issues Needing Further Improvement
In 1997, we noted that SBA's draft strategic plan did not include a
discussion of how the external factors would be taken into account when
assessing progress toward goals. This observation holds true for the
current strategic plan. For the external factor mentioned above, economic
conditions, the plan states that if the economy remains strong, surety
bond guaranties will remain constant or decrease, but if the economy
deteriorates, demand will increase. However, the plan does not state how
SBA will assess success or failure in meeting its goals in relation to
this factor.
In its 1997 draft plan, SBA did not describe how program evaluations were
used to establish or revise strategic goals or include a schedule for
future program evaluations. In the 2001-2006 plan, SBA states, "We have
used lessons learned in our performance monitoring and program evaluations
to influence the strategic direction contained in this plan." The plan
includes findings from six completed program evaluations; however, no
further detail is given as to how these program evaluations were used to
establish
15U.S. General Accounting Office, Major Management Challenges and Program
Risks: Small Business Administration, GAO-01-260 (Washington, D.C.:
January 2001).
Appendix III Observations on Agencies' Strategic Plans
or revise the strategic goals. While the current plan gives examples of
future program evaluations, such as conducting a benchmark study on the
HUBZone program to assess the changes in employment and investment in
distressed urban and rural communities, it does not include a schedule of
future evaluations. SBA states that for the next several years, the agency
plans to systematically review programs that offer the most financial risk
to the government and also the programs that can offer tips on how to
improve efforts.
Observations on Changes in the Quality of SSA's Strategic Plan
SSA's strategic plan for 2003-2008 is well structured and contains all of
the required elements under GPRA. In 1997, we noted that SSA's draft
strategic plan contained all six required components, but suggested a
number of ways it could be strengthened. SSA has addressed some of the
issues we previously identified, such as associating specific programs
with goals and identifying external factors that may affect goal
achievement. SSA could further improve its strategic plan through (1)
ensuring that its strategic objectives will assist SSA in achieving its
strategic goals, (2) explicitly describing the effect of external factors
on goal attainment, (3) providing timetables or schedules for achieving
results, (4) providing details on how each performance and accountability
challenge will be addressed, (5) clearly explaining how program
evaluations were used in formulating the strategic plan, and (6)
discussing the manner in which SSA has coordinated with other agencies,
especially those that serve the same beneficiaries. Table 10 summarizes
SSA's progress in addressing the required elements of GPRA.
Table 10: SSA's Progress in Addressing Required Elements of Strategic
Planning under GPRA
Element of strategic Included in initial draft Included in current
planning strategic plan strategic plan
Mission statement Yes. SSA's mission statement Yes. SSA's mission
was statement has not
appropriate and reflective of changed substantially.
its new status
as an independent agency.
Long-term goals Yes. Long-term goals were established. Yes. The goals'
relationship to specific However, the relationship between long-programs
is more defined, but SSA's goal for term goals and specific programs was
achieving solvency of the social security unclear and did not identify the
results to be system is ambitious, given SSA's mission achieved. and
responsibilities. Key outcomes are identified for each goal. SSA's major
management challenges are not all clearly linked to individual goals or
objectives.
Appendix III Observations on Agencies' Strategic Plans
(Continued From Previous Page)
Element of strategic planning Included in initial draft strategic plan Included
in current strategic plan
Strategies Yes. The strategic plan was generally complete with regard to
processes and
Yes. SSA added some timetables, but the
technology, but did not include timetables or schedules. Some of the
success was predicated on changes in processes or technology improvements.
required resources are not specified. Some of the success is still
predicated on technological improvements.
Relationship between long-term goals and Yes. SSA provided numerous
performance Yes. SSA provided one or more key
annual goals measures relating to strategic goals and outcomes for each
strategic objective. SSA objectives, and plans for developing new notes
that success in meeting the objectives measures were discussed. It was
will be measured in the annual performance sometimes difficult to link
measures with plans by progress in achieving the key their appropriate
objectives. It was also outcomes. hard to discern which objectives did not
yet have performance goals. Some data were expressed by program, while
other data were aggregated.
External factors Yes. The report mentioned Yes. External (environmental)
several key factors are
external factors, but did listed in a separate section.
not explicitly link However, the
factors to general goals plan does not explicitly link
and state how they factors to general
could have affected goal goals and state how they could
attainment. Also, affect goal
the report did not discuss attainment. Specific effects
needed changes are not
(by Congress) to ensure discussed-most examples are
solvency. vague.
Evaluations Yes. The report contained a broad Yes. Future evaluations
(with brief discussion of program evaluations, but the descriptions) are
listed, but there is no evaluations were not clearly described. discussion
of how current evaluations are Also, SSA did not describe how the used to
establish or revise specific evaluations were used to establish or revise
goals/objectives. The plan states that SSA specific goals/objectives.
Finally, there was used internal and external (GAO, IG) no schedule for
completing future evaluations to determine strategic plans and evaluations
or methodologies. objectives.
Sources: GAO/HEHS-97-179R) and Social Security Administration, Social
Security Administration Strategic Plan 2003-2008, (Washington, D.C.:
2000).
Strategic Plan Strengths and SSA's mission statement changed very little
between 1997 and 2003. OMB Improvements from Fiscal Circular A-11 notes
that the mission statement should be brief and define Year 1997 Plan the
basic purpose of the agency, with particular focus on its core programs
and activities. SSA's statement conforms to this guidance-it reads, "To
advance the economic security of the nation's people through compassionate
and vigilant leadership in shaping and managing America's social security
programs."16
16In the 2003-2008 plan, the word "advance" replaced the word "promote."
Appendix III Observations on Agencies' Strategic Plans
In 1997, the relationship between SSA's long-term goals and specific
programs was unclear and did not identify the specific results to be
achieved. Since that time, SSA has improved this linkage and better
articulated intended results, including quantifiable goals. For example,
as part of its strategic goal to ensure superior stewardship of Social
Security programs and resources, SSA notes that one of its key outcomes is
to increase Supplemental Security Income (SSI) payment accuracy to 96
percent (free of preventable error) by 2008.
SSA improved upon its linkage between long-term goals and annual goals in
its fiscal year 2003-2008 strategic plan. Under each strategic goal in
this plan, SSA provided one or more key outcomes for each strategic
objective; in its 1997 draft strategic plan, we found it difficult to link
measures with the appropriate objectives and discern which objectives did
not yet have performance goals.
Critical Strategic Planning Issues Needing Further Improvement
Not all of SSA's strategic objectives and associated performance measures
will allow SSA to achieve its related strategic goals. Specifically, the
solvency goal in SSA's current strategic plan reads, "To achieve
sustainable solvency and ensure Social Security programs meet the needs of
current and future generations," but the sole associated objective-through
education and research efforts, support reforms to ensure sustainable
solvency and more responsive retirement and disability programs-will not
allow SSA, on its own, to reach that goal. While SSA's mission is to
advance the economic security of the nation's people, it is not
unilaterally responsible for achieving solvency in social security
programs.
An agency's strategic plan is expected to contain strategies for achieving
the goals articulated. In 1997, SSA's strategic plan was generally
complete with regard to processes and technology, but did not include
timetables or schedules for results. While the current strategic plan
contains processes, anticipated progress in technology, and some
timetables, it does not contain timetables or schedules for all of the
results. For example, as part of its strategic objective to "efficiently
manage Agency finances and assets and effectively link resources to
performance outcomes," SSA's key outcomes include (1) competing or
converting 50 percent of commercial positions and (2) to "get to green" on
all five President's Management Agenda (PMA) items. SSA has neither
identified the required resources to achieve these goals nor has it
identified a time frame for achieving them.
Appendix III Observations on Agencies' Strategic Plans
In our review of SSA's 1997 draft strategic plan, we noted that SSA
described several key external factors that may affect its programs, but
did not explicitly link such factors to its general goals and state how
these factors could affect goal attainment. In the current strategic plan,
SSA identifies environmental (external) factors: demographics, health and
disability trends, technological advances, and workforce trends. However,
as we found in our earlier review, the effects of these factors on
specific performance goals are not specified, even though SSA notes that
they drive the development of such goals.
SSA noted that it considered major management challenges identified by GAO
when it determined its strategic goals and objectives, but not all of
these challenges are clearly addressed in the plan. While these challenges
are not clearly identified, SSA addresses them to some degree throughout
the plan. For example, SSA's strategic goal to "Strategically manage and
align staff to support SSA's mission" addresses the governmentwide
challenge of strategic human capital management.17
SSA includes a list of major strategic process and program evaluations
scheduled for the fiscal years 2003-2008 time period, organized by
strategic goal. However, SSA does not list ongoing evaluations or mention
how the results of these evaluations were used to prepare the current
strategic plan. SSA notes that many of the hundreds of process and program
evaluations conducted annually were designed to evaluate and improve
internal processes falling below the strategic level. However, some of the
ongoing evaluations are associated with specific strategic goals; thus,
their outcomes could be discussed in the context of the strategic goals
with which they are affiliated.
SSA's strategic plan contains a very limited discussion of its
interactions with other agencies that have similar goals or serve the same
beneficiaries. In an interview, SSA officials noted that SSA has extensive
interactions with other agencies on such issues as earnings accuracy and
medical information, but the level of interaction varies by initiative.
SSA's strategic plan would benefit from a broader discussion of these
interactions, especially if they were broken down by initiative. For
example, as part of the objective to increase the accuracy of earnings
records, SSA notes that it will collaborate with the Internal Revenue
Service (IRS) to achieve more
17U.S. General Accounting Office, Major Management Challenges and Program
Risks: Social Security Administration, GAO-03-117 (Washington, D.C.:
January 2003).
Appendix III Observations on Agencies' Strategic Plans
accurate wage reporting as part of its means and strategies to reduce the
size of the suspense file.18 It would be helpful if SSA offered more
details as to the nature and extent of its collaboration with IRS.
Observations on Changes in the Quality of DOT's Strategic Plan
In our review of DOT's July 1997 draft strategic plan, we found that the
plan only met three of the six elements required by GPRA.19 The plan did
not meet GPRA's requirements to describe (1) strategies for achieving the
goals, (2) a linkage between DOT's long-term goals and annual performance
goals, and (3) the external factors that could significantly affect DOT's
ability to achieve its goals. Further, for the three elements that the
plan did meet, each had weaknesses. In comparison, DOT's 2003-2008 draft
strategic plan has improved on several areas we identified in our 1997
review.20 However, we still found areas where DOT could improve. Table 11
summarizes these findings.
Table 11: DOT's Progress in Addressing Required Elements of Strategic Planning
under GPRA
Element of strategic Included in initial draft Included in current draft
planning strategic plan strategic plan
Mission statement Yes. DOT's mission Yes. The mission statement
statement was continues to
comprehensive and covers cover its major functions
its major and operations and
functions and operations. more explicitly states
DOT's statutory
authority.
Yes. Five long-term goals Yes. Five strategic
Long-term goals encompassed objectives encompass
DOT's major functions and DOT's major functions and
operations. operations.
However, it was not clear Outcome goals and
as to how DOT candidate performance
would measure success for measures for each
most of its strategic goal help show
how DOT will measure
goals. success.
Strategies No. While the plan listed six Yes. Strategies are listed by
corporate strategic
management strategies for objective and include
achieving its discussions on
long-term goals, it did not leadership, building
describe the expertise, and
operational processes, the technology.
skills, the
technology, and the resources
required to
meet them.
18The suspense file contains information on earnings that cannot be
matched to an individual's record due to an invalid name/Social Security
number combination.
19GAO/RCED-97-208R.
20At the time of our review, the Department of Transportation was in the
process of revising its strategic plan. A draft copy of the updated
strategic plan, dated July 1, 2003, was used for this review.
Appendix III Observations on Agencies' Strategic Plans
(Continued From Previous Page)
Element of strategic Included in initial draft Included in current
planning strategic plan draft strategic plan
Relationship between long No. The plan did not Yes. The plan includes
term-goals and describe how performance
annual goals performance goals would measures and refers to
be related to the the performance
long-term goals. plan for further
information on annual
performance goals.
External factors No. Four external factors were identified, Yes. Several
external factors are listed for but other key factors were not included in
the each strategic objective. Generally the plan plan. Only one external
factor was gives descriptions of how these factors discussed in terms of
how it could have could affect the achievement of goals. affected DOT's
ability to accomplish its goals.
Evaluations Yes. Program evaluations used in No. An extensive table
describes the scope
establishing goals and a and methodology and the
schedule of future completion date of
evaluations were discussed. program evaluations for
However, the fiscal years 2003-
plan did not provide enough 2008. However, the plan does
information to not
determine the scope and specifically mention which or
methodology or how previous
the key issues to be addressed evaluations were used in
in future developing the
evaluations. plan.
Sources: GAO/RCED-97-208R and U.S. Department of Transportation, U.S.
Department of Transportation Draft Strategic Plan for Fiscal Years
2003-2008 (Washington, D.C.: 2003).
Strategic Plan Strengths and Improvements from Fiscal Year 1997 Plan
The mission statement contained in DOT's 2003-2008 draft strategic plan is
an improvement over the one contained in the 1997 draft plan. DOT's
mission, as stated in the 2003-2008 draft strategic plan, is "to develop
and administer policies and programs that contribute to providing fast,
safe, efficient, and convenient transportation at the lowest cost
consistent with the national objectives of general welfare, economic
growth and stability, the security of the United States, and the efficient
use and conservation of the resources of the United States." The mission
statement covers the major functions and operations of the department. In
our July 1997 report on DOT's 1997 draft strategic plan, we noted that the
mission statement could be improved by including language from the
department's enabling legislation to focus the mission statement more
directly on DOT's core activities. We gave an example of adding the
department's purpose to develop transportation policies and programs that
"contribute to providing fast, safe, efficient, and convenient
transportation at the lowest cost" from DOT's enabling legislation. The
mission statement in the 2003-2008 plan includes such language.
As in its 1997 draft strategic plan, DOT's 2003-2008 draft strategic plan
meets the requirement of GPRA to include long-term goals and objectives
for the major functions and operations of the department. The 2003-2008
draft strategic plan contains five strategic objectives (long-term goals)
that cover the major functions and activities of the department and are
results
Appendix III Observations on Agencies' Strategic Plans
oriented. Besides the strategic objectives of "safety," "mobility,"
"global connectivity," "environmental stewardship" and "security," the
current draft also contains an "organizational excellence" objective to
"advance the department's ability to manage for results and achieve the
goals of the PMA."
Each strategic objective section in the 2003-2008 draft plan contains
strategies for attaining DOT's outcomes and objectives. The strategies for
each strategic objective are listed in the categories of "leadership,"
"building expertise," and "technology." For example, a "technology"
strategy for the "mobility" strategic objective is to "examine ways to
encourage cargo movements by water through the development of barge and
fast vessel technologies to bring new capacity to our intermodal
transportation system." The plan states that this strategy supports DOT's
outcomes of reduced congestion in all modes of transportation and
increased reliability throughout the system. The strategies for each
strategic objective generally describe the operational processes, the
skills, and the technology required to meet DOT's goals and objectives.
The current draft strategic plan also states that the resources and
programs listed in DOT's annual performance plans and budgets are
necessary to achieve DOT's outcomes and to execute the strategies. In
contrast, the 1997 draft strategic plan provided insufficient information
to describe the operational processes, the skills, the technology, and the
resources required to meet DOT's long-term goals, as required by GPRA.
Also, each strategic objective section in the current draft plan contains
crosswalks between outcomes in the strategic plan and performance measures
in the annual performance plans and reports. These crosswalks show the
measures that will be used to measure progress in achieving most of DOT's
outcomes and strategic objectives. For example, the performance measure
"number of passengers in international markets with open skies aviation
agreements" is related in a crosswalk to the outcome "reduced barriers to
trade in transportation goods and services." Together, the measure and
outcome will show progress toward DOT's global connectivity strategic
objective. This is an improvement from DOT's 1997 draft strategic plan
when we noted that although supporting documents showed that DOT had
developed information on how to measure each outcome goal, this
information was not included in the draft.
In contrast to DOT's 1997 draft strategic plan, the 2003-2008 draft plan
lists several external factors for each strategic objective and generally
discusses how these factors could affect the department's ability to
achieve its
Appendix III Observations on Agencies' Strategic Plans
outcomes and objectives. For example, one of the external factors for
DOT's environmental stewardship strategic objective is that DOT faces a
significant challenge to control and minimize air, water, and noise
pollution. The plan states that if DOT cannot control and minimize this
pollution, the department may encounter a public backlash that may impede
system improvement. For the external factors relating to the safety and
mobility strategic objectives, the plan lists both positive and negative
consequences the factors could have on achieving goals. One example would
be the possible effects the expansion and integration of the
telecommunications and e-commerce industry sectors could have upon
transportation safety. The plan states that this could affect the
achievement of DOT's safety objective by leading to unsafe practices, such
as the use of cell phones and other personal devices while driving. On the
other hand, these technologies could also contribute to safety by alerting
responders to the location of crashes and vehicles in distress. The 1997
draft plan identified four external factors and only discussed how one of
those factors could affect DOT's ability to accomplish its goals.
The current draft strategic plan includes an extensive table listing
program evaluations to be completed during fiscal years 2003-2008. The
table includes the name of the program to be evaluated, which strategic
goal(s) the program supports, the scope and methodology of the evaluation,
and the fiscal year during which the evaluation will be completed. DOT's
1997 draft strategic plan only listed the titles for the evaluations
scheduled for 1997 and 1998, which was insufficient to determine the scope
and methodology.
DOT's current draft plan lists crosscutting programs by strategic
objective. The discussions of crosscutting programs include the goal of
each program, which of DOT's outcomes each supports, and the agencies
involved. For example, the goal of aviation security is to prevent
explosives, weapons, and other dangerous items from being placed aboard
aircraft. This program supports DOT's security outcome of rapid recovery
of transportation in all modes from intentional harm and natural
disasters. DOT, through the Federal Aviation Administration, leads this
program, which involves the Transportation Security Administration,
Federal Bureau of Investigation, U.S. Customs Service, and U.S. Postal
Service, among others. Previously, the 1997 draft did not provide evidence
that DOT coordinated with other agencies that had programs and activities
that were crosscutting or similar to DOT's.
Appendix III Observations on Agencies' Strategic Plans
DOT's major management challenges, which we identified, are generally
discussed in the 2003-2008 draft strategic plan, organized by related
strategic objective. For example, we noted in our January 2003 report that
one major management challenge that DOT faces is building human capital
strategies.21 The current draft strategic plan includes a discussion of
human capital in DOT's organizational excellence objective. A separate
section of this discussion addresses our concerns regarding human capital
strategies and includes several milestones to address these concerns.
These milestones include conducting workforce planning for missioncritical
occupations in fiscal year 2003, and implementing a departmentwide
performance management system beginning in fiscal year 2003, and a uniform
branding and marketing approach to attract, acquire, and retain diverse
high-quality talent. DOT's 1997 draft strategic plan did not adequately
address the major management challenges we had previously identified.
Critical Strategic Planning Issues Needing Further Improvement
As stated above, the 2003-2008 plan provides a clear picture of how
success will be measured for most of DOT's strategic objectives and
outcomes. However, this clarity is not provided for a few strategic
objectives and outcomes. We noted the same issue in our review of DOT's
1997 draft strategic plan. For example, in the current plan three of the
outcome goals for the global connectivity strategic objective lack
corresponding performance measures. These outcomes are enhanced
international competitiveness of U.S. transport providers and
manufacturers, harmonized and standardized regulatory and facilitation
requirements, and the most competitive, cost-effective and efficient
environments for passenger travel. The plan states that the measures are
to be determined. However, without these measures it is unclear how
progress will be measured because the outcomes themselves do not lend
themselves to measurement.
While the current strategic plan shows improvement in the schedule for
future program evaluations, it does not sufficiently describe the
evaluations used in establishing or revising DOT's strategic objectives.
DOT states that detailed descriptions of completed program evaluations are
presented in its 2002 performance and accountability report. Further, the
plan states that DOT considered the results of completed program
21U.S. General Accounting Office, Major Management Challenges and Program
Risks: Department of Transportation, GAO-03-108 (Washington, D.C.: January
2003).
Appendix III Observations on Agencies' Strategic Plans
evaluations, as well as reports from DOT's Inspector General and GAO, in
writing the strategies to achieve its strategic objectives and outcomes.
The plan does not describe which or how program evaluations were used to
write the strategies.
Appendix IV
Observations on Agencies' Annual Performance Plans
Under GPRA, agencies are to prepare annual performance plans after the
development of their strategic plans. These annual plans are to establish
the connections between the long-term strategic goals outlined in the
strategic plans and the day-to-day activities of managers and staff. One
of our objectives was to assess the changes in the overall quality of
agencies' goals, strategies, and data articulated in their annual
performance plans. To meet this objective, we judgmentally selected six
agencies-Education, DOE, HUD, SBA, SSA, and DOT-using criteria such as
agency size, primary program types, and previous GAO reviews. To assess
the overall quality and improvements made to the agencies' performance
plans, we relied on requirements contained in GPRA and accompanying
committee report language,1 guidance to agencies from the OMB for
developing performance plans,2 best practices identified in our published
work,3 previous GAO evaluations,4 interviews with agency officials, and
our knowledge of agencies' operations and programs.
Key Elements of Information for Annual Performance Plans
Although GPRA does not require a specific format for the performance plan,
it does require the plan to (1) identify annual performance goals and
measures for each of an agency's program activities, (2) discuss the
strategies and resources needed to achieve annual performance goals, and
(3) provide an explanation of the procedures the agency will use to verify
and validate its performance data. We categorized each agency's plan based
on the degree to which it collectively addressed these three
characterizations.
To assess the degree to which an agency's plan provides a clear picture of
intended performance across the agency, we examined whether it included
(1) sets of performance goals and measures that address program results,
(2) baseline and trend data for past performance, (3) performance goals or
1Government Performance and Results Act of 1993, Committee on Governmental
Affairs, United States Senate, S. Rpt. No. 58, 103d Cong. 1st Sess.
(1993).
2OMB Circular No. A-11: Part 6, Preparation and Submission of Strategic
Plans, Annual Performance Plans, and Annual Program Performance Reports
(Washington, D.C.: June 2002).
3GAO/GGD/AIMD-99-215 and The Results Act: An Evaluator's Guide to
Assessing Agency Annual Performance Plans, GAO/GGD-10.1.20 (Washington,
D.C.: April 1998).
4GAO/HEHS-98-172R, GAO/RCED-98-194R, GAO/RCED-98-159R, GAO/RCED-98-180R,
GAO/RCED-98-200R, and GAO/HEHS-98-178R.
Appendix IV Observations on Agencies' Annual Performance Plans
strategies to resolve mission-critical management problems, and (4)
identification of crosscutting programs (i.e., those programs that
contribute to the same or similar results), common or complementary
performance goals and measures to show how differing program strategies
are mutually reinforcing, and planned coordination strategies.
To assess the degree to which an agency's plan provides a specific
discussion of strategies and resources the agency will use to achieve
performance goals, we examined whether it included (1) budgetary resources
related to the achievement of performance goals, (2) strategies and
programs linked to specific performance goals and descriptions of how the
strategies and programs will contribute to the achievement of those goals,
(3) a brief description or reference to a separate document of the human
capital, information, and other resources required to achieve results,5
and (4) strategies to leverage or mitigate the effects of external factors
on the accomplishment of performance goals.
Finally, to assess the degree to which an agency provides confidence that
its performance information will be credible, we examined how each report
discussed the quality of the data presented. To help improve the quality
of agencies' performance data, Congress included a requirement in the
Reports Consolidation Act of 2000 that agencies assess the completeness
and reliability of their performance data. Under the Act, agencies were to
include this assessment in the transmittal letter with their fiscal year
2000 performance reports. Agencies were also required to discuss in their
report any material inadequacies in the completeness and reliability of
their performance data and discuss actions to address these inadequacies.
5The Homeland Security Act (Pub. L. No. 107-296), also requires that
agencies provide a description of how the performance goals and objectives
are to be achieved, including the operations, processes, training, skills
and technology, and the human capital, information, and other resources
and strategies required to meet those performance goals and objectives.
Appendix IV Observations on Agencies' Annual Performance Plans
For each of these elements, we characterized each agency's fiscal year
1999 and fiscal year 2004 plan in one of four ways, based on the degree to
which the plan contained informative practices associated with that
element. Thus, to address the first element concerning the degree to which
the plan provided a clear picture of performance, we characterized each
plan in one of four ways: (1) clear, (2) general, (3) limited, or (4)
unclear. To address the second element, on the extent to which a plan
includes specific discussions of strategies and resources, we
characterized each plan as (1) containing specific discussions of
strategies and resources, (2) general discussions, (3) limited
discussions, or (4) no discussions. Finally, to address the third element
on the extent to which a plan provides confidence that performance
information will be credible, we characterized each plan as providing (1)
full confidence, (2) general confidence, (3) limited confidence, or (4) no
confidence. In conducting our reviews, we compared our assessments of
agencies' fiscal year 2004 plans to our assessments of plans from fiscal
year 1999 using similar criteria.6 A more detailed discussion of our scope
and methodology and the criteria we used can be found in appendix I.
Table 12 summarizes our characterizations of the six agencies' annual
performance plans based on our current review of fiscal year 2004 plans
and our previously published reviews of 1999 plans. Although the
characterization of agency performance plans did not change significantly
between the 1999 and the 2004 plans, the majority of agencies' plans
showed some improvement.
6GAO/HEHS-98-172R, GAO/RCED-98-194R, GAO/RCED-98-159R, GAO/RCED-98-180R,
GAO/RCED-98-200R, and GAO/HEHS-98-178R.
Appendix IV Observations on Agencies' Annual Performance Plans
Table 12: Characterizations of Agencies' Annual Performance Plans
Characterizations
Picture of intended performance (unclear, limited, general, clear) Strategies
and resources (no discussions, limited, general, specific)
Data credible (no, limited, general, full) Agency
1999 2004 1999 2004 1999 2004
Department of Limited General Limited General Limited General
Education
Department of Limited Limited General General Limited Limited
Energy
Department of Limited General Limited General Limited General
Housing
and Urban
Development
Small Business Limited General Limited General Limited General
Administration
Social Security Limited Clear Limited General No General
Administration
Department of General Clear General Specific Limited Full
Transportation
Sources: GAO/HEHS-98-172R; GAO/RCED-98-194R; GAO/RCED-98-159R;
GAO/RCED-98-180R; GAO/RCED-98-200R; GAO/HEHS98-178R; and Department of
Education, FY 2004 Annual Performance Plan (Washington, D.C.: 2003);
Department of Energy, Annual Performance Plan, Fiscal Year 2004
(Washington, D.C.: 2003); Housing and Urban Development, Annual
Performance Plan, Fiscal Year 2004 (Washington, D.C.: 2003); Small
Business Administration, Budget Request & Performance Plan: FY 2004
Congressional Submission (Washington, D.C.: 2003); Social Security
Administration, Annual Performance Plan, Fiscal Year 2004 (Washington,
D.C.: 2003); and Department of Transportation, Fiscal Year 2004
Performance Plan (Washington, D.C.: 2003).
The remainder of this appendix discusses our observations on how the
quality of each of the agencies' annual performance plans we reviewed has
changed since the agencies submitted their first performance plans in
1999. We did not independently verify or assess the information we
obtained from agency annual performance plans. If an agency chose not to
discuss its efforts concerning elements in the plan, it does not
necessarily mean that the agency is not implementing those elements.
Appendix IV Observations on Agencies' Annual Performance Plans
Observations on Changes in the Quality of Education's Annual Performance
Plan
Education's fiscal year 2004 annual plan7 provides a general picture of
intended performance across the agency-an improvement over the 1999
plan-because the measures and indicators adequately indicate progress
toward meeting annual targets; the measures are objective, measurable, and
quantifiable; and baseline and trend data are included where available.
However, the relationship between the goals and measures in volume 2 and
the long-term goals in volume 1 is not clear and the plan does not make
clear whether, and, if so, how, all of the program activities in the
department's budget are covered by the annual performance goals.8 In
another improvement over the 1999 plan, volume 1 of the 2004 plan provides
a general discussion of Education's strategies and resources to achieve
its goals by presenting strategies and resources and the projected
distribution of fiscal year 2004 funding and staffing for each long-term
goal. The plan also adequately recognizes and discusses external factors
that could affect the department's performance. However, the resources and
many of the strategies are not directly linked to the achievement of
individual annual performance goals and no strategies or resources are
designated for the goals and measures in the program performance plans in
volume 2. Lastly, the 2004 plan provides general confidence that agency
performance information will be credible. The 2004 plan contains
information on data sources for most of its measures, and for some,
identifies limitations. The plan also includes an appendix entitled
"Information Quality Guidelines" which recognizes data quality as a major
challenge for the agency and says that its improvement is a top priority.
7Education's plan states that its fiscal year 2004 annual plan includes
both department-level measures and program performance plans. These are
organized into two volumes: the Annual Plan Fiscal Year 2004 U.S.
Department of Education includes the departmentlevel measures and the FY
2004 Program Performance Plan: U.S. Department of Education includes the
program performance plans with their individual program measures. These
volumes are presented in a slightly different electronic format for the
public and other parties in general, which is available at Education's Web
site. The two volumes will henceforth be referred to as Education's 2004
annual plan, or, where applicable, volume 1 and volume 2.
8Education's 2004 annual plan represents its annual performance goals as
"targets." According to GPRA, the definition for "performance goal" is "a
target level of performance."
Appendix IV Observations on Agencies' Annual Performance Plans
Education's Fiscal Year 2004 Performance Plan Provides a General Picture
of Intended Performance
Education's 2004 annual plan generally defines expected performance.
Measures and indicators are formulated so as to adequately indicate
progress towards meeting annual targets, seem to sufficiently cover key
performance aspects, and adequately capture important program
distinctions. The plan contains about 360 measures between volumes 1 and
2, an improvement over the 860 contained in its 1999 plan, which we judged
to be potentially excessive for an annual performance plan and possibly
interfering with Education's ability to assess its performance. Unlike in
our review of the department's 1999 annual plan, the measures and
indicators in the 2004 plan are objective, measurable, and quantifiable.
For example, most measures and indicators are set up to measure
percentages, cost, counts, or other numerical values with measurable,
quantifiable 2004 targets. In most cases where a measurable target is not
given, the plan provides a reasonable explanation, such as a new program
or the measure being new, or a case where data are not collected or
available each year. In most cases, the plan provides trend data for
measures, which provides a helpful context for assessing the relevance of
the 2004 targets, or an explanation of why such data were not provided
(e.g., the baseline has not yet been established because the measure
and/or program are new).
In our review of Education's 1999 annual plan, we said that greater
outcome measure use would make future annual plans more useful. The 2004
plan frequently includes outcome goals and measures, such as a measure for
the number of states meeting their eighth-grade mathematics achievement
targets under the long-term goal to improve mathematics and science
achievement for all students.
Volume 1 of Education's 2004 annual plan directly aligns strategies,
action steps, measures, and targets with each of Education's long-term
goals and six strategic goals,9 containing the same strategic goals,
long-term goals, and mission as the 2002-2007 strategic plan. In our
review of the 1999 plan, we also found that the plan had performance goals
in volume 1 that were directly linked to its mission, strategic goals, and
objectives. However, the relationship between the goals and measures in
volume 2 and the long-term goals in volume 1 was not made clear in the
department's 2004 annual plan, which was similar to what we found in our
review of Education's 1999
9In this report, we refer to the multiyear, long-term objectives in
Education's annual plan as "long-term goals." The strategic goals included
in the plan represent overarching statements of aim or purpose that are
used to group Education's long-term goals.
Appendix IV Observations on Agencies' Annual Performance Plans
plan-that the department could more directly link the goals and measures
in volume 2 with the strategic objectives (long-term goals). Education's
fiscal year 2002 Performance and Accountability Report includes a table
making it clear that these programs and their goals and measures are
aligned across Education's strategic and long-term goals. By including
such a table in its annual performance plan, Education could clearly show
the link between the goals and measures in volume 2 and the long-term
goals in volume 1.
Although volume 2 of the 2004 annual plan states that it contains
individual program performance plans for all major programs and many
smaller programs, the annual plan does not make clear whether, and, if so,
how all of the program activities in the department's budget are covered
by performance goals. In contrast, we found that the 1999 plan provided
sufficient information to determine which performance goals and measures
in volume 2 covered which program activities in the budget and whether all
were covered. For example, the 1999 plan contained tables indicating the
funding levels for the program activities in the department's budget and
how those activities related to the programs in volume 2.
In our review of Education's 1999 annual plan, we gave the agency credit
for addressing the need to coordinate with other federal agencies having
related strategic goals or performance goals. However, we further noted
that Education could build on its foundation by identifying performance
goals that reflect activities being undertaken to support programs of a
crosscutting nature and specifying the activities each agency would
undertake and what it expects to achieve within the fiscal year. While
selected action steps in the 2004 plan refer to instances where the
department will coordinate or cooperate with other federal agencies, the
plan does not include steps or goals for most crosscutting issues
identified in Education's strategic plan. For example, for a crosscutting
issue identified in the strategic plan on safe and drug-free schools and
communities, Education said it partners with the Departments of Justice
(Justice) and HHS to promote drug and alcohol education programs and to
disseminate information to schools and private organizations. The
department also coordinates closely with the Office of National Drug
Control Policy, and works closely with the Office of Juvenile Justice and
Delinquency Prevention Programs to share innovative ideas and promote
prevention strategies and programs. The relevant annual plan sections in
both volumes 1 and 2 do not identify goals or action steps related to
these
Appendix IV Observations on Agencies' Annual Performance Plans
interactions. Additionally, according to our report on Education's June
1997 draft strategic plan,10 the department has numerous crosscutting
programs and activities, such as those related to early childhood and
employment training, and the 2004 annual plan does not address them all.
Education's 2004 annual plan discusses applicable goals, measures, and
strategies for two governmentwide major management challenges regarding
strategic human capital management and information security, as well as
three of the four major management challenges we identified for Education
in our January 2001 Performance and Accountability Series.11 For example,
for its student financial assistance programs, the department has
developed performance targets for fiscal year 2004 related to being
removed from our high-risk list, increasing the default recovery rate, and
decreasing grant overpayments to students. Also, a key strategy under its
goal to improve the strategic management of the department's human capital
is to develop a 5-year human capital plan, including developing a
recruitment plan and relevant training programs. For the fourth of
Education's major management challenges-promoting coordination with other
federal agencies and school districts to help build a solid foundation of
learning for all children-the 2004 plan did not include specific goals or
measures, but it did discuss some related strategies and steps, such as
using partnerships with other federal programs to promote development of
intervention strategies and methods to address the high incidence of
learning disabilities and illiteracy among adolescents attending high
schools.
Education's Fiscal Year 2004 In our review of Education's 1999 annual
plan, we found that the plan had a Performance Plan Provides limited
discussion of how the department's strategies and resources would a
General Discussion of help achieve its annual performance goals. The 2004
plan includes
strategies and resources under each of its long-term goals to be used to
Strategies and Resources achieve its annual performance goals in volume 1,
including the projected distribution of fiscal year 2004 funding and
staffing, in both dollars and fulltime-employees (FTE), for each long-term
goal, under which the annual performance goals are organized. However, the
resources and many of the
10GAO/HEHS-97-176R.
11U.S. General Accounting Office, Performance and Accountability
Series-Major Management Challenges and Program Risks: A Governmentwide
Perspective, GAO-01-241 (Washington, D.C.: January 2001) and GAO-01-245.
Appendix IV Observations on Agencies' Annual Performance Plans
strategies are not directly linked to the achievement of individual annual
performance goals and no strategies or resources are designated for the
goals and measures in the program performance plans in volume 2. Overall,
the plan does not discuss how resources were allocated to each goal, a
rationale for how the resources will contribute to improving performance,
or the relationship of capital asset investments, including those for
information technology (IT), to the achievement of specific goals.
However, the plan does include a performance measure and goal for the cost
and schedule of IT investments and a strategy for completing the
department's enterprise architecture, which is to be used to guide IT
capital decisions. In addition, the department has a plan for human
capital management and a new performance appraisal system that is meant to
link employee performance standards to the department's strategic
priorities, but neither had been fully implemented.
In our review of the 1999 plan, we said that external factors that could
affect performance were not discussed and that such factors are important
for a department like Education because much of what it hopes to achieve
depends on others and external events. In its 2004 plan, Education clearly
acknowledges that improving support for its state, local, and
institutional partners, who have the direct ability to influence outcomes
the department seeks, is a major challenge. The plan contains numerous
activities to handle this challenge, including, for example, to provide
support and technical assistance, improve grant monitoring, and fund an
annual survey of states' efforts. Moreover, although not labeled as
addressing external factors, the plan has specifically related strategies
and/or action steps for most external factors identified in Education's
2002-2007 strategic plan.
Education's Fiscal Year 2004 Performance Plan Provides General Confidence
That Performance Data Will Be Credible
In our review of Education's 1999 annual plan, we found that it did not
provide sufficient confidence that its performance information would be
credible. For example, the 1999 plan did not sufficiently recognize
limitations in Education's data for its elementary and secondary education
programs. In comparison, Education's 2004 plan recognizes limitations in
Education's data for many of its elementary and secondary education
programs, as well as for other programs. In many of these cases, the plan
also discusses plans to address these limitations. Also, the plan includes
an appendix containing an abbreviated form of its "Information Quality
Guidelines" and a sample checklist for statistical data from its complete
guidelines. The appendix recognizes data quality as a major challenge to
the department's successful implementation of GPRA and says that the
improvement of data quality is a top priority. The checklist includes
Appendix IV Observations on Agencies' Annual Performance Plans
several steps related to the verification and validation of data, such as
evaluating data quality, including known limitations; addressing the
reliability of data sources; and ensuring reproducibility of findings
using the same data and methods of analysis. In addition, the plan usually
identifies the sources of data and, although not in volume 1, includes a
column on sources and data quality for each measure in volume 2, usually
with an item entitled "Validated By" and, in some cases, "Limitations." In
the end, the lack of direct control over the implementation of its
programs, including the collection of data, is a significant data quality
challenge that Education must face.
The 2004 annual plan also contains several action steps on new or changing
information systems that relate to improving the collection of information
for measuring performance. For example, under its strategy to reduce
Education's partners' data reporting burden, the plan includes an action
step to develop and implement the Performance-Based Data Management
Initiative collection system. This step is directly related to the plan's
measure to reduce the burden hours of Education program data collections
per year.
Observations on Changes in the Quality of DOE's Annual Performance Plan
Compared to the fiscal year 1999 plan we reviewed, DOE's fiscal year 2004
performance plan continued to provide a limited picture of intended
performance. Although the plan included more results-oriented annual
performance measures, it still provided a limited linkage between its
reported annual goals and its mission, strategic plan goals, and program
activities within its budget request. Furthermore, the 2004 plan provided
a general discussion of strategies and resources, similar to our 1999
findings. Finally, the 2004 plan provided a limited level of confidence
that data will be credible by making little progress in reporting on the
procedures it uses to ensure data quality or identifying significant data
limitations, which is consistent with our 1999 findings.
DOE's Fiscal Year 2004 While DOE has improved its development of annual
performance Performance Plan Provides measures-referred to as targets-by
making them more results oriented, a Limited Picture of the overall
picture of performance is limited by the lack of alignment
between its annual and strategic goals and minimal discussion of
Intended Performance coordination with other agencies. Our review of DOE's
1999 plan found that many measures were unclear, appeared limited in
scope, or were not very useful indicators of performance. We found these
problems in the
Appendix IV Observations on Agencies' Annual Performance Plans
performance plans for subsequent years as well. For the 2004 plan,
however, the majority of the performance measures related to each goal
were results oriented and pertained specifically to the performance of
fiscal year 2004. An example of one measure requires DOE to train 4,000
federal employees by the end of fiscal year 2004 in energy management best
practices that support National Energy Policy education goals.
DOE provided a limited link between its reported annual goals and its
mission, strategic plan goals, and program activities within its budget
request. While the 2004 annual performance plan goals address all of the
major program activities in DOE's budget, the goals and mission of the
2004 plan do not align with the mission and goals for the 2003 draft
strategic plan. This represents a set back because in our review of DOE's
1999 annual performance plan, we found that DOE clearly linked its annual
goals to the agency's mission, strategic plan goals, and its program
activities within its budget request. DOE officials told us the lack of
linkage between the performance plan and the strategic plan was a matter
of timing. According to these officials, the department originally updated
its strategic plan at the same time as the annual performance plan, which
was finalized in the early months of 2003, and the goals of each plan
coincided, but the draft strategic plan goals were revised in the latter
part of the year and no longer align with the 2004 performance plan.
DOE's ability to show coordination with other agencies is also limited. In
1999, we reported that DOE did not adequately show that it coordinated
with other agencies that have related strategic or performance goals.
DOE's 1999 plan contained very little evidence of specific goals and
measures that addressed crosscutting programs and only briefly described
coordination with other agencies. The 2004 plan does not specifically
describe how coordination is taking place among crosscutting programs, but
does identify groups that it is collaborating with on certain programs. In
response, DOE officials told us the plan does not discuss what specific
collaboration activities are taking place because it would require
reporting too much detail for a performance plan. DOE officials stated
that it collaborates at the program level, rather than the agency level,
because the program plans pertain to an organizational layer lower than
the annual performance plan.
Appendix IV Observations on Agencies' Annual Performance Plans
Finally, the plan briefly mentions that the department has been
identifying challenges and working on ways to address them. According to
DOE officials, when developing annual targets for the department,
management challenges are considered but not mentioned specifically in the
report. Our review of management challenges in 2002 found that DOE had
addressed all eight of its challenges in its 2003 annual performance
plan.12 In comparing these challenges to the 2004 plan, we found that DOE
continues to have goals that address the eight challenges we identified in
2002 and the additional challenges that we identified in our 2003
performance and accountability report.13
DOE's Fiscal Year 2004 Performance Plan Provides a General Discussion of
Strategies and Resources
DOE provided a general discussion of the strategies and resources that it
will use to achieve its annual performance goals. DOE's 1999 plan
partially provided clear and reasonable strategies for achieving
performance goals, how strategies would contribute to achieving the
performance goals, and key external factors that might affect performance.
For each of the 2004 annual performance goals, DOE included a "Means and
Strategies" section in the plan that described how each goal will be
achieved. For example, one strategy identified to meet its goal of
contributing unique, vital facilities to the biological and environmental
sciences is to conduct peer reviews of the facilities to assess the
scientific output, user satisfaction, and the overall cost-effectiveness
of each facility's operations, and their ability to deliver the most
advanced scientific capability. The 2004 plan includes a brief discussion
of the department's overall needs, particularly in the areas of human
capital, financial, and logistical resources. The plan also identified
budget amounts for each of its goals. DOE's 1999 plan partially identified
the resources needed to accomplish annual performance goals.
The plan also provided a general discussion of the external factors that
could affect achievement of the goals, but it did not specifically discuss
actions on how the external factors will be addressed. For example, the
plan states that external factors related to DOE's goal of achieving
reliable, affordable, and environmentally sound energy supplies, such as
renewable fuels, include program funding, the state of the economy, the
availability of
12GAO-03-225. 13GAO-03-100.
Appendix IV Observations on Agencies' Annual Performance Plans
conventional supplies, the cost of competing technologies, and the
continuation of federal tax incentives and other national-level policies.
DOE's Fiscal Year 2004 Performance Plan Provides Limited Confidence That
Performance Data Will Be Credible
DOE has made limited progress on reporting the procedures it uses to
ensure data quality. Its 1999 plan described how DOE would ensure that its
performance information is sufficiently complete, accurate, and
consistent, but did not discuss in detail DOE procedures on how to help
ensure the quality of data or the process of collecting the data. The plan
also did not identify significant data limitations and how they may affect
DOE's ability to achieve performance goals. However, the 2004 plan showed
some improvement over the 1999 plan by describing credible procedures to
verify and validate performance information and specific program
evaluations are mentioned for each goal. The plan also discusses that DOE
acquired new commercial software for performance tracking through remote
data entry, monitoring, and oversight by program offices and managers. The
2004 plan only identifies data limitations and any new or modified systems
very briefly for a few relevant goals. According to DOE officials, with a
few exceptions, its plans do not discuss data limitations because DOE
writes goals that are not affected by data limitations. The goals are
written to ensure that the data will be there to meet performance targets.
However, our 2003 performance and accountability series identified several
DOE management challenges where data quality was a concern, such as
further upgrades needed for cyber security to ensure adequate protection
of data and information systems and additional information on the results
of contractors' performance to keep projects on schedule and within
budget.
Observations on Changes in the Quality of HUD's Annual Performance Plan
HUD's annual performance plan for fiscal year 2004 improves upon areas
where we previously reported shortcomings and generally meets the criteria
set forth in GPRA. HUD's 2004 plan provides a general picture of intended
performance by covering all the programs contained in HUD's budget and
linking program activities to strategic goals and objectives. The plan
also improved by providing specific information on HUD's strategies and
activities along with performance measures it will use to assess progress
toward its goals and discussing relevant external factors that could
affect the attainment of certain program objectives. HUD also provides
greater confidence that performance data will be credible by thoroughly
discussing the data it will use for measuring progress toward its goals.
Nevertheless, the plan could be further enhanced if it included more
Appendix IV Observations on Agencies' Annual Performance Plans
specific information on how funds will be allocated to achieve program
objectives, explain how HUD will contribute to crosscutting efforts along
with other agencies, and what steps it will take to mitigate the impact of
external factors on its programmatic objectives.
HUD's Fiscal Year 2004 Performance Plan Provides a General Picture of
Intended Performance
Since our review of HUD's annual performance plan for fiscal year 1999,14
HUD has made progress in developing an annual performance plan that
generally reflects the department's mission and provides a general picture
of intended performance. HUD's most recent performance plan covers the
program activities contained in its budget, and generally links program
activities to strategic goals and objectives, key items missing from its
plan for fiscal year 1999. HUD has also improved the quality of its
performance plan by including performance measures that generally indicate
how the department will gauge progress toward achieving its goals. For
example, the performance plan lists a series of performance measures for
each objective that can be used to indicate progress towards the
department's goals and expected performance. These measures are also
objective and a number of them have been quantified, another key area
where HUD has improved since its first performance plan. For example,
activities supporting HUD's long-term strategic objective to "Improve the
Physical Quality and Management Accountability of Public and Assisted
Housing" include, among other things, eliminating 100,000 units of the
worst public housing. According to the current plan, the department
intends to demolish 10,000 of these units in fiscal year 2004.
While HUD's most recent annual performance plan generally identifies other
agencies it will coordinate with to address crosscutting efforts, it does
not discuss how it plans to work with these agencies to address these
crosscutting activities. For example, the plan states that the Interagency
Working Group on Limited English Proficiency will ensure that persons with
limited English proficiency will have meaningful access to funded and
federally conducted programs and activities. However, the plan does not
discuss what HUD's contribution to this multiagency effort will be, what
strategies it will employ, or how it will measure progress toward
achieving the strategies of this multiagency effort.
14GAO/RCED-98-159R.
Appendix IV Observations on Agencies' Annual Performance Plans
HUD's current performance plan is an improvement compared to its fiscal
year 1999 plan as it describes steps HUD will take to address major
management challenges. One of HUD's strategic goals, "Embrace High
Standards of Ethics, Management, and Accountability," identifies five
objectives that cover management challenges, some of which have been
raised by GAO and the HUD IG. These objectives discuss plans to rebuild
HUD's human capital and diversify its workforce; improve HUD's management,
internal controls, and resolve audit issues; improve accountability,
service delivery, and customer service; ensure program compliance; and
improve internal communication and employee involvement.
HUD's Fiscal Year 2004 Performance Plan Provides a General Discussion of
Strategies and Resources
HUD's most recent performance plan also improves upon earlier plans we
reviewed in providing readers an idea of the strategies that HUD will
employ to carry out its goals. Each strategic goal in the annual
performance plan contains a section titled "Means and Strategies," which
describes activities HUD will pursue to support that goal. For example, to
support its goal of "Increasing Homeownership Opportunities," HUD will
fund low-income homeowner assistance programs to provide approximately
40,000 families with down payments and closing costs on their homes,
473,199 families with home purchase and homeownership counseling, and
about 232,370 families with rental counseling.
HUD's performance plan also discusses the strategies it will employ to
address the department's human capital issues, such as the upcoming
potential wave of employees planning to retire and the need to equip staff
with the desired knowledge and skills. For example, HUD completed a staff
resource estimation and allocation system in 2002, and it will conduct a
comprehensive workforce analysis in 2004 to serve as the basis to fill
mission-critical skill gaps through succession planning, hiring, and
training initiatives in its Five-Year Human Capital Management Strategy.
Although HUD has made progress in linking its resources to strategies, it
could improve the discussion by linking funding allocations to specific
performance goals, thus making the plan more informative. The plan
discusses budget and staff allocations for programs supporting each
strategic goal. For instance, portions of HUD's Community Development
Block Grants Fund and Home Investment Partnership Program, with a combined
budget authority for fiscal year 2004 of more than $2.5 billion and staff
of 203, support the strategic goal of promoting "Decent Affordable
Housing." However, it is unclear what resources will be used to pursue
Appendix IV Observations on Agencies' Annual Performance Plans
specific performance targets for each program. Additionally, HUD does not
mention in its plan how IT and capital resources will be used to support
its programs.
Anticipating that some aspects of the department's strategic goals are
intertwined with broader phenomena, the performance plan also discusses
several external factors relevant to each strategic goal that could affect
HUD's ability to meet its objectives. For example, for its strategic goal
"Promote Decent and Affordable Housing," HUD states that broad economic
factors can affect opportunities for low-income workers relying on the
department for rent assistance to make progress towards selfsufficiency.
However, it is unclear from the performance plan what actions, if any, HUD
has put in place to mitigate the effect of these external factors.
HUD's Fiscal Year 2004 Performance Plan Provides General Confidence That
Performance Data Will Be Credible
HUD has also made significant progress in providing assurance that the
department will be able to use credible data to gauge progress towards
achieving its goals. HUD identifies the steps it (or others) will take to
verify and validate the performance data to ensure that what is reported
on HUD's performance will be credible. For example, for its objective
"Increasing Minority Homeownership," HUD will rely on, among other
indicators, the rate of minority homeownership from the Current Population
Survey conducted monthly by the U.S. Census Bureau. HUD will not verify
the data because the Bureau performs that task. Additionally, HUD also
includes in its most recent performance plan a discussion of the inherent
limitations of the data it will use and generally discusses steps it will
take to improve the measure, providing the reader with a clearer
expectation of what HUD will be able to report.
Observations on SBA's 2004 performance plan shows progress made over the
agency's 1999
performance plan. In contrast to our review of SBA's 1999 plan,15 the
2004Changes in the Quality plan provides a general picture of intended
performance by discussingof SBA's Annual coordination between SBA and
other federal agencies on crosscutting Performance Plan activities.
Resource analysis sections throughout the plan provide a
general discussion of how SBA has previously used its resources to achieve
its goals and how it intends to use future resources for the same
purposes.
The 2004 plan also provides general confidence that SBA's performance
15GAO/RCED-98-200R.
Appendix IV Observations on Agencies' Annual Performance Plans
data will be credible by including more detail on how SBA verifies and
validates its data, as well as by identifying data limitations. However,
several areas of the plan could be improved, such as clearly linking SBA's
performance indicators, performance goals, and programs.
SBA's Fiscal Year 2004 Performance Plan Provides a General Picture of
Intended Performance
SBA's fiscal year 2004 performance plan provides a general picture of
intended performance. The performance goals and performance indicators in
the plan are generally objective, measurable, and quantified. Performance
indicators are listed throughout the plan by related programs and
strategic programmatic goals. Performance goals and outcome goals are
listed for each strategic programmatic goal. In our review of SBA's fiscal
year 1999 performance plan, we noted that SBA's performance goals were
objective and measurable, its performance measures were generally
objective and quantified, and that the performance goals in the plan were
clearly linked to SBA's strategic goals and objectives.
Like the performance measures contained in its 1999 plan, the 2004 plan's
performance indicators will be useful in assessing progress towards SBA's
performance goals. For example, the performance indicator "Regulatory Cost
Savings to Small Business" will adequately show progress for the
corresponding performance goal "Cost savings for small business due to the
efforts of the Office of Advocacy." In this example, the performance
indicator, which is listed under the Advocacy Program, can be linked to a
performance goal because of a crosswalk that relates outcome goals,
performance goals, and programs. However, there is not always such a clear
link between all of SBA's performance indicators and performance goals
because indicators are listed by program instead of by performance goal.
The BusinessLaw.gov program is linked to three performance goals: "number
of users of BusinessLaw.gov," "reduced cost to businesses and regulatory
agencies," and "increased rate of compliance." While the first two
performance goals appear related to the first two indicators listed in the
BusinessLaw.gov program section, there is no clear relationship between
any of the other performance indicators for this program and the third
performance goal, "increased rate of compliance."
SBA's 2004 performance plan contains annual performance goals that
generally cover the agency's budget activities. The 2004 performance plan
contains a budget crosswalk that "shows how the goals relate to specific
and general program areas." This is an improvement over the 1999 plan,
which we noted contained a budget crosswalk, but the categories in it did
not match SBA's budget accounts or activities by name or account number.
Appendix IV Observations on Agencies' Annual Performance Plans
However, the performance plan does not seem to cover all of SBA's
programs. Three "advocacy" programs listed in the crosswalk do not seem to
be contained in the plan: Business.gov, Disability Initiative, and
National Women's Business Council.
Each strategic programmatic goal section in the 2004 plan contains a
discussion of crosscutting issues. Several examples of coordination
efforts are given, such as SBA working with the Department of Defense to
integrate the PRO-Net system with the Central Contractor Registry and SBA
partnering with the Federal Acquisition Institute to develop on-line
training courses for small business programs. In contrast, SBA's 1999
performance plan provided little information on SBA's coordination efforts
with other entities whose programs and activities crosscut those of SBA.
SBA's 2004 performance plan generally addresses performance and
accountability challenges we have previously identified. For example, we
have previously stated that SBA needs to strengthen its performance in
human capital management. The 2004 plan includes outcome goals,
performance goals, and programs to address SBA's strategic management of
human capital in a section on the PMA.
SBA's Fiscal Year 2004 Performance Plan Provides a General Discussion of
Strategies and Resources
The 2004 performance plan provides a general discussion of the strategies
and resources SBA will use to achieve its goals. Each strategic
programmatic goal and each of the goals for the PMA contains a discussion
of the strategies for accomplishing the goals. These discussions provide a
broad overview of the strategies used at the strategic programmatic goal
level. For example, the plan includes a strategy for SBA's strategic
management of human capital. The strategy lays out SBA's Transformation
and Human Capital plans, which will be used to implement a new vision of
SBA. In its 1999 performance plan, SBA discussed strategies for most of
its performance goals, although for some of the goals the strategies were
missing.
Each strategic programmatic goal and several of the PMA goals contain
brief discussions of external factors that could affect the achievement of
SBA's goals. These discussions include actions to address external
factors, such as working with an Interagency Acquisition Working Group
under the Procurement Executives Council to develop supplemental
performance measures to better evaluate the success of its programs. In
1998, we noted that SBA's 1999 plan recognized certain external factors
and contained a discussion of actions SBA could take to mitigate the
effects of such factors
Appendix IV Observations on Agencies' Annual Performance Plans
for one of its strategic goals. We stated that it would be useful for SBA
to include a similar discussion of external factors and mitigation
strategies for its other strategic goals.
Each program listed throughout the plan has a resource analysis section
that describes how resources were used in fiscal year 2002. Some of these
analyses also include planned resources for fiscal year 2004. For example,
the resource analysis section for the Small Business Development Centers
(SBDC) program states that for fiscal year 2004 SBA requested
approximately the same level of funding as in fiscal year 2002. In 2002,
85 percent of the funds were for grants, while the other 15 percent
covered field support, program management, and overhead costs such as
rent, legal services, human resources, and information technology support.
Some of the resource analyses also contained pie charts of the breakdown
of costs. This is an improvement over SBA's 1999 performance plan, which
we found did not specifically identify the human or technological
resources that SBA would need to achieve its performance goals.
SBA's Fiscal Year 2004 Performance Plan Provides General Confidence That
Performance Data Will Be Credible
SBA's 2004 performance plan provides general confidence that its
performance data will be credible. An appendix of the performance plan
contains verification and validation information, as well as data
limitations and remedies for these limitations for most of SBA's
performance indicators. However, the appendix does not include this
information for the performance indicators of the disaster loan program,
nor are any of the PMA performance indicators discussed in the appendix.
Generally, the discussions of SBA's verification and validation processes
for its indicators in the 2004 plan are one-or two-sentence statements.
For one of the indicators, "number of jobs created and retained by the
7(a) loan program," SBA states that it does not have access to the data
for verification purposes. SBA also notes that it does not independently
verify some of the external data it gathers, as is stated in the
verification discussion of the indicator, "504 loans to emerging market
firms." This is an improvement over SBA's 1999 performance plan, which
included brief descriptions, often only one or two words, on the means it
used to verify and validate its data. We noted in our report on the 1999
plan that these appeared to be sources of data for the measures rather
than means to verify and validate the data.
The data limitations contained in SBA's 2004 performance plan are
generally one-sentence statements and the same limitations are used for
Appendix IV Observations on Agencies' Annual Performance Plans
multiple indicators. For example, several limitations, such as "the
measure is based on the number of approved loans" or "information is
derived from loan approval data," are used for multiple indicators. The
appendix also lists remedies for the data limitations for each indicator.
For example, a limitation of SBA's indicator "small business appointments
conducted with procurement officials" is that the indicator may not
capture unscheduled appointments. The remedy for this limitation is to
keep track of both scheduled and unscheduled appointments. The discussion
of data limitations and their remedies in the 2004 plan shows progress
over SBA's 1999 plan, which did not contain a discussion of data
limitations.
Observations on Changes in the Quality of SSA's Annual Performance Plan
Compared to the fiscal year 1999 plan we reviewed, SSA's performance plan
provided a clear picture of intended performance by (1) defining expected
performance, (2) offering trend data, which helps track progress toward
performance goals, and (3) using objective, measurable, and quantifiable
performance measures. SSA also provided general information on its
strategies and resources, somewhat better than our 1999 findings. Finally,
the plan provided a general level of confidence that data will be credible
by describing the Inspector General's (IG) involvement in data testing,
providing data sources and definitions, and identifying some data
weaknesses, an improvement over the 1999 plan. However, SSA's performance
plan still does not fully discuss the agency's coordination with other
agencies, identify performance goals that clearly cover all the program
activities, address how SSA plans to use the information from the
evaluations to improve program results, identify the resources needed to
address each performance goal, and discuss data verification and
validation procedures for its internal systems.
Appendix IV Observations on Agencies' Annual Performance Plans
SSA's Fiscal Year 2004 Performance Plan Provides a Clear Picture of
Intended Performance
Overall, SSA's fiscal year 2004 performance plan has improved over its
1999 plan.16 The 1999 plan only provided a partial picture of SSA's
intended performance across the agency. In June 1998, we reported that
SSA's 1999 Annual Performance Plan contained performance goals, many of
which were measurable and linked to the agency's strategic goals17 and
objectives; some of the performance goals related to particular strategic
goals were objective, measurable, and quantifiable. However, other goals
were not measurable or quantifiable and did not define the level of
performance to be achieved, thus making it difficult to see how SSA would
assess success.
SSA's fiscal year 2004 plan provides a much clearer picture of intended
performance through (1) defining expected performance, (2) offering trend
data, which helps track progress toward performance goals, and (3) the use
of objective, measurable, and quantifiable performance measures. For
example, as part of the strategic objective to "Prevent fraudulent and
erroneous payments and improve debt management," SSA provided historical
data on the outcome measure "Percent of SSI payments free of preventable
error (overpayments and underpayments)" from fiscal years 1999-2001 and
projected goals for fiscal years 2002-2004.
While we found significant improvements in SSA's 2004 annual performance
plan over its 1999 plan, we also found some weaknesses. For example,
coordination efforts with other entities, such as federal agencies, state
and local entities, and others, are not well identified. According to SSA
officials, SSA coordinates with other federal agencies, such as the IRS
and the Immigration and Naturalization Service, as well as Veterans
Administration, on information-sharing initiatives. However, these types
of coordination efforts are mentioned only briefly, if at all, in the 2004
annual performance plan.
In its 2004 plan, SSA includes a list of major program evaluations it
plans to conduct during 2003-2004, with a brief description of the
evaluations, their associated strategic goals, and projected completion
dates. However, there is no indication how SSA plans to use the
information from the evaluations
16Much of this improvement took place between the 1999 and 2000 plans. We
reported that SSA's fiscal year 2000 performance plan showed significant
improvement over its 1999 plan in U.S. General Accounting Office,
Observations on the Social Security Administration's Fiscal Year 2000
Performance Plan, GAO/HEHS-99-162R (Washington, D.C.: July 20, 1999).
17SSA reduced its strategic goals from five to four in its 2003-2008
strategic plan.
Appendix IV Observations on Agencies' Annual Performance Plans
to improve program results. The plan could be enhanced if the descriptions
of these evaluations included the manner in which SSA planned to use the
information gathered to improve its programs.
SSA's Fiscal Year 2004 Performance Plan Provides a General Discussion of
Strategies and Resources
SSA's 1999 plan had little discussion of the relationship between SSA's
mission, goals, and budget activities. Throughout the document, the fiscal
year 2004 plan included clearer discussions of the linkage between SSA's
mission and goals. It also provided performance data dating back to fiscal
year 1999, essential to making comparisons between prior and proposed
levels of performance. The 2004 performance plan noted that the Limitation
on Administrative Expenses account, SSA's basic administrative account, is
an annual appropriation that covers everything from salaries and benefits
of SSA federal employees (excluding IG) to systems and telecommunications
activities. SSA provided information on the funding sources of this
account, including some of its budget accounts.
In its fiscal year 2004 plan, SSA provided information on the strategies
it plans to use in addressing its key strategic objectives. The plan
included a summary chart, showing the strategic objectives associated with
each strategic goal, as well as the performance measures under each
objective. In addition, the "means and strategies" section associated with
each strategic objective identified strategies that support items in the
PMA, GAO and IG major management challenges, and Social Security Advisory
Board recommendations. In our October 2002 report Performance and
Accountability: Reported Agency Actions and Plans to Address 2001
Management Challenges and Program Risks,18 we noted that SSA identified
directly related goals and measures for five of its six challenges, and
had strategies (without goals or measures) for the sixth challenge.
It is difficult to determine whether or not the annual performance plan
identifies annual performance goals that cover all of the program
activities in the agency's budget, as well as the financial, human
capital, and information technology resources needed to address each
individual goal. General human capital requirements and goals are
identified as part of SSA's strategic goal to strategically manage and
align staff to support SSA's mission. The plan is neither structured by
program activity nor account. SSA noted that it aligned its strategic
goals, performance measures, and
18GAO-03-225.
Appendix IV Observations on Agencies' Annual Performance Plans
budget with its major functional responsibilities rather than by program
accounts since direct service and support employees provide services
linked to these functional responsibilities, as opposed to a specific
program. However, SSA does not indicate what it means by "functional
responsibilities," nor does it show a clear link between its strategic
goals and such responsibilities.
As in the fiscal year 1999 plan, the fiscal year 2004 plan included a
discussion of external factors that could affect the achievement of its
goals.19 SSA identified strategies to alleviate some, but not all of, the
factors. For example, SSA plans to mitigate the loss of institutional
knowledge through SSA's "retirement wave" through the use of employee
development programs, redeploying positions to direct service, hiring
Presidential Management Interns, and the increased use of hiring
flexibilities. However, the discussion of factors affecting SSA's solvency
strategic goal merely notes that Social Security programs must respond to
related developments.
SSA's Fiscal Year 2004 Performance Plan Provides General Confidence That
Performance Data Will Be Credible
SSA's 1999 plan stated that the Office of the Inspector General was
responsible for reviewing the data systems underlying its performance
measures, but did not provide further details that would assure the reader
that SSA is taking the steps necessary to ensure data integrity. In
contrast, SSA's fiscal year 2004 plan provided data sources and
definitions for each performance measure. SSA's fiscal year 2004 plan
identified data limitations related to performance measures, as well as
some efforts to correct or address data weaknesses. When performance
indicators and goals are not quantified, SSA describes its benchmarks for
goal achievement. For example, for the outcome measure "Developing new
performance management systems," SSA defines "Implementing the new SES
system" as its goal for 2003.
As in the fiscal year 1999 plan, SSA notes that the IG's office is
involved in the data system reliability process. In the fiscal year 2004
plan, SSA went further to explain the IG's four-point approach to
reviewing performance measures, including assessing whether the reported
performance measure data are valid. SSA also noted that performance data
for its quantifiable measures are generated by automated management
information and
19SSA refers to external factors as environmental factors.
Appendix IV Observations on Agencies' Annual Performance Plans
workload measurement systems, as a by-product of routine operations.
However, there is no discussion of verification and validation procedures
for data generated by these systems.
Observations on Changes in the Quality of DOT's Annual Performance Plan
DOT's annual performance plan for fiscal year 2004 showed evidence of
improvements in areas that we previously identified had shortcomings in
our 1998 review of DOT's 1999 performance plan.20 The 2004 plan provides a
clear picture of intended performance with DOT's measures and performance
goals now being clearly linked to the department's strategic objectives. A
specific discussion of DOT's strategies and resources in the plan includes
numerous and detailed strategies for achieving DOT's performance goals,
and the resources needed for those strategies. Procedures to verify and
validate data, as well as known data limitations, are described for each
performance measure providing full confidence in the credibility of DOT's
performance data. Still, the performance plan could be improved by
including a discussion of, and performance measures for, each of DOT's
program activities and by more consistently describing DOT's role in
crosscutting programs.
DOT's Fiscal Year 2004 Performance Plan Provides a Clear Picture of
Intended Performance
DOT's fiscal year 2004 performance plan shows evidence of many of the same
strengths as, and a few improvements over, its fiscal year 1999
performance plan and provides a clear picture of intended performance. The
2004 plan lists outcome goals, performance goals, and measures by
strategic objective, all of which are generally objective, quantifiable,
and can show progress toward DOT's strategic objectives. For example, the
measure "fatalities per 100 million vehicle-miles of travel" will gauge
progress toward the performance goal "reduce highway fatalities per 100
million vehicle-miles traveled to no more than 1.0 in 2008, from 1.7 in
1996." The data gathered by the measure will also show progress toward the
related outcome of "reduce the number of transportation-related deaths"
for DOT's "safety" strategic objective. This is an improvement over DOT's
1999 plan in which we found that DOT's performance goals typically covered
only a portion of the strategic goals and the link between annual
performance goals and strategic goals could be improved.
20GAO/RCED-98-180R.
Appendix IV Observations on Agencies' Annual Performance Plans
DOT's plan also presents trend and baseline data for each performance
measure and goal. For the example given above, the plan indicates the
performance targets from 1999 to 2004 and also presents actual data for
these targets for 1999 to 2002. In its 1999 plan, DOT had provided
baseline data for most of its performance goals and measures as well. This
information, along with the clearly linked performance goals and strategic
objectives, helps to show DOT's progress in achieving its goals.
As in the 1999 plan, DOT's 2004 performance plan generally covers each
program activity in its budget request for fiscal year 2004. An appendix
to the performance plan lists DOT's program activities and indicates the
proposed funding level for each program by strategic objective. However,
as in its 1999 plan, a few programs do not seem to be linked to the
strategic objectives elsewhere in the plan. Capital grants to the National
Passenger Rail Corporation (Amtrak) and the Bureau of Transportation
Statistics' Office of Airline Information are both linked to the "mobility
& economic growth" strategic objective in the budget crosswalk, but they
do not appear in the discussions contained within that strategic objective
section. When the 2004 plan was published in February 2003, DOT had not
yet released its new reform strategy for Amtrak, which was made public in
July 2003. Still, the inclusion of information on Amtrak, as well as a
discussion of the Bureau of Transportation Statistics' Office of Airline
Information, would provide a clearer picture of how DOT intends to achieve
its goals.
The discussions of each performance goal have sections entitled "Other
Federal Programs with Common Outcomes." In this section, the plan
describes crosscutting programs and other agencies with which DOT works.
For example, the plan states that the Research and Special Programs
Administration of DOT continues to develop the National Pipeline Mapping
System with the Federal Energy Regulatory Commission, the National Oceanic
and Atmospheric Administration, the Department of Energy, the U.S.
Geological Survey and others, in order to help analyze risks to
environmentally sensitive and populated areas. This supports DOT's efforts
to reduce pipeline incidents. Yet for several goals, coordination efforts
are not described. One example of this is in the highway congestion
section where the plan states that the Federal Highway Administration
works closely with the Department of the Interior, Department of
Agriculture, and Department of Defense agencies to improve mobility on
federally owned lands. However, the plan does not describe the specific
actions that are being taken to improve mobility. Our 1998 report stated
that DOT's contribution or role was not described in many of the
crosscutting programs listed in DOT's 1999 performance plan.
Appendix IV Observations on Agencies' Annual Performance Plans
The 2004 performance plan generally addresses performance and
accountability challenges we previously identified. The discussions of
these management challenges are included in the plan by the performance
goal and programs to which they are related. For example, in discussing
highway safety, DOT addresses our concerns on transportation safety,
specifically through the use of safety belts. The strategies include
continuing the National Highway Traffic Safety Administration's safety
belt outreach to high-risk populations and encouraging states to embrace
"Click It or Ticket" as the message or theme for their Buckle Up
Campaigns. A performance measure related to this management challenge
included in the plan is "percentage of front seat occupants using safety
belts." However, not all of the management challenges have related
measures and goals. For example, we have previously identified building
human capital strategies as a management challenge for DOT. A section
within the plan focuses on an "organizational excellence" objective to
implement the PMA. Strategic management of human capital strategies is
discussed in this section but no goals or measures are given to show DOT's
progress with these strategies. Still, this shows some improvement over
the 1999 plan, which generally covered management challenges, but did so
in a separate appendix without explaining how the challenges were related
to the rest of the plan. We noted that this area could be improved by
including goals and measures related to resolving these challenges.
DOT's Fiscal Year 2004 Performance Plan Provides a Specific Discussion of
Strategies and Resources
DOT's 2004 performance plan shows several improvements over its 1999
performance plan, providing a specific discussion of strategies and
resources. Discussions of each performance goal include a section titled
"Strategies and Initiatives to Achieve 2004 Target." These sections
include a variety of means by which DOT intends to accomplish its
performance goals. One example would be for DOT's performance goal to
"reduce pipeline hazmat (hazardous materials) spilled 30 percent by 2006,
from the last five years' average spill rate." The strategies for this
goal include enforcing operator qualification requirements, expanding
monitoring technology that can help prevent construction-related damage to
pipelines, and developing regulatory standards for leak detection
technology. This shows progress from when we reported that DOT's 1999
performance plan lacked sufficient information to clearly link the
strategies to performance goals in many cases.
In contrast to its 1999 performance plan, DOT's 2004 performance plan
generally discusses the human, capital, information, and other resources
needed to meet its performance goals. Each performance goal section of
Appendix IV Observations on Agencies' Annual Performance Plans
the performance plan includes a graph showing the enacted funding for
fiscal year 2002, and the proposed funding for fiscal years 2003 and 2004.
The organizational excellence objective for DOT describes DOT's human
capital and information technology resources and strategies. For example,
one of DOT's strategies for strategic management of human capital is to
"establish a corporate approach to target recruitment efforts, with
special emphasis on cross-modal, mission-critical occupations," which
includes a pilot program for centrally recruiting and training entry-level
employees for one or more mission-critical occupations.
The 2004 plan also discusses external factors that could hamper DOT's
ability to achieve its performance goals. In our review of DOT's 1999
performance plan, we noted that the plan could be improved by recognizing
more external factors and by discussing actions that DOT could take to
mitigate the effects of these factors. In contrast, external factors are
listed for most of the performance goals in the 2004 plan. For its
transportation accessibility goals, DOT states that as the population
ages, more people will require accessible public transit, for which states
and local agencies decide how best to allocate federally provided
resources. One of the strategies that addresses this external factor is
the Special Needs of Elderly Individuals and Individuals with Disabilities
grants, which DOT states will help meet the transportation needs of the
elderly and persons with disabilities when regular transportation services
are unavailable, insufficient, or inappropriate to meet their needs.
DOT's Fiscal Year 2004 Performance Plan Provides Full Confidence That
Performance Data Will Be Credible
The 2004 plan provides full confidence that DOT's performance data will be
credible. As in the 1999 performance plan, the 2004 performance plan
contains a section, entitled "Performance Data and Performance
Measurement," that discusses the means that DOT uses to verify and
validate its data. But unlike the 1999 plan in which this discussion was
broad and not linked to specific goals and measures, the 2004 plan also
contains an appendix that provides the following for each of DOT's
measures: the source of the data, limitations of the data, observations on
the quality of the data, work planned or ongoing to improve data quality,
and any known biases. Finally, DOT has compiled source and accuracy
statements,21 which provide more detail on the methods used to collect the
21Bureau of Transportation Statistics, Source & Accuracy Compendium,
http://www.bts.gov/statpol/SAcompendium.html (Washington, D.C.: Aug. 15,
2003).
Appendix IV Observations on Agencies' Annual Performance Plans
data, sources of variation and bias in the data, and methods used to
verify and validate the data.
The presentation of data limitations in DOT's 2004 performance plan also
shows progress from its 1999 plan. The Performance Data and Performance
Measurement section includes a general discussion of DOT's data
limitations. This discussion includes limitations for the internal and
external data used by the department. Specific limitations for internal
data can be found in the aforementioned source and accuracy compendium,
while details on the limitations of external data are given in the
appendix on performance measures. In our report on DOT's 1999 performance
plan, we stated that information on data limitations was lacking for most
measures and that the plan could be improved by more consistently
addressing the data limitations throughout the plan.
Appendix V
Observations on Agencies' Annual Performance and Accountability Reports
To help Congress and the President determine agencies' actual performance
and progress in achieving strategic plan goals, GPRA requires each agency
to prepare a report on program performance for the previous fiscal year.1
One of our objectives was to assess the overall quality of agencies'
annual performance and accountability reports and the extent to which
selected elements of agency reporting have improved. To meet this
objective, we judgmentally selected six agencies-Education, DOE, HUD, SBA,
SSA, and DOT-using criteria, such as agency size, primary program type,
and previous GAO reviews. To assess the overall quality and improvements
made to the agencies' performance and accountability reports, we relied on
requirements and guidance contained in GPRA and accompanying committee
report language,2 guidance to agencies from OMB for developing performance
reports,3 interviews with agency officials, the Chief Financial Officers
Act,4 our previous reports,5 and our knowledge of agencies' operations and
programs. To assess the quality of the six agencies' performance and
accountability reports, we categorized each report based on the degree to
which it addressed three characterizations: (1) picture of performance,
(2) link between resources and results, and (3) credibility of performance
information.
To assess the degree to which an agency's report provided a clear picture
of performance across the agency, we reviewed the extent to which the
report addressed elements required by GPRA. The annual performance report
should:
o describe the performance indicators established in the agency's annual
performance plan, along with the actual program performance achieved
compared with the performance goals expressed in the plan for that fiscal
year;
1Office of Management and Budget, Memorandum: Program Assessment Rating
Tool (PART) - Presentation in Congressional Justifications, M-03-06
(Washington, D.C.: 2003).
2Government Performance and Results Act of 1993, Committee on Governmental
Affairs, United States Senate, S. Rpt. No. 58, 103d Cong. 1st Sess.
(1993).
3OMB Circular No. A-11, Part 6, Preparation and Submission of Strategic
Plans, Annual Performance Plans, and Annual Program Performance Reports
(Washington, D.C.: June 2002).
4Chief Financial Officers Act of 1990 (Pub. L. No. 101-576).
5GAO-02-372 and Executive Guide: Creating Value Through World-class
Financial Management, GAO/AIMD-00-134 (Washington, D.C.: Apr. 1, 2000).
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
o review the success of achieving the performance goals of the fiscal
year;
o provide actual results for the 3 preceding fiscal years;
o evaluate the performance plan for the current fiscal year relative to
the performance achieved toward the performance goals in the fiscal year
covered by the report;
o explain and describe where a performance goal has not been met or a
corresponding level of achievement if an alternative form is used, as well
as why the goal was not met, plans and schedules for achieving the
established performance goal, and if the performance goal is impractical
or infeasible;
o describe the use and assess the effectiveness of achieving performance
goals of any waivers; and
o include the summary findings for those program evaluations completed
during the fiscal year covered by the report.6
We also looked at the extent to which the reports clearly discussed
progress achieved in addressing the major management challenges previously
identified by us or others. For agencies that choose to issue a
performance and accountability report, the Reports Consolidation Act of
2000 requires that the report include a summary of the most serious
management and performance challenges facing the agency, as identified by
their IG, and a brief assessment of the agency's progress in addressing
those challenges.
In assessing the clarity of the performance information, we also looked at
selected qualitative characteristics used by the Association of Government
Accountants, in conjunction with the Chief Financial Officers Council, in
assessing performance and accountability reports for the Certificate of
Excellence in Accountability Reporting.7 These characteristics included
(1) whether there was a clear relationship between the performance
6The Homeland Security Act (Pub. L. No. 107-296) requires agencies to
include a review of the performance goals and evaluation of the
performance plan relative to the agency's strategic human capital
management.
7Association of Government Accountants, Certificate of Excellence in
Accountability Reporting: Reviewers Checklist, Fiscal Year 2001.
(Washington, D.C.).
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
information in the report and the goals and objectives contained in the
strategic and annual performance plans, (2) the extent to which the agency
limited the measures it discussed to those that were most significant for
its programs, and (3) the extent to which the report was user friendly by
being well-organized, concise, readable, and making effective use of
graphics to ease understanding of narrative information. We characterized
the clarity of each report in one of four ways: (1) clear, (2) general,
(3) limited, or (4) unclear, based on the extent to which the 2002 report
addressed the elements required by GPRA and the other informative
practices we described.
Both GPRA and the CFO Act emphasized the importance of linking program
performance information with financial information as a key feature of
sound management and an important element in presenting to the public a
useful and informative perspective on federal spending. Similarly, the
current administration's ambitious agenda for performance budgeting,
calling for agencies to better align budgets with performance goals and
focus on capturing full budgetary costs and matching these costs with
output and outcome goals, suggests that agencies need to develop
integrated financial and performance management systems that will enable
the reporting of the actual costs associated with performance results.
Although linking resources to performance goals is not a requirement of
GPRA, the committee report for GPRA suggested that developing the capacity
to relate the level of program activity with program costs, such as costs
per unit of result, costs per unit of service, or costs per unit of
output, should be a high priority. We have reported that world-class
financial management practices call for enterprisewide systems to
integrate financial and operating data to support both management decision
making and external reporting requirements. To assess the degree to which
an agency's report discussed the relationship between resources and
results, we characterized each report as having a (1) clear relationship,
(2) general relationship, (3) limited relationship, or (4) no
relationship.
Finally, to assess the degree to which an agency's plan provided
confidence that the agency's performance information would be credible, we
examined how each report discussed the quality of the data presented. To
help improve the quality of agencies' performance data, Congress included
a requirement in the Reports Consolidation Act of 2000 that agencies
assess the completeness and reliability of their performance data. Under
the act, agencies were to begin including this assessment in the
transmittal letter with their fiscal year 2000 performance reports.
Agencies were also required to discuss in their report any material
inadequacies in the
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
completeness and reliability of their performance data and discuss actions
to address these inadequacies.
We have previously reported on other practices that enhance the
credibility of performance data that are not specifically required by
GPRA.8 For instance, discussions of standards and methods used by agencies
to assess the quality of their performance data in their performance
reports provides decision makers greater insight into the quality and
value of the performance data. We also reported on additional practices,
in several agencies' performance reports, that would help foster
transparency to the public and assist decision makers in understanding the
quality of an agency's data. The additional practices we observed included
(1) discussions of data quality, including known data limitations and
actions to address the limitations and (2) discussions of data
verification and validation procedures. To address the extent to which a
report provided confidence that performance information was credible, we
characterized each report as providing (1) full confidence, (2) general
confidence, (3) limited confidence, or (4) no confidence.
In conducting our reviews, to the extent information was available in
prior assessments, we compared our findings of agencies' fiscal year 2002
reports to our assessments of reports for fiscal year 1999.9 A more
detailed discussion of our scope and methodology and the criteria we used
can be found in appendix I. Table 13 shows the results of our assessment
of the six agencies' reports.
8GAO-02-372.
9GAO/HEHS-00-128R, GAO/RCED-00-209R, GAO/RCED-00-211R, GAO/RCED-00-207R,
GAO/HEHS-00-126R, and GAO/RCED-00-201R.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Table 13: Characterizations of Agencies' Fiscal Year 2002 Annual
Performance and Accountability Reports
Characterizations Picture of
performance (unclear, limited, general, clear) Resources linked to results
(no, limited, general, clear)
Department/agency
Data credible (no, limited, general, full)
Department of Education Limited Clear General
Department of Energy General Limited Limited
Department of Housing and General No General
Urban Development
Small Business Administration Limited General General
Social Security Administration General Limited General
Department of Transportation General No Full
Sources: U.S. Department of Education, U.S. Department of Education FY
2002 Performance and Accountability Report (Washington, D.C.: 2003); U.S.
Department of Energy, Performance and Accountability Report, Fiscal Year
2002 (Washington, D.C.: 2003); U.S. Department of Housing and Urban
Development, Fiscal Year 2002 Performance and Accountability Report
(Washington, D.C.: 2003); Small Business Administration, Fiscal Year 2002
Performance and Accountability Report (Washington, D.C.: 2003); Social
Security Administration, Performance and Accountability Report, Fiscal
Year 2002 (Washington, D.C.: 2002); and U.S. Department of Transportation,
Fiscal Year 2002 Performance and Accountability Report (Washington, D.C.:
2003).
The remainder of this appendix discusses our observations on the quality
of the agencies' annual performance and accountability reports we reviewed
and, to the extent information was available from our prior reviews, how
the quality has changed since the agencies submitted their first reports
on fiscal year 1999 performance. We did not independently verify or assess
the information we obtained from agency annual performance reports. If an
agency chose not to discuss its efforts concerning elements in the report,
it does not necessarily mean that the agency is not implementing those
elements.
Observations on the Quality of Education's Fiscal Year 2002 Performance
and Accountability Report
Education's fiscal year 2002 Performance and Accountability Report
comprises two volumes-the main volume and a second volume including
performance reports for the agency's individual programs. In our
assessment, we did not review the second volume, which includes very
detailed, discrete, and disaggregated performance information with over
350 individual measures for the Office of Civil Rights, IG, and 117
Education programs in 60 clusters.
Although Education's report included many features designed to present its
performance information clearly, the overall clarity was limited by the
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
significant amount of performance information that was unavailable to show
Education's performance results. In contrast, Education's report very
clearly related its performance to its costs by using both graphics and
text to provide the agency's estimate of appropriations associated with
achieving each of its six strategic goals, 24 objectives (long-term
goals), and individual programs. Finally, Education provided a general
level of confidence in the quality of its data, primarily because of its
recognition of the challenges it faces on the timeliness, reliability, and
validity of its data. Education's recent efforts in undertaking a
performance-based data management initiative in partnership with state
leaders to allow timely and ready access to high-quality achievement and
other performance data, which the IG said would address many of the
related concerns identified during IG audits, also aided in the level of
confidence in the data.
Education's Fiscal Year 2002 Report Provided a Limited Picture of
Performance
Education's 2002 performance report is directly aligned with the goals and
measures in the agency's 2002-2007 strategic plan and its 2002-2003 annual
plan. Of the 210 measures included in the agency's strategic plan and
annual plan, 120 were identified for measurement in fiscal year 2002, and
all of these are addressed in the performance report. The report contains
sections on changes planned to enhance performance on the basis of
results. For each measure, the performance report includes trend data,
with a table showing actual data from fiscal years 1999 through 2002; in
some cases, the table also includes data from fiscal year 1998. When data
are not provided, the table indicates that they were not applicable or not
available. Overall, the report contains clear, succinct figures and a
table summarizing the status of all 120 measures, as summarized in figure
19.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Figure 19: Summary of Education's Performance Indicators for Fiscal Year 2002
1.7%
Target almost met
4.2%
Set baseline
Target not met
Data not expected
Target met or exceeded
Data not yet available
Source: GAO analysis of U.S. Department of Education's FY
2002Performanceand Accountability Report.
However, while Education's 2002 report does review the levels of success
for its performance goals10 for fiscal year 2002, there is a critical
limitation. As we observed in our review of Education's 1999 report,11
data were not yet available for many measures in the 2002 report.
Specifically, 2002 data were available for only 41 of the 120 measures;
the rest were characterized as "Pending: Data Not Yet Available" (63) or
"Incomplete: Data Not Expected" (16). Despite the numerous strengths in
Education's 2002 performance report, the picture of performance for
Education presented in this report is limited mainly because of the lack
of data for so many of its 2002 targets. However, Education recognizes the
challenges created by its limited access to timely, reliable data:
We still face significant challenges to meeting our national education
goals. Primary among these challenges is access to timely, reliable data
on our performance in meeting our goals and implementing our programs. Our
efforts to identify effective and ineffective programs
10Education's annual performance goals are represented by its targets.
11GAO/HEHS-00-128R.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
in the Department are severely limited by the frequent absence of useful
data about them. In FY 2002 we designed a performance-based data
management initiative which will provide much more robust information
about our programs and strategic objectives, as well as provide a strong
foundation for educational research and evaluation.
This data management initiative is being undertaken in partnership with
state leaders and the software industry and is expected to result in an
electronic data system that will allow ready access to high-quality
achievement and other performance data in a timely and seamless manner in
the future. Because the lack of data for so many of its targets blurs the
picture of performance, Education should take every possible step to
complete, as quickly as possible, its newly established performance-based
data management initiative.
While lacking data for so many of its measures, Education's 2002 report
provides an explanation for measures with pending or incomplete data. For
pending data, the report states that comparisons to targets will be made
in the subsequent performance and accountability report, in addition to
citing the department's performance-based data management initiative. The
report further indicates that measures with incomplete data were so
characterized because methods to collect data were not ready in time to
measure fiscal year 2002 results, data collection did not occur, or data
collection was delayed. The report goes on to say that, for these
measures, Education will put methods in place to measure fiscal year 2003
results, develop other data sources, or revise its measures to measure
results differently. In addition, for each incomplete measure, the report
clearly describes why data are incomplete and what will be done to address
the situation. For example, for its measure on the percentage of states
with complete school accountability systems in place, as required by the
No Child Left Behind Act,12 the report explains that the requirements
under this act are more extensive than in the past, that states that had
met prior requirements may not yet meet new requirements, that the
department had decided regulation would be necessary, and that regulations
had not been finalized to define a complete school accountability system.
In addition, the report almost always included explanations of performance
and sometimes provided information on why targets were not met when that
was the case. However, such information was not always easy to find,
12Pub. L. No. 107-110, January 8, 2002. The No Child Left Behind Act of
2001 is a reauthorization of the Elementary and Secondary Education Act,
one of the major pieces of authorizing legislation for Education.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
as the report did not always include it in the same area as the related
measure.
For most measures, including those that did not meet their targets, the
report provided information on the steps Education is taking or plans to
take to improve or enhance performance. For example, the 2002 target for a
measure on the number of children attending charter schools was 690,000
and the actual result was 575,000. To improve performance on this measure,
Education's report says that it is distributing guidance and information
to encourage parents to consider charter schools, using both publications
and its Web site to promote charter school enrollment, and sponsoring a
charter schools Web site with information on federal assistance for
charter schools. However, it was not always clear how the steps cited
would improve or enhance performance. For example, for four measures on
advanced placement (AP) achievement that were not met in 2002, the
strategy given for improving performance is to continue to support
increasing AP achievement through the Advanced Placement Incentives
program, but the report does not include an explanation of, or any
information on, this incentives program.
Education's report included an appendix entitled "Findings from FY 2002
Evaluations" and included summaries of the findings from nine GAO reports
and eight other studies completed in fiscal year 2002. For example, the
Education for Homeless Children and Youth Program: Learning to Succeed
report comprised two studies that found that homeless students are best
served when promising practices are implemented as part of a comprehensive
and coordinated education program for the homeless.
The 2002 performance and accountability report also contained an appendix
consisting of the IG's summary of serious management challenges, including
financial management, federal student aid programs, information
technology, program performance and accountability, and human capital. For
each of these challenges, the IG provided information on Education's
progress in addressing them. Under program performance and accountability,
for example, the IG pointed out that a significant amount of the data used
to measure education programs were provided by state and local education
entities and that it is imperative that these data are accurate, so as to
provide Congress, OMB, and the public with an objective measure of the
success of education programs. The IG said that Education has recognized
the importance of improving data quality and addressed this issue in its
performance plan.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
The IG's summary of serious management challenges also included references
to GAO's high-risk list with respect to federal student aid programs and
human capital management, and Education's report included measures related
to both of these areas. Specifically, for its 2002 measure to have Federal
Student Aid (FSA) leave the GAO high-risk list by 2003 and not return, the
report states that the department "almost met" its 2002 target by
achieving 94 percent of its FSA High Risk Plan, and it described the
shortfall as not significant or material. In contrast, in our review of
Education's 1999 Performance Report,13 we noted that the department did
not have goals, objectives, or measures related to problems with its
student assistance programs. In addition, for the six measures in
Education's 2002 report under its strategic goal to improve the strategic
management of its human capital, the department reports that four targets
were pending, one was incomplete, and one had set the baseline. With
respect to the GPRA requirement14 that agencies' performance reports
include a review of the performance goals and evaluation of the
performance plan relative to the department's strategic human capital
management, Education's report discusses its human capital management
strategic goal and related performance goals in the context of its human
capital management plan, One-Ed.
Education's 2002 Report Showed a Clear Relationship between Resources and
Results
Education's 2002 report included information for each of its six strategic
goals and 24 objectives that clearly linked the department's resources
with its efforts to achieve specific results. While the department was not
able to break the costs down by each of its measures and targets, the
report used both graphics and text to provide the department's estimate of
appropriations associated with achieving each of its six strategic goals,
24 objectives (long-term goals), and individual programs. For example, for
each of its objectives, the report used a pie chart to show the percentage
of the department's appropriation that supports the objective, the rest of
the strategic goal the objective falls under, and the other five strategic
goals. An example is shown in figure 20 for the objective to ensure that
all students read on grade level by the third grade.
13GAO/HEHS-00-128R.
14As amended by the Homeland Security Act of 2002.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Figure 20: Inputs: Allocating Funds for Education's Objective to Ensure
That All Students Read on Grade Level by the Third Grade
Education's 2002 performance report provided general confidence that the
agency's data were credible because of its recognition of the challenges
it faces on the timeliness, reliability, and validity of its data; its
straightforward disclosure of these challenges; and its recent efforts to
address them. In Education's transmittal letter, the department secretary
said that the information contained in the report is "as complete and
reliable as we have available." However, Education's report recognized
that one of the agency's significant challenges to meeting its national
education goals is access to timely, reliable data on performance and that
the lack of useful data severely limits efforts to identify effective and
ineffective programs. The report further explained that 97 percent of the
department's funding is awarded to third parties, including, for example,
Other Goal 2
Objective 2.1
Other Goals
Source: U.S. Department of Education's FY 2002 Performance and
Accountability Report(Washington, D.C. January 2003).
The text accompanying each chart listed the dollar amount supporting the
objective's activities, the percentage of the strategic goal's allocation
that amount represented, the individual programs that supported the
objective, and the dollar amount from salaries and expenses that was
included in the dollar amount for the objective. In addition, in the
report's appendixes, a table summarizing fiscal year 2002 appropriations
and staffing allocated by goal and objective also included the FTEs under
staffing for each strategic goal and objective. Another table provided a
percentage breakdown of each objective's appropriations by 146 agency
programs.
Education's Fiscal Year 2002 Report Provided General Confidence That
Performance Data Were Credible
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
state and local agencies, that have an impact on the measurement of
results, especially the timing of data collection. Thus, Education
recognized in its report that it had limited control over the data it must
use to report results. Similarly, in the IG's summary of serious
management challenges, the report noted that Education needs to improve
its controls over the timeliness, reliability, and validity of data.
Moreover, Education's report included information on recent steps it has
taken to address its data challenges. In addition to a discussion of its
data management initiative to develop an electronic data system providing
access to timely, high-quality data, the report included an appendix with
the agency's Information Quality Guidelines and draft Data Quality
Standards presented in an abbreviated format. The discussion of data
quality standards recognizes the importance of data quality concepts to
the process of developing high-quality performance measures. The eight
standards provided are: validity, accurate definitions, accurate counts,
editing, calculation, timeliness, reporting, and burden reduction. To
facilitate the use of the standards, Education reported that it created a
data quality checklist and regularly held classes to teach staff how to
apply the standards. The IG's summary of serious management challenges
gave the department credit for this effort, pointing out that these
guidelines address many of the concerns identified during IG audits and
that the department plans to disseminate these guidelines to the chief
state school officers.
The report also provided explanations of data sources and data quality for
most measures. For example, for the measures on the percentages of 12th
grade students scoring at or above the basic and proficient levels on the
National Assessment for Educational Progress (NAEP) reading test, the
source is given as: "U.S. Department of Education; National Center for
Education Statistics, (NAEP); The Nation's Report Card, Reading." Under
data quality, the report states that NAEP data are validated using
rigorous National Center for Education Statistics statistical standards.
For most of the measures, the explanations of data quality contain similar
information on data validation. However, the data quality information only
sometimes identifies what limitations are relevant, if any. For example,
for a measure on the percentage of managers satisfied with services
received from Education's Office of Management when hiring staff, the
department relied on an internal survey of managers for its data. Although
the response rate for this survey was 22 percent, the report did not say
whether this was a limitation to the data collected. For a few of
Education's measures, the report stated that no data limitations had been
noted. It would be better if
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
the agency clearly stated whether the data for each measure had
limitations or not, and, if so, what they were.
Observations on the Quality of DOE's Fiscal Year 2002 Annual Performance
and Accountability Report
DOE's 2002 Annual Performance and Accountability Report provided a general
picture of intended performance by explaining in detail the progress made
in meeting performance measures and addressing management challenges. It
also provided a limited discussion of the costs incurred to achieve DOE's
performance goals by organizing its report by major program activities,
the costs of these activities, and their corresponding goals. Finally, the
report provided a limited level of confidence that data will be credible
because the report did not include a discussion on data limitations.
DOE's Fiscal Year 2002 Report Provided a General Picture of Performance
DOE's 2002 Performance and Accountability Report provided a general
picture of performance in meeting its goals and measures. The report
contained a detailed explanation of progress for each of its performance
measures, which were referred to as "targets," by identifying whether each
measure was met, not met, or had mixed results, as shown in figure 21. In
addition, the report identified the operational processes, technology,
human capital, and other resources used to achieve each performance
measure. The results for the past 3 years of performance measures related
to each goal were also reported so that performance trends could be
identified. However, the report did not clearly explain how the results of
the performance measures reported contributed to achieving the performance
goals in DOE's annual performance plan and strategic plan.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Figure 21: Summary of DOE's Performance Indicators for Fiscal Year 2002
Met
Mixed
Not met
.
Each performance measure that was not met or had mixed results contained a
plan of action to achieve the measure in the future. In addition, the
majority of fiscal year 2002 measures that were not met contained a clear
explanation as to why they were not met or had mixed results. For example,
a measure that required Southeastern Power Administration to meet its
planned repayment of principal of federal investment was not met due to
severe drought. DOE's report explained that to achieve the measure in the
future, Southeastern plans to change its rate design, propose rate
increases to obtain greater revenue, and increase cost recovery from fixed
charges.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
DOE also discussed the progress it made in addressing performance and
accountability challenges. The report identifies significant issues for
fiscal year 2002 that, as stated in the report, merit a higher level of
attention and focus in the department. Each of the challenges were linked
to a goal and its related performance measure(s). In addition, actions
taken to address each challenge and the progress made on those actions
were identified. For example, one of the challenges identified was the
need for DOE to meet federal requirements for improved and more
cost-effective use of information technology. A related goal to deal with
this challenge was for DOE to promote the effective management of
information technology resources in the department. To address this
challenge, the report stated that DOE realigned its management structure
for information technology issues, established an enterprisewide license
for Microsoft software, and launched an e-government applications task
force to identify high-priority e-government investments, among other
actions. In our prior review of DOE's 2001 performance report, we also
found that DOE had made progress in addressing all eight of its major
management challenges.15
It is unclear, however, how program evaluations were used to assess
performance because DOE did not include a summary of program evaluation
findings in either the fiscal year 1999 or 2002 reports. According to DOE
officials, a section on program evaluations was not included in fiscal
year 2002 and one is not planned for fiscal year 2003 in order to limit
the amount of detail included in the report.
DOE's Fiscal Year 2002 Report Showed a Limited Relationship between
Resources and Results
DOE's 2002 performance report provided a limited discussion of how its
resources were related to its performance. As in our review of the 1999
report, DOE's 2002 report, which also includes DOE's financial
information, continued its practice linking the department's performance
information to the costs of its program activities. For the majority of
its program activities, the 2002 report included the program activity's
net costs and related performance information for fiscal years 1999
through 2002. During our review of DOE's 1999 performance report, we were
supportive of DOE's efforts to link its performance goals and measures to
the program activities in the President's Budget. However, DOE has not
moved beyond presenting its performance and cost information by program
activity, instead of by strategic or annual performance goal or objective.
A report
15GAO-03-225.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
that presented cost and performance information by performance goals in
addition to other presentations would more clearly identify the costs
associated with the achievement of each goal. According to DOE officials,
the department plans to link its individual performance measures to the
costs of program activities in future reports.
DOE's Fiscal Year 2002 Report Provides Limited Confidence That Performance
Data Are Credible
DOE's reporting on data credibility has improved but is still limited.
Based on our review of DOE's 1999 performance report, a key improvement
made to the 2002 performance report was the department's ability to report
on its data validation and verification processes. DOE's 1999 report did
not discuss the implementation of DOE's verification and validation plan
or provide any evidence that the data quality was sufficient for assessing
the department's performance. The 2002 report met some requirements of the
Reports Consolidation Act by including a statement in the report's
transmittal letter assessing the completeness and reliability of the data.
The letter did not discuss any material inadequacies with DOE's
performance data. The report also included a high-level discussion on how
DOE will validate and verify its data and refers the reader to its 2003
annual performance plan for further details. For example, the report
stated that DOE's end-of-year reporting process includes certification by
heads of organizational elements on the accuracy of reported results,
followed by a review for quality and completeness by DOE's Office of
Program Analysis and Evaluation.
Although the department has improved on reporting its data verification
and validation processes, it has not improved on reporting any existing
data limitations. Neither the 1999 nor the 2002 report included an overall
discussion of the limitations to the data or steps DOE would take to
address those limitations, although the 2002 performance report did
identify minor data limitations for a few specific goals. DOE officials
stated that a discussion on data limitations was not included in the 2002
report because the department already reports on this information in the
annual performance plan and they thought it was redundant to put it in the
report.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Observations on the Quality of HUD's Fiscal Year 2002 Annual Performance
and Accountability Report
Compared to its fiscal year 1999 performance report, HUD's fiscal year
2002 Performance and Accountability Report provides a general picture of
what the department accomplished by including, among other things, a
report card listing its performance indicators with the corresponding
achievement, a list of program evaluations concluded during the fiscal
year, trend information for some of its performance indicators, a
discussion of the department's attempts to address its performance and
accountability challenges, and visual aids to illustrate information on
its performance. In a few instances, the report mentions whether certain
performance goals are impractical or unfeasible. However, the report is
not as clear as it could be because it does not (1) explain how it plans
to address performance targets that were not met during the fiscal year;
(2) include an evaluation of the fiscal year 2003 performance plan
relative to the performance information presented in the performance
report for the fiscal year; or (3) include an evaluation of the fiscal
year 2003 performance plan relative to the performance information
presented in the performance report for fiscal year 2002. The report does
not show the relationship between resources and results by linking
expended dollar amounts to specific program objectives. The report
provides general confidence to the reader that the data presented are
credible by providing background information on each performance indicator
and discussing the results and analysis of the most recent data.
HUD's 2002 Report Provided a General Picture of Performance
Overall, HUD's fiscal year 2002 Performance and Accountability Report
provides a general understanding of what the department's mission is and
what it accomplished during the previous fiscal year.16 Since we first
reviewed its report for fiscal year 1999, HUD has made progress in
developing its performance report to comply with GPRA.17 In reviewing
HUD's fiscal year 1999 performance report, we noted that it only contained
performance information for three of the department's four outcome
measures. HUD's report for fiscal year 2002 includes a report card for
each strategic goal listing performance targets that were met. The report
card also provides an explanation for many performance targets that were
not marked as being met during the fiscal year. Although the report
includes
16U.S. Department of Housing and Urban Development, Fiscal year 2002
Performance and Accountability Report (Washington, D.C.: 2003).
17GAO/RCED-00-211R.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
visual aids to enhance readers' understanding of progress made toward
attaining performance targets, HUD could enhance the report by providing a
summary of performance targets for all strategic objectives met (or not)
during the fiscal year.
HUD's performance report does not meet GPRA's requirement of including an
evaluation of its fiscal year 2003 performance plan relative to the
performance attained by the department in fiscal year 2002. Including this
evaluation could provide readers some assurance that HUD takes into
account prior performance information, such as unmet goals, to manage its
performance in the fiscal year already under way.
The report suggests that some of the performance targets that were not met
during the fiscal year were impractical or unfeasible. For example, for
its goal of "Increase the Rate of Homeownership," HUD mentions that the
indicator can be resistant to increases above an undetermined level
because homeownership is not practical or desirable for all households.
Broad economic conditions, including employment, incomes and interest
rates can affect homeownership rates. Likewise, HUD will no longer track a
performance indicator that measures the percentage of low-income housing
units containing threats to health and safety, such as exposed wiring,
unvented heaters, holes in the floor, and rodents. HUD mentions that this
indicator is not included in the fiscal year 2003 annual performance plan
because of the difficulty of attributing the results to its programs.
In several instances, HUD's annual performance report lacks a discussion
of how it plans to address unmet performance targets as required by GPRA.
For example, while HUD substantially met almost half of its performance
targets in fiscal year 2002, the report does not mention what steps the
department will take to address some of its unmet performance targets (see
fig. 22).
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Figure 22: Summary of HUD's Performance Indicators for Fiscal Year 2002
2%
Program not funded
2%
Data not reliable
No explanation
Source: GAO analysis of HUD's FiscalYear 2002 Performance and
Accountability Report.
aRather than stating if some performance targets were met or not, HUD
provided the following explanations: data not available; no performance
goal for this fiscal year; third quarter of calendar year (last quarter of
fiscal year, not entire fiscal year); calendar year ending in the current
fiscal year; calendar year ending the previous fiscal year; other
reporting period; results too complex to summarize; and baseline newly
established.
HUD continued to build upon the strengths of its earlier report by
including trend information for some performance indicators it used to
measure progress toward its targets during the past fiscal year. While not
presented consistently throughout the report, trend information provides a
context to understand HUD's performance and helps to show the extent to
which HUD exceeded or fell short of expectations set for its performance
targets. For instance, for HUD's performance indicator that tracks
increases in the share of welfare families residing in public housing that
move from welfare to work each year, the report mentions that in fiscal
year 2002 the rate was 13.1 percent compared to 19.9 percent in fiscal
year 2001. In preparing to implement the Temporary Assistance for Needy
Families program, HUD originally estimated this indicator to be around 6.5
percent in fiscal year 1997.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
HUD's fiscal year 2002 report also mentions that it concluded several
program evaluations during the fiscal year, a key requirement absent from
its performance report for fiscal year 1999. The report also provides a
brief summary of the main findings of these program evaluations. In fiscal
year 2002, HUD concluded and published reports on 21 program evaluations
covering five of its strategic goals.
Similar to our findings on HUD's previous performance reports,18 HUD's
fiscal year 2002 report discusses the steps the department took to address
decade-long management challenges. For example, while HUD's report
mentions that deficiencies remain in its financial management systems, in
fiscal year 2002 the department initiated a project to design and
implement an integrated financial system. Similarly, to address staffing
imbalances and human capital challenges, HUD implemented the last phase of
its Resource Estimation and Allocation Process in January 2002 and started
to implement the Total Estimation and Allocation Mechanism, a tool that
collects actual workload accomplishments and staff usage within the
various operating components at HUD.
HUD's 2002 Report Showed No Relationship between Resources and Results
While HUD has made some improvements in how it presents cost information
in its report, it is still not useful for linking program objectives to
specific dollar expenditures. HUD's report provides a summary of the cost
of operations by each reporting segment, such as the total amount of money
spent by the Federal Housing Authority and the Public and Indian Housing
programs, and states that the total cost for fiscal year 2002 operations
was $33 billion. However, the report does not reconcile these costs to
specific program performance objectives, limiting the reader's ability to
understand how HUD used its resources to carry out its objectives during
the fiscal year.
HUD's 2002 Report Provided HUD's fiscal year 2002 performance report
generally informs the reader on General Confidence That critical issues
about the reliability of its performance data, an issue that Performance
Data Are was not discussed in detail in its earlier report. In its
transmittal letter,
HUD briefly discusses that in some instances the data used in the
reportCredible were either incomplete and/or unreliable. The report
includes background information, results, analysis, and a discussion of
the data used for each
18GAO-03-225.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
performance indicator during the fiscal year. In discussing the data, in
some instances HUD points out issues concerning data validity and accuracy
and mentions steps HUD will take to correct problems. For example, to
address problems with its indicator on the number of homeowners who have
been assisted with the HOME program, HUD has established a team of
managers, technical staff, and contractors to make a series of
improvements to the Integrated Disbursement and Information System
beginning in fiscal year 2003, which should reduce the need for data
cleanup.
Observations on the Quality of SBA's Fiscal Year 2002 Annual Performance
and Accountability Report
SBA's fiscal year 2002 annual performance report shows several
improvements over the agency's initial report. The report shows a general
relationship between resources and results by including an analysis of
resources used by each program in fiscal year 2002. A section on data
validation and verification, which includes data limitations and remedies
for those limitations, provides a general level of confidence in the
credibility of SBA's performance data. While the 2002 report includes a
scorecard to show the agency's overall performance, the report provides a
limited picture of performance due to a lack of plans to meet unmet goals
in the future and data that were unavailable to show progress towards a
large share of SBA's performance goals, among other reasons.
SBA's 2002 Report Provided a Limited Picture of Performance
SBA's 2002 performance report19 provides a limited picture of its
performance. SBA's 2002 report includes a scorecard that shows overall
agency performance for its 2002 goals, including trend data (when
available) from fiscal year 1999 to fiscal year 2002, the fiscal year 2002
goal, and a column showing the percentage of the fiscal year 2002 goal
achieved. However, based on the performance information provided in the
fiscal year 2002 performance report, it can be difficult to gauge SBA's
progress in achieving its goals. This is similar to our findings on SBA's
1999 report, which we noted was unclear as to how well SBA performed in
achieving several of its performance goals for two of the key outcomes
addressed in
19Small Business Administration, SBA's Performance & Accountability Report
for Fiscal Year 2002 (Washington, D.C.: 2003).
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
our 2000 report.20 Figure 23 summarizes SBA's progress on its 19
performance goals for fiscal year 2002.
Figure 23: Summary of SBA's Performance Goals for Fiscal Year 2002
Goal achieved
Goal not achieved
Data not available
n=19
Source: GAO analysis of SBA's FY 2002 Annual Performance and Accountability
Report.
Data were unavailable for 10 of SBA's 19 performance goals in 2002. For
the nine goals that had performance data available, SBA met seven. SBA's
2002 performance report included explanations for all of its goals that
were unmet, deemed infeasible, or for which data were not available. For
example, the report states that "homes restored to pre-disaster
conditions" and "businesses restored to pre-disaster conditions" are no
longer goals because SBA is reviewing its outcome measures for the
disaster loan program. Also, data were not available for the "customer
satisfaction" goal in the disaster assistance program because a Customer
Service Survey for disaster loan recipients was not issued during the
fiscal year due to having not received final clearance from OMB. This
contrasts to our findings on SBA's 1999 report when the agency did not
provide explanations for not
20Our review of SBA's fiscal year 1999 report, GAO/RCED-00-207R, focused
on our observations on only three of SBA's key outcomes, as well as the
major management challenges addressed in the performance report. Since our
review of SBA's 2002 performance report used somewhat different assessment
criteria, we could not make valid comparisons on all aspects of the
reports.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
meeting several of its performance goals. However, for the two goals that
were unmet in 2002, "start-ups receiving 7(a) and 504 financing" and "jobs
created and retained by SBIC clients," the report does not describe any
plans for achieving the goals in the future. The lack of information for
over half of SBA's performance goals and the absence of plans for
achieving unmet goals limits our ability to assess the overall progress
the agency made in fiscal year 2002, as well as the likelihood that it
will improve its performance in the future.
Several other factors limited our ability to evaluate SBA's performance in
its fiscal year 2002 report. The report presents performance data in
accordance with the goal structure of SBA's 2003-2008 draft strategic
plan. The goal structure contained in the fiscal year 2002 performance
report does not directly correspond with the goal structure presented in
the 2002 performance plan. Only one of the report's strategic goals, "Help
Families and Businesses Recover from Disasters," directly corresponds to
the 2002 performance plan. Similarly, not all of the performance goals
listed in performance scorecards in both documents correspond. For
example, the performance goal "start-ups receiving 7(a) and 504 loans
viable 3 years after receiving loan," listed in the scorecard in the 2002
report, is not listed in the 2002 performance plan. The report states that
based on 2002 results SBA made "substantial modifications" to its fiscal
year 2003 goals, but the report does not specifically discuss how the
performance achieved in 2002 could affect the achievement of the 2003
goals. Finally, the report does not include the findings of program
evaluations completed in fiscal year 2002. SBA states that it was unable
to conduct program evaluations in 2002 due to a lack of funding and that
the agency has requested funding for program evaluations in fiscal years
2003 and 2004.
Similar to our findings on SBA's 1999 performance report, the agency
continues to provide information indicating the agency's progress in
addressing management challenges that have been previously identified. For
example, we have previously observed that SBA needs to improve the quality
of the performance measures that it uses for the disaster loan program.
SBA states in its 2002 report that the agency is in the process of
reevaluating its measures for the disaster loan program, and specifically
that the methodology for measuring the number or percentage of homes and
businesses restored through the program will be addressed by this review.
SBA's 2002 performance report also discusses the agency's strategic
management of human capital. One section of the report relating to the
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
PMA describes SBA's transformation plan, which is to realign the agency's
"organization, operation, and workforce to better serve its small business
customers." An appendix to the report identifies fully developing and
implementing the agency's human capital management strategy as one of
SBA's major challenges. This section lists the actions that SBA needs to
take to address this challenge as well as the progress the agency has made
in implementing these actions. However, the report does not include a
review of the performance goals and evaluation of the performance plan
relative to the agency's strategic human capital management, as required
by the Homeland Security Act of 2002.
The report also contains two broad overviews and an appendix of GAO audits
and recommendations, as well as a description of management challenges
identified by the agency's Inspector General. One chart, entitled "Status
of GAO Reviews Conducted at SBA in FY 2002," shows the review title,
status of the review (open or closed), and the number of recommendations
that came from these reviews. Another chart, entitled "Number of Open GAO
Recommendations at End of FY 2002," lists the GAO report number and title,
the year it was issued, and the number of recommendations remaining open.
Further detail is provided in an appendix to the performance report, which
lists GAO's outstanding recommendations, the status of the
recommendations, and the estimated date of completion. Another appendix
includes a report from SBA's Acting IG that describes the most serious
management challenges SBA faced in fiscal year 2002.
SBA's 2002 Report Showed a General Relationship between Resources and
Results
SBA's 2002 performance report contains analyses of resources and results
for SBA's programs that show a general relationship between resources and
results. In the description of each program's performance, the report
includes an analysis of the resources used by each program. For example,
the fiscal year 2002 cost of the Advocacy Program was estimated to be $8
million, with 50 percent of the funds going to support the Office of
Advocacy, 14 percent funding economic research, 11 percent for SBA's
executive direction support, 16 percent for fixed costs, and 9 percent
going to human resources, information technology, and procurement. The
report contains crosswalks that show the relationship between SBA's
strategic goals, outcome goals, performance goals, and programs. The
resources used by each program can then be linked through this crosswalk
to performance goals to generally show the resources needed for the
results achieved towards the goals. However, the connection of resources
to
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
results could be more explicitly stated in the report if results and
resources were also presented by performance goal.
SBA's 2002 Report Provided General Confidence That Performance Data Are
Credible
SBA's 2002 performance report provides general confidence that the
agency's performance data are credible. In the letter transmitting its
2002 performance report, SBA states that the performance data for its
credit and procurement assistance programs are complete and reliable,
based on a systematic review of these data. SBA further states that it is
"working to improve the completeness and reliability of the performance
data for the advice provided to small business through SBA's resource
partners." Data for this aspect of SBA's performance are collected through
surveys, which the agency notes are neither consistent nor comparable, and
from which client responses are difficult to obtain. This could be seen as
a material inadequacy, of which a discussion is required by the Reports
Consolidation Act. SBA discusses the actions it will take to address the
quality of the surveys by stating in the transmittal letter that it is
working to improve the survey instruments it uses to obtain performance
data.
SBA provides a detailed discussion of each performance indicator in a
section of the report on data validation and verification. For each
indicator, the report provides a definition, source, information on
validation, and means for data verification. The verification process for
several measures includes audits, independent reviews, and consistency
checks. However, for these measures this is the only discussion of
verification procedures and no further details are provided. Also, for
several measures, such as "number of start-up firms financed by 7(a) &
504" and "regulatory cost savings to small businesses," SBA states that it
does not independently verify the data.
The report also addresses limitations to its data in the section on data
validation and verification. In this section SBA states that it faces many
challenges in acquiring high-quality data on both outputs and outcomes,
from both internal and external sources. The strategies that SBA will use
to address the shortcomings of its data quality are contained in this
section as well, which include ensuring the validity of performance
measures and data, fostering organizational commitment and capacity for
data quality, assessing the quality of existing data, responding to data
limitations, and building quality into the development of performance
data.
The section on data validation and verification in the 2002 report
discusses how SBA plans to remedy the limitations for each indicator. For
example,
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
for the "customer satisfaction rate" measure of the disaster loan program,
the report states that the surveys used for this measure only determine
the satisfaction of those who received disaster loans and therefore do not
address the satisfaction of those who did not receive the loans. The
remedy listed for this limitation is to expand the survey to include all
applicants. This is an improvement from SBA's 1999 report, which we noted
did not discuss data limitations that could affect the quality of data
used by SBA to assess its performance.
Observations on the Quality of SSA's Fiscal Year 2002 Annual Performance
and Accountability Report
On the whole, SSA's 2002 performance report generally showed the agency's
progress towards its annual goals for fiscal year 2002.21 It showed
continued emphasis on outcome-oriented goals and identified relevant
results that were linked to individual strategic objectives. It also
provided trend information, typically back to fiscal year 1999, and
contained a brief discussion of the program evaluations completed during
fiscal year 2002. SSA's strategic goals were linked to financial resources
at a very high level, but none of the performance goals were associated
with costs; thus, the cost of achieving (or not achieving) a particular
goal was not clear. Additionally, the SSA Commissioner certified that
SSA's data presentation was credible, but missing data and a lack of
documentation of the methods and data used to measure its performance
reduced the overall quality of the document.
SSA's 2002 Report Provided SSA's 2002 performance report exhibited a
general description of
a General Picture of performance, including the identification of 14 key
performance indicators
Performance out of a total of 69 indicators. Similar to our review of
SSA's fiscal year 1999 report,22 we found that many of SSA's fiscal year
2002 goals and indicators were outcome oriented. In the fiscal year 2002
report, SSA plainly summarized progress on its 69 performance indicators,
as shown in figure 24.
21Social Security Administration, Performance and Accountability Report,
Fiscal Year 2002 (Washington, D.C.: 2002).
22Our review of SSA's fiscal year 1999 performance report
(GAO/HEHS-00-126R) focused on our observations on five of the agency's key
outcomes, as well as the major management challenges addressed in the
performance report. Since our review of SSA's 2002 performance report used
different assessment criteria, we could not make valid comparisons on all
aspects of the reports.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Figure 24: Summary of SSA's Performance Goals for Fiscal Year 2002
Data not yet available
Not met
Almost met
Met
Source: SSA's FY 2002 Annual Performance and Accountability Report.
In SSA's 1999 annual performance report, performance measures focused on
activities rather than results, so it was difficult to determine the
agency's real progress in achieving results. For example, the measures
directly related to the outcome "long-term disability benefits are reduced
because people return to the workplace" did not sufficiently track
progress toward this key outcome. One of the measures was to "begin
implementation of the `Ticket to Independence' program, contingent upon
enactment of supporting legislation in FY 1998."23 This measure was
neither quantifiable nor measurable, and did not measure the number of
beneficiaries achieving this outcome.
In the 2002 report, SSA identified relevant results that are linked to
strategic objectives. For example, one of SSA's objectives related to the
strategic goal "deliver citizen-centered, world-class service" was to
"maintain the accuracy, timeliness, and efficiency of service to people
applying for Old Age and Survivors Insurance (OASI) and Supplemental
Security Income (SSI) Aged benefits." SSA reported on the timeliness of
OASI and SSI claims, as well as the implementation of software and
infrastructure for paperless processing of claims, as the relevant
results.
23This program was an administration proposal to test allowing disabled
beneficiaries to choose their own public or private vocational
rehabilitation provider.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
In its 1999 performance report, SSA noted that a number of its goals were
not met, such as those relating to accurate and timely disability
determinations. Also, data on the accuracy of decisions at the initial
level were not available, and accuracy at the appellate level was not
measured. In its 2002 report, 10 percent of the goals were not met and 10
percent were almost met. SSA's report provided explanations for 17
performance goals SSA did not meet. However, not all of the explanations
actually identified the reasons for SSA's not meeting its goals. For
example, SSA did not meet its goals for the performance indicators
"percent of 800-number calls handled accurately-payment" and "percent of
800-number calls handled accurately-service." The explanation noted that
several quality initiatives were implemented, but SSA did not provide
explanations as to why the goals were not met. SSA also noted that some of
its performance indicators were being eliminated in favor of more focused
and outcomebased goals. In some cases, SSA identified future plans to
improve performance.
In SSA's 2002 performance and accountability report, trend information was
generally available for comparison of data back to fiscal year 1999. This
information was helpful in making an assessment whether SSA was making
progress towards its goals. SSA noted that it addressed all the IG's list
of major management challenges in its report, and that it addressed the
major management challenges we identified in its annual performance plan.
SSA also addresses the progress it made against certain challenges GAO and
the IG identified during fiscal year 2002 in its performance and
accountability report.24 For example, SSA highlights components of its SSI
Corrective Action Plan that are geared to improve the administration of
the SSI program and get it removed from our high-risk list. The IG's
report noted that SSA needs to have performance goals and measures that
address the major management challenges facing SSA, as they are not all
addressed. For example, performance measures were not established to
address problems with the Earnings Suspense File and the integrity of the
representative payee process.
24In our October 2002 report GAO-03-225, we noted that SSA reported
progress on all six of its major management challenges in its fiscal year
2001 annual performance report.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Finally, SSA's performance and accountability report contained a
discussion of the program evaluations conducted, organized by strategic
goal. However, the program evaluations SSA identified were typically
surveys of people who did business with SSA or assessments of internal
needs, such as a survey of training effectiveness and water/air quality
surveys. While this is a slight improvement over its 1999 report, where
there was only a brief summary of program evaluations, it would be helpful
for SSA to report on whether and how its evaluations have helped answer
questions about program performance and results. We have previously
reported that evaluations can help agencies improve their measurement of
program performance and/or understanding of performance and how it might
be improved.25
SSA's 2002 Report Showed a Limited Relationship between Resources and
Results
In the fiscal year 2002 performance and accountability report, SSA's
performance goals were not aligned by budget account-rather, they were
associated with strategic goals, which in turn cross budget accounts and
programs. Thus, the monetary, human capital, and technological resources
necessary to achieve many performance goals were not adequately described.
The financial statements show a schedule of financing and a schedule of
budgetary resources for each of SSA's major programs, and operating
expenses were associated with four out of the five strategic goals.26
However, these resources were not broken down by performance goal, and
were not linked to outcomes. Additionally, as reported by the IG, SSA
needs to further develop its cost accounting system, which it began to use
in fiscal year 2002; such a system would help to link costs with
performance.27
SSA's 2002 Report Provided While SSA provides data sources and definitions
for the data supporting its General Confidence That performance
indicators, some data issues continue to detract from SSA's Performance
Data Are performance report. SSA's transmittal letter noted that the
performance
and financial data presented are fundamentally complete and reliable,
andCredible that no material inadequacies were identified. Data sources
are identified
25We characterized program evaluations and their uses in GAO/GGD-00-204.
26SSA noted that its fifth strategic goal, "Valued Employees," supports
the accomplishment of all its basic functions, so its resources are
inherently included in the other four goals.
27SSA began to implement an improved cost accounting system in fiscal year
2002, which will be phased in over the next 3 to 4 years.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
for many of the performance indicators, such as findings from evaluations
and quality assurance reports. In certain cases, data limitations are
identified; for example, SSA noted that data to support the "Percent of
SSNs issued accurately" goal does not include SSNs (social security
numbers) assigned via the Enumeration-at-Birth process and major errors
identified by the Office of Quality Assurance that do not include these
SSNs result in SSN cards being issued erroneously. Some data verification
procedures are noted in the report, but verification procedures are not
consistently discussed and data reliability and completeness is not
ensured. The IG noted that of the 21 measures it reviewed, 16 were
reliable; data or documentation of the methods used to measure SSA's
performance were not available for the other five measures. The IG went
further to say that even for the performance measures found to be
reliable, SSA lacks documentation of the methods and data used to measure
its performance. Finally, data were not available for 17 percent of the
performance goals, so it was difficult to assess whether or not progress
had been made in those areas.
Observations on the Quality of DOT's Fiscal Year 2002 Annual Performance
and Accountability Report
DOT's fiscal year 2002 performance report provided information that
generally showed the department's performance and progress toward its
goals. Summary tables within the report showed when DOT met or did not
meet its targets and the report supplied brief analyses as to whether or
not DOT would likely meet its targets for fiscal year 2003 based on actual
performance in 2002. A separate section of the report on performance data
completeness and reliability, along with an on-line compendium, provides a
full level of confidence in DOT's performance data. However, the report
does not clearly show the relationship between resources and results.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
DOT's 2002 Report Provided a General Picture of Performance
DOT's fiscal year 2002 performance report28 provides a general picture of
the department's performance. The report includes performance summary
tables, which show the progress made toward each strategic objective.
These performance summaries include actual performance data from fiscal
years 1996 to 2002 when possible, as well as the performance targets for
fiscal year 2002 and whether or not the target was met. Similarly, we
noted in our 2000 report29 reviewing DOT's 1999 performance report that
performance information was clearly articulated, with summary tables
listing the fiscal year 1999 goals and trend data, and checkmarks to
indicate whether or not goals were met. Figure 25 summarizes DOT's overall
performance on its 40 performance targets, as reported in its 2002 report.
Figure 25: Summary of DOT's Performance Indicators for Fiscal Year 2002
Met
Not met
Data not available
According to the report, DOT met 24 (60 percent) of its performance
targets. Fourteen (35 percent) of DOT's performance targets were not met.
28U.S. Department of Transportation, Fiscal Year 2002 Performance and
Accountability Report (Washington, D.C.: 2003).
29Our review of DOT's fiscal year 1999 report, GAO/RCED-00-201R, focused
on our observations regarding only four of the department's key outcomes,
as well as the major management challenges addressed in the performance
report. Since our review of DOT's 2002 performance report used different
assessment criteria, we could not make valid comparisons on all aspects of
the reports.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
The report provides explanations for why five of these targets were unmet.
For example, the target for the measure "number of passengers (in
millions) in international markets with open skies aviation agreements" of
the "mobility and economic growth" strategic objective was unmet. The
target was set at 59.7 million passengers, while DOT's preliminary
estimate for this measure indicated there were 57 million passengers. The
report states that this target was unmet because passenger travel
diminished in fiscal year 2002 due to the impact that the events of
September 11, 2001, had on air travel. However, the report did not provide
explanations describing why the nine other targets were not met. A DOT
official stated that explanations for these unmet targets were not
included in the 2002 report due, in part, to time constraints. In our 2000
report of DOT's 1999 performance report, we stated that for all of its
unmet goals except transit fatalities, the department provided
explanations for not meeting the goals related to the outcomes we
observed.
We noted in our report on DOT's 1999 performance report that the
department supplied strategies to achieve its unmet goals in the future,
for the areas we reviewed. However, of the 14 unmet targets in the fiscal
year 2002 report, DOT provided future plans to achieve only two. For
example, the report provided a plan for future achievement of the unmet
target "percent of environmental justice cases unresolved after one year."
The report stated that DOT's External Complaint Tracking System was being
revised "to track complaints more closely, in a more timely way, and with
a higher level of data quality." DOT is also developing guidance requiring
more intensive legal staff involvement in external civil rights
complaints, especially environmental justice cases. A DOT official stated
that future plans were not included in the 2002 report for the other unmet
targets due, in part, to time constraints.
Data were unavailable for two of DOT's measures, "cumulative average
percent change in transit passenger-miles traveled per transit market" and
"employment sites (in thousands) made accessible by Job Access and Reverse
Commute (JARC) transportation services." The report explains the reasons
why data were unavailable for both of these measures and includes plans to
provide these data in the future. Data were unavailable for the JARC
program performance measure because DOT had not received data from JARC
grantees to verify that fiscal year 2002 program targets had been
achieved. DOT states that a new reporting system is being implemented,
which should improve data gathering performance.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
Although the report does not identify any performance goals that were
impractical or infeasible, it states that the measure on transit
passengermiles traveled had been changed in fiscal year 2002 because "it
placed excessive emphasis on increasing ridership in the Nation's very
largest urban areas." However, after using the measure for one year, DOT
concluded that the measure should once again be modified to account for
changes in the level of employment in each urban area. The report states
that a recent study found that changes in the level of employment are a
key economic factor related to changes in the level of transit ridership.
Another strength of DOT's fiscal year 2002 performance report is an
analysis of whether or not DOT will likely meet its planned performance
targets for fiscal year 2003. Each discussion of performance goals
contains an evaluation of the fiscal year 2003 performance plan target and
whether or not it will likely be met based on the fiscal year 2002 data.
For example, for its highway fatality rate targets, DOT says that the
National Highway Traffic Safety Administration and the Federal Motor
Carrier Safety Administration will be challenged to meet the established
fiscal year 2003 targets because targets had not been met for fiscal year
2002. In other instances where DOT is sure that it will meet its targets
for fiscal year 2003, it simply states so, as in the case of its aviation
safety targets.
DOT's 2002 performance report also includes information on completed
program evaluations. There is a separate section in the report that
discusses DOT's fiscal year 2002 program evaluations. Summaries of the
findings of these evaluations are discussed in this section. For example,
an evaluation of the Noise Set-Aside Portion of the FAA (Federal Aviation
Administration) Airport Improvement Program found that funding for the
program's noise compatibility projects was variable from year to year,
making it difficult to forecast annual population benefits.
As we found in the 1999 report, the major management challenges that DOT
faces are generally discussed in the 2002 report. These discussions were
contained within the program section to which they relate. The report also
showed the progress DOT has made in addressing its management challenges.
For example, we noted in our January 2003 report30 on DOT's major
management challenges that FAA's financial management systems remained at
high risk. DOT's 2002 report states that FAA created an Interim Fixed
Asset System to centrally control and
30GAO-03-108.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
account for its property and that in fiscal year 2003, FAA will convert to
use Delphi, DOT's financial accounting system.
The 2002 performance report included a discussion on strategic human
capital management as part of DOT's "organizational excellence" strategic
objective. This discussion included a brief overview of DOT's human
capital plan as well as strategies for strategic human capital management.
For example, the report noted that FAA was redirecting 37,300 employees
into a results-oriented Air Traffic Organization, "freeing most of the FAA
to manage better and modernize more efficiently." However, the report did
not include a review of the performance goals and evaluation of the
performance plan relative to the agency's strategic human capital
management, as required by the Homeland Security Act of 2002.
DOT's 2002 Report Showed DOT's 2002 performance report did not show the
relationship between the No Relationship between resources it used and the
results it achieved in fiscal year 2002. The Resources and Results
financial portion of the report provided a statement of net cost for each
of
DOT's programs in fiscal year 2002. The report could be improved by
providing net cost information for DOT's performance goals in the
performance section of the report, similar to the funding information
provided in the 2004 performance plan.
DOT's 2002 Report Provided Full Confidence That Performance Data Are
Credible
DOT's 2002 performance report provided a full level of confidence that the
department's performance data were credible. In his transmittal letter,
the Secretary of DOT stated that the 2002 report "contains performance and
financial data that are substantially complete and reliable." The letter
also stated that a section of the report assessed the inadequacies of
DOT's performance data and provided plans to remedy those inadequacies.
The "Performance Data Completeness and Reliability" section of the 2002
report generally discussed data completeness, reliability, and
limitations. This section discussed an overall limitation in DOT's
performance data. The report stated that much of DOT's performance data
came from external sources, and therefore, the department had no direct
control over the quality of these data. The report continues by stating
that DOT takes limitations to its external data into account when it uses
these data.
Appendix V Observations on Agencies' Annual Performance and Accountability
Reports
The 2002 report noted that DOT has compiled source and accuracy statements
that provide detail on the methods used to collect performance data,
sources of variation and bias in the data, methods used to verify and
validate the data, as well as data limitations.31 However, the online
Source and Accuracy Compendium does not include this information for the
goals and measures related to the department's organizational excellence
objective. The compendium states that a small number of source and
accuracy statements are not yet completed and that they will be added upon
completion.
Finally, the 2002 report also described strategies being undertaken to
address the quality of data used by DOT. The report stated that a DOT
intermodal working group addressed data quality issues by developing
departmental statistical standards and by updating source and accuracy
statements for all of DOT's data programs. The working group also worked
to improve quality assurance procedures, evaluate sampling and nonsampling
errors, and develop common definitions for data across modes.
31Bureau of Transportation Statistics, Source & Accuracy Compendium,
http://www.bts.gov/statpol/SAcompendium.html (Washington, D.C.: Aug. 15,
2003).
Appendix VI
GAO Federal Managers' Survey Data
Q1. What is your current grade level?
GS/GM-13 or equivalent (percent)
GS/GM-14 or equivalent (percent) GS/GM-15 or equivalent (percent)
Senior
Executive Service or equivalent
(percent)
Other - please
specify -Continue with next question
(percent)
Number of respondents
35.2 24.6 7.9 0.9 500
Q1a. If you answered "Other" in question 1 above, please enter your
response below.
Writing comment (percent)
Number of respondents
3
Q2. In total, for how many years have you been a supervisor and/or a
manager in the federal government?
Number of Mean Median Minimum Maximum respondents
13 1 43497
Q2no. Or, If you have never been a supervisor or a manager in the federal
government, please check the box below.
Percent
Number of respondents
3
Q3. In your current role, approximately how many government employees are
you responsible for? (Please answer for your permanent position. Please
specify the total number. If none, enter 0.) Enter numeric digits only.
Employees:
Number of Mean Median Minimum Maximum respondents
15 1 11,000 475
Q4. Please indicate where you currently work. (If you are currently on
temporary assignment or on detail, please answer for your permanent work
location.)
Headquarters of my department or agency (percent)
A field office of my department or agency (percent)
Other - please specify -Continue with next question
(percent)
No answer (percent)
Number of respondents
28.9 60.7 9.5 1.0 503
Appendix VI
GAO Federal Managers' Survey Data
Q4a. If you answered "Other" in question 4 above, please enter your response
below.
Writing comment (percent)
Number of respondents
38
Q5. For those program(s)/operation(s)/project(s) that you are involved
with, to what extent, if at all, do you consider your agency's strategic
goals when participating in the following activities? (Check one box in
each row.)
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
a. Setting program
priorities 33.9 45.3 11.4 4.7 1.8 2.2 0.7
b. Allocating resources 30.3 39.9 18.9 4.8 1.8 3.1 1.3
c. Adopting new program
approaches or changing
work processes 33.6 39.2 16.5 4.6 2.4 1.9 1.8
d. Developing or refining
program performance
measures 29.0 37.0 16.3 8.3 4.1 4.4 0.9
Q6. Are there performance measures for the program(s)/operation(s)/
project(s) that you are involved with?
Yes No Do not know No answer Number of (percent) (percent) (percent)
(percent) respondents
6.2 3.8 1.0 503
Q7. To what extent, if at all, do you agree with the following statements
as they relate to performance measures for the program(s)/operation(s)/
project(s) that you are involved with? (Check one box in each row.) We
have performance measures that...
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
a....tell us how many things
we produce or services we
provide. (Output
measures) 25.8 28.7 19.4 8.4 4.8 1.8 11.0 503
b....tell us if we are
operating efficiently.
(Efficiency measures) 15.2 27.5 25.8 13.4 5.5 1.5 11.0 503
c....tell us whether or not
we are satisfying our
customers. (Customer
service measures) 16.8 29.8 21.6 12.6 6.1 1.6 11.4 503
Appendix VI
GAO Federal Managers' Survey Data
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
d....tell us about the quality
of the products or services
we provide. (Quality
measures) 16.0 30.3 23.5 12.6 5.6 0.7 11.3
e....demonstrate to
someone outside of our
agency whether or not we
are achieving our intended
results. (Outcome
measures) 19.1 35.9 19.0 10.5 2.6 1.5 11.3
f....link our product or
service costs with the
results we achieve. (Cost
benefit measures) 12.5 18.7 19.7 20.0 11.7 6.0 11.3
Q8. For those program(s)/operation(s)/project(s) that you are involved
with, to what extent, if at all, do you use the information obtained from
performance measurement when participating in the following activities?
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
a. Setting program
priorities 13.0 38.7 21.7 10.8 3.4 1.2 11.0
b. Allocating resources 14.9 36.6 20.3 10.8 3.7 1.8 12.0
c. Adopting new program
approaches or changing
work processes 13.9 34.8 24.2 10.2 4.1 0.9 11.9
d. Coordinating program efforts with other internal or external
organizations 10.1 32.2 29.4 10.9 3.6 2.7 11.1
e. Refining program
performance measures 14.4 28.8 24.7 13.2 3.4 4.1 11.3
f. Setting new or revising
existing performance goals 16.1 32.4 21.0 11.8 2.9 3.5 12.3
g. Setting individual job
expectations for the
government employees I
manage or supervise 18.4 33.6 22.0 9.3 2.8 2.4 11.4 503
h. Rewarding government
employees I manage or
supervise 17.7 33.3 21.8 8.2 4.0 3.0 11.9 503
i. Developing and
managing contracts 7.3 18.8 19.3 10.4 8.1 24.4 11.7 503
Appendix VI
GAO Federal Managers' Survey Data
Q9. Based on your experience with the program(s)/operation(s)/project(s)
that you are involved with, to what extent, if at all, have the following
factors hindered measuring performance or using the performance
information?
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
a. Setting program
priorities 14.2 20.3 32.1 19.7 9.1 3.9 0.7
b. Different parties are
using different definitions
to measure performance 10.3 23.5 24.6 24.6 10.4 5.6 1.0
c. Difficulty obtaining
valid or reliable data 7.5 22.1 26.0 27.2 12.0 4.1 1.0
d. Difficulty obtaining
data in time to be useful 7.6 17.5 23.9 26.0 18.3 4.4 2.2
e. Lack of incentives
(e.g., rewards, positive
recognition) 12.9 15.8 25.0 24.1 16.6 4.8 0.8
f. Difficulty resolving
conflicting interests of
stakeholders, either
internal or external 9.5 19.3 25.5 22.0 14.5 8.1 1.0
g. Difficulty distinguishing
between the results
produced by the program
and results caused by
other factors 7.4 18.7 24.3 28.6 13.7 6.3 1.0
h. Existing information
technology and/or
systems not capable of
providing needed data 9.7 19.4 20.6 26.6 17.0 5.4 1.3
i. Lack of staff who are
knowledgeable about
gathering and/or
analyzing performance
information 10.1 17.5 23.4 26.2 16.5 5.0 1.3
j. Lack of ongoing top
executive commitment or
support for using
performance information
to make program/funding
decisions 9.7 16.0 18.4 24.0 23.1 7.9 1.0 503
Appendix VI
GAO Federal Managers' Survey Data
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
k. Lack of ongoing
Congressional
commitment or support
for using performance
information to make
program/funding
decisions 7.1 16.6 18.1 16.5 17.1 23.8 0.7
l. Difficulty determining
how to use performance
information to improve
the program 5.3 12.8 30.1 26.6 18.8 5.6 0.7
m. Concern that OMB will
micromanage programs
in my agency 7.9 10.8 14.7 19.2 26.6 19.7 1.0
Q10. To what extent, if at all, do you agree with the following
statements? (Check one box in each row.)
To a very To a No basis to great To a great moderate To a small To no
judge/Not extent extent extent extent extent applicable No answer Number
of (percent) (percent) (percent) (percent) (percent) (percent) (percent)
respondents
a. Agency managers/
supervisors at my level
have the decision making
authority they need to
help the agency
accomplish its strategic
goals. 9.5 30.1 28.1 25.2 5.6 0.6 1.0
b. Agency managers/
supervisors at my level
are held accountable for
agency accomplishment
of its strategic goals. 14.6 42.9 24.1 12.7 3.3 1.5 1.0
c. Agency managers/
supervisors at my level
are held accountable for
the results of the
program(s)/
operation(s)/project(s)
they are responsible for. 24.2 46.5 17.2 7.2 3.0 0.9 1.0 503
Appendix VI
GAO Federal Managers' Survey Data
To a very To a No basis to great To a great moderate To a small To no
judge/Not extent extent extent extent extent applicable No answer Number
of (percent) (percent) (percent) (percent) (percent) (percent) (percent)
respondents
d. Employees in my
agency receive positive
recognition for helping the
agency accomplish its
strategic goals. 9.7 27.1 31.2 22.0 7.2 1.5 1.3
e. My agency's top
leadership demonstrates
a strong commitment to
achieving results. 24.4 37.1 21.8 9.9 2.9 2.6 1.3
f. My agency is investing
the resources needed to
ensure that its
performance data is of
sufficient quality. 9.5 21.4 29.3 21.9 7.4 9.3 1.0
Q11. To what extent, if at all, do you agree with the following
statements? (Check one box in each row.) The following items focus on the
program(s)/operation(s)/project(s) that you are responsible for.
To a very To a No basis
to
great To a great moderate To a small To no judge/Not
extent extent extent extent extent applicable No answer
Number of
(percent)
(percent) (percent) (percent) (percent) (percent) (percent) respondents
a. The individual I report to
periodically reviews with
me the results or outcomes
of the program(s)
operation(s)/project(s) that
I am responsible for. 18.8 36.1 23.1 15.3 5.3 0.4 1.0
b. Funding decisions for
the program(s)/
operation(s)/project(s) I am
responsible for are based
on results or outcome
oriented performance
information. 4.5 20.9 23.1 26.4 16.5 7.5 1.0
c. Staffing and personnel
decisions for the
program(s)/
operation(s)/project(s) I am
responsible for are based
on results or outcome
oriented performance
information. 4.2 20.9 27.8 23.2 19.0 3.7 1.3 503
Appendix VI
GAO Federal Managers' Survey Data
To a very To a No basis
to
great To a great moderate To a small To no judge/Not
extent extent extent extent extent applicable No answer
Number of
(percent)
(percent) (percent) (percent) (percent) (percent) (percent) respondents
d. Changes by
management above my
level to the program(s)/
operation(s)/project(s) I am
responsible for are based
on results or outcome
oriented performance
information. 2.6 20.5 25.2 26.5 15.3 8.2 1.6
e. It is easy to motivate
employees to be more
results-oriented in the
program(s)/operation(s)/
project(s) I am responsible
for. 3.5 22.0 33.6 29.0 7.8 2.8 1.3
f. I have sufficient
information on the validity
of the performance data I
use to make decisions. 4.7 32.6 30.2 20.3 7.1 3.2 1.8
Q12. During the past 3 years, has your agency provided, arranged, or paid
for training that would help you to accomplish the following tasks? (Check
one box in each row.)
Yes No No answer Number of (percent) (percent) (percent) respondents
a. Conduct strategic planning 46.7 52.3 1.0 503
b. Set program performance goals 48.8 50.2 1.0 503
c. Develop program performance
measures 42.9 55.5 1.6 503
d. Assess the quality of performance
data 35.3 63.5 1.3 503
e. Use program performance
information to make decisions 40.7 56.8 2.4 503
f. Link the performance of
program(s)/operation(s)/project(s) to
the achievement of agency strategic
goals 40.8 57.0 2.2 503
g. Implement the requirements of the
Government Performance and Results
Act (GPRA or the Results Act) 31.9 66.7 1.3 503
Appendix VI
GAO Federal Managers' Survey Data
Q13. In your opinion, to what extent, if at all, do you believe you need
training (or additional training) in order to help you to accomplish the
following tasks? (Check one box in each row.)
To a very To a No basis to great To a great moderate To a small To no
judge/Not extent extent extent extent extent applicable No answer Number
of (percent) (percent) (percent) (percent) (percent) (percent) (percent)
respondents
a. Conduct strategic
planning 8.2 18.3 27.8 30.8 11.2 2.8 0.9
b. Set program
performance goals 8.6 19.7 27.6 27.2 12.7 3.0 1.2
c. Develop program
performance measures 9.6 21.1 29.2 23.7 11.2 3.6 1.6
d. Assess the quality of
performance data 8.9 22.1 25.7 25.7 13.3 2.5 1.9
e. Use program
performance information
to make decisions 9.5 22.1 23.7 26.9 13.8 2.2 1.9
f. Link the performance of
program(s)/operation(s)/p
roject(s) to the
achievement of agency
strategic goals 12.0 21.6 24.2 27.3 11.2 2.8 0.9
g. Implement the
requirements of the
Government Performance
and Results Act (GPRA or
the Results Act) 12.0 25.0 26.4 17.6 11.1 6.9 1.0
Q14. What, in your opinion, can the Federal government do to improve its
overall focus on managing for results?
Writing comment (percent)
Number of respondents
503
Q15. Prior to receiving this questionnaire , which of the following statements
best describes your awareness of GPRA?
I had heard I had heard I had heard
of of of
I had heard GPRA and had GPRA and had GPRA and
of
GPRA but had a low level moderate had extensive
no of
I had knowledge of knowledge of knowledge of
never its its its knowledge of
heard of requirements. requirements. requirements. its No answer
GPRA. requirements. Number of
(percent) (percent) (percent) (percent) (percent) (percent)
respondents
19.5 13.5 24.5 35.4 5.6 1.5 503
Appendix VI
GAO Federal Managers' Survey Data
Q16. For those program(s)/operation(s)/project(s) that you are involved
with, to what extent,if at all, do you consider the annual performance
goals set forth in your agency's GPRAannual performance plan when
participating in the following activities?
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
a. Setting program
priorities 7.5 19.8 24.4 12.7 13.5 20.8 1.2
b. Allocating resources 5.2 16.8 23.9 15.7 14.7 22.5 1.2
c. Adopting new program approaches or changing work processes 5.9 21.3
23.8 13.5 13.3 21.1 1.2
d. Coordinating program efforts with other internal or external
organizations 5.4 16.2 26.4 15.7 12.8 22.3 1.2
e. Developing or refining
program performance
measures 5.7 18.1 21.5 15.9 14.7 22.9 1.2
f. Setting individual job
expectations for the
government employees I
manage or supervise 5.7 22.0 19.7 15.4 15.0 20.5 1.6
g. Rewarding
government employees I
manage or supervise 5.9 19.5 20.4 16.1 16.2 20.5 1.5
h. Developing and
managing contracts 3.3 12.3 14.9 13.3 16.4 37.1 2.7
Q17. During the past 3 years, have you been involved in these activities?
(Check one box in each row.) GPRA-related activities
Yes No No answer Number of (percent) (percent) (percent) respondents
a. Developing ways to measure
whether program performance goals
are being achieved. 45.8 52.7 1.5 503
b. Gathering and analyzing data to
measure whether programs are
meeting their specific performance
goals. 50.6 47.9 1.5 503
c. Using measures for program
performance goals to determine if the
agency's strategic goals are being
achieved. 42.6 55.9 1.5 503
d. Assessing the quality of data used
in measuring performance. 39.7 58.2 2.1 503
Appendix VI
GAO Federal Managers' Survey Data
Q18. To what extent, if at all, do you agree with the following
statements? (Check one box in each row.) Extent I agree with the following
statements:
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
a. The objectives of my
program(s)/operation(s)/pr
oject(s) are in alignment
with my agency's strategic
plan under GPRA. 10.1 29.4 15.8 7.2 1.2 35.0 1.2
b. The costs associated
with implementing GPRA
have taken time or funds
away from other important
activities or projects. 4.8 7.7 13.1 15.1 10.6 46.1 2.5
c. The benefits to my
agency that are achieved
by implementing GPRA
are worth the costs
incurred in doing so (e.g.,
in time, money, and effort). 3.8 11.3 15.8 12.4 7.4 48.0 1.3
d. GPRA strategic and
annual performance plans
are mostly a repackaging
of goals, measures, and
objectives that were
already being used within
my agency. 5.7 22.7 19.1 7.7 2.8 40.1 1.9
e. Managerial
effectiveness is impeded
by the lack of integration
between GPRA and other
federal management
programs. 3.8 7.7 16.3 13.4 6.8 50.5 1.5
f. My agency considers
contributions to and
comments on GPRA plans
or reports from
managers/supervisors at
my level: 3.5 10.6 15.6 15.0 11.7 42.1 1.5 503
Appendix VI
GAO Federal Managers' Survey Data
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
g. Agency managers/
supervisors at my level
use GPRA annual
performance plans to
manage their
program(s)/operation(s)/pr
oject(s). 3.6 7.8 18.5 16.4 17.7 33.9 2.1
h. GPRA's planning and
reporting requirements
impose a significant
paperwork burden. 2.9 7.4 17.1 15.9 7.3 47.7 1.8
i. GPRA has caused
agency managers/
supervisors at my level to
place a greater emphasis
on getting input from
appropriate stakeholders
on their interests and
expectations. 2.5 7.0 20.8 13.1 12.2 42.2 2.1
Q19. To what extent, if at all, do you believe that GPRA has improved your
agency's ability to deliver results to the American public.
To a very To a To a
great great moderate
extent extent extent No basis
(Continue to
(Continue (Continue with To a judge/Not
with with question small
question question 19a.) extent To no applicable No answer Number of
19a.) 19a.) extent
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
1.6 6.9 14.5 25.2 11.6 38.0 2.1 503
Q19a. Please briefly describe how GPRA has improved your agency's ability
to deliver results to the American public.
Writing comment Number of (percent) respondents
139
Appendix VI
GAO Federal Managers' Survey Data
Q20. To what extent, if at all, do you believe the following persons or
entities pay attention to your agency's efforts under GPRA? (Check one box
in each row.)
To a very To a No basis
to
To a moderate To a To no judge/Not
great great small
extent extent extent extent extent applicable No answer Number of
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
a. Department Secretary
(if applicable) 10.3 18.3 10.6 6.2 3.6 47.1 3.9
b. Agency head other
than Department
Secretary (if applicable) 11.4 20.6 12.9 6.9 2.1 42.9 3.2
c. The individual I report
to 8.5 16.0 19.3 14.0 13.2 26.6 2.4
d. Managers and
supervisors at my level 5.1 13.2 19.7 19.5 16.0 24.1 2.4
e. Employees who report
to me 2.5 8.3 13.8 18.1 30.7 24.0 2.7
f. Office of Management
and Budget 14.1 16.7 11.4 4.0 1.8 49.0 2.9
g. Congressional
committees 7.8 14.5 13.0 8.2 3.5 50.3 2.6
h. The audit community
(e.g., GAO, Inspectors
General) 11.7 16.9 11.5 7.3 1.2 48.2 3.2
i. The general public 1.5 2.5 6.1 17.5 26.4 42.2 3.9
Q21. To what extent, if at all, do you believe that efforts to implement
GPRA to date have improved the program(s)/operation(s)/project(s) in which
you are involved?
I have not
been
sufficiently To a
involved in
GPRA to have To a very To a moderate To a
an great small
opinion. great extent extent extent To no No answer Number of
extent extent
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
48.7 2.9 5.2 14.7 16.2 10.0 2.4 503
Q22. To what extent, if at all, do you believe that efforts to implement
GPRA to date have improved your agency's programs/operations/projects?
I have not
been
sufficiently To a
involved in
GPRA to have To a very To a moderate To a
an great small
opinion. great extent extent extent To no No answer Number of
extent extent
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
47.9 2.3 6.2 17.0 16.3 8.7 1.5 503
Appendix VI
GAO Federal Managers' Survey Data
Q23. To what extent, if at all, do you believe implementing GPRA can improve
your agency's programs/operations/projects in the future ?
To a very To a To a To a No basis
great moderate small
great extent extent extent To no to judge No answer Number of
extent extent
(percent) (percent) (percent) (percent) (percent) (percent) (percent) respondents
3.4 11.9 23.7 16.7 5.6 35.7 3.0 503
Q24. If you have been involved to any extent in implementing GPRA for the
program(s)/operation(s)/project(s) you are involved with, what has been
your greatest difficulty, and in what ways, if any, do you think this
difficulty could be addressed?
Writing comment Number of (percent) respondents
26.0 503
Q25. If you have additional comments regarding any previous question or
any comments/suggestions concerning GPRA, please use the space provided
below.
Writing comment Number of (percent) respondents
16.5 503
Note: Percents reported are weighted percents based on the population
size. Unweighted N reported for each item.
Appendix VII
Agencies Subject to the Chief Financial Officers Act
The Chief Financial Officers Act of 1990 (the CFO Act) created the
position of Chief Financial Officer in each executive department and in
each major executive agency in the federal government. The agencies
covered by the CFO Act are:
1. Agency for International Development
2. Department of Agriculture
3. Department of Commerce
4. Department of Defense
5. Department of Education
6. Department of Energy
7. Department of Health and Human Services
8. Department of Housing and Urban Development
9. Department of the Interior
10. Department of Justice
11. Department of Labor
12. Department of State
13. Department of Transportation
14. Department of the Treasury
15. Department of Veterans Affairs
16. Environmental Protection Agency
17. Federal Emergency Management Agency1
1The Federal Emergency Management Agency became part of the Department of
Homeland Security in March 2003.
Appendix VII
Agencies Subject to the Chief Financial
Officers Act
18. General Services Administration
19. National Aeronautics and Space Administration
20. National Science Foundation
21. Nuclear Regulatory Commission
22. Office of Personnel Management
23. Small Business Administration
24. Social Security Administration2
2Formerly part of the Department of Health and Human Services, the Social
Security Administration once again became an independent agency on March
31, 1995. Congress established the position of Chief Financial Officer
within SSA in the Social Security Independence and Program Improvements
Act of 1994.
Appendix VIII
Comments from the Office of Management and Budget
Appendix VIII
Comments from the Office of Management
and Budget
Appendix IX
Comments from the Department of Energy
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Now on pp. 53-54.
See comment 1.
See comment 2.
See comment 3.
See comment 1. Now on p. 56.
Appendix IX
Comments from the Department of Energy
See comment 4.
See comment 5.
Appendix IX
Comments from the Department of Energy
The following are our comments on DOE's letter dated January 15, 2004.
GAO Comments The Director of the Office of Program Assessment and
Evaluation forwarded written comments from DOE on a draft of this report.
DOE disagreed with several of our conclusions concerning its 2004 Annual
Performance Plan and 2002 Performance and Accountability Report. We
incorporated the additional information and perspectives of DOE into our
report as appropriate.
1.
2.
We stated that, when compared to the 1999 Annual Performance Plan, DOE's
2004 Annual Performance Plan continued to provide a limited picture of
intended performance. DOE disagreed, stating that the plan provided a
clear picture of intended performance because a description of each
program was contained in each of the general goals. We agree that a
description of programs is provided for each goal, and we also stated in
our draft report that improvement was made in developing results-oriented
performance measures that pertained specifically to fiscal year 2004.
However, in our view, describing each program does not sufficiently
explain DOE's expectations for intended performance. More specifically, we
found the overall picture of intended performance was limited because DOE
did not specifically describe how it coordinates with other agencies to
accomplish crosscutting programs and did not provide a clear link between
its annual goals and its mission and strategic goals. In our draft report
we acknowledged that a link did not exist between the performance plan and
the strategic plan because the strategic plan was revised after the
performance plan was finalized. Nevertheless, DOE did not revise the final
2004 performance plan to reflect its changes in strategic goals. The lack
of alignment between the performance plan and strategic plan goals limits
the usefulness of the performance plan to support managers and staff in
their day-to-day activities in achieving DOE's long-term strategic goals.
In response to our observation that DOE provided a general discussion of
the strategies and resources needed to achieve its performance goals, DOE
stated that its 2004 annual performance plan provided specific strategies
and resources that will be used to achieve performance goals. DOE also
noted that funding was included at the general goal level. We agree that
funding was included at the general strategic goal level. However, better
plans relate resources to the achievement of performance goals. DOE did
not provide resource information at the performance goal level.
Furthermore, while DOE
Appendix IX
Comments from the Department of Energy
discussed external factors that could affect its ability to achieve its
performance goals at a high level, it did not discuss any specific
strategies to mitigate those factors.
3. DOE disagreed with our characterization that its 2004 annual
performance plan provided limited confidence that performance data will be
credible. The department stated that the introduction section of its plan
contained specific sections on assessing results and validating and
verifying data, as well as discussed a software package used to document
and track performance. In our draft report, we stated that DOE's plan
showed some improvement over its 1999 plan by describing credible
procedures to verify and validate performance information and by
mentioning specific program evaluations for each goal. We also noted that
DOE acquired new commercial software for performance tracking through
remote data entry, monitoring, and oversight by program offices and
managers. However, we concluded that DOE's reporting of credible
performance data was limited because its plan does not specifically
identify data limitations overall or for each of its goals. As we stated
in our report, we found this to be of particular concern because, as we
mentioned in our 2003 performance and accountability series, DOE has
several management challenges where data quality is a concern.
4. Concerning its 2002 Annual Performance and Accountability Report, DOE
stated that it provided a plan of action for addressing the causes of
targets that were not met. We agree, and in our draft report we state that
all targets that were not met or had mixed results contained a plan of
action to achieve the target in the future. We also found that the
majority of DOE targets that were not met or had mixed results contained
clear explanations. We revised our text in the final version of this
report to make these findings more evident.
5. Finally, DOE disagreed with our finding that it did not provide a
discussion of the relationship between the strategic plan, performance
plan, and the performance report in its 2002 Performance and
Accountability Report. DOE stated that the introductory section of the
report contains a paragraph that discusses the linkages between these
three reports. However, although the performance and accountability report
links these documents by organizing its results section according to
strategic goals, associated program performance goals and targets, it did
not succinctly demonstrate how the results relate to
Appendix IX
Comments from the Department of Energy
the annual and long-term strategic goals. We modified the draft
accordingly to clarify this point.
Appendix X
Comments from the Department of Housing and Urban Development
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix X
Comments from the Department of Housing
and Urban Development
See comment 1.
Appendix X
Comments from the Department of Housing
and Urban Development
See comment 2.
Appendix X
Comments from the Department of Housing
and Urban Development
See comment 3.
Appendix X
Comments from the Department of Housing
and Urban Development
See comment 4.
See comment 5.
Appendix X
Comments from the Department of Housing
and Urban Development
See comment 6.
Appendix X
Comments from the Department of Housing
and Urban Development
The following are our comments on HUD's letter dated January 9, 2004.
GAO Comments HUD provided written comments and disagreed with several of
our observations, which we address below. HUD also mentioned that all of
the areas we suggested for further improvement were already in process or
being considered. Where appropriate, we incorporated HUD's comments and
perspectives to clarify our report.
1. HUD did not agree with our observation that the link between longterm
and intermediate goals in its strategic plan is difficult to discern. The
department mentioned that the need for a direct link between longterm and
intermediate goals is not apparent, as they are aligned with strategic
goals. GPRA requires that an agency's strategic plan contain, among other
things, a description of the relationship between the longterm goals and
objectives and the annual performance goals. In addition, OMB's June 2002
Circular A-11 states that the strategic plan should briefly outline how
annual performance goals relate to the longterm, general goals, and how
they help determine the achievement of the general goals. Federal agencies
can help readers understand how they move from general goals to specific,
measurable outcomes by discussing how they plan to measure progress in
achieving the long term-goals in their strategic plan. For example, for
its strategic goal of "Increase Homeownership Opportunities," HUD mentions
that one of its long-term performance measures is to combat predatory
lending. Readers can review the intermediate measures listed under that
goal to get a sense of how HUD plans to accomplish this objective. For
example, HUD mentions that beginning in the third quarter of fiscal year
2003, field offices will report all activities related to predatory
lending to headquarters each quarter. However, not all long-term measures
listed in the strategic plan have a corresponding intermediate performance
measure.
Appendix X
Comments from the Department of Housing
and Urban Development
2. HUD disagreed with our observation that it did not explain in its
strategic plan how it used the results of program evaluations to update
the current plan and did not include a schedule for future evaluations. As
we have previously reported, program evaluations are individual,
systematic studies that use objective measurement and analysis to answer
specific questions about how well a program is working and, thus, may take
many forms. Where a program aims to produce changes that result from
program activities, outcome or effectiveness evaluations assess the extent
to which those results were achieved. Where complex systems or events
outside a program's control also influence its outcomes, impact
evaluations use scientific research methods to establish the causal
connection between outcomes and program activities and isolate the
program's contribution to those changes. A program evaluation that also
systematically examines how a program was implemented can provide
important information about why a program did or did not succeed and
suggest ways to improve it.1 In its strategic plan, HUD provides a few
examples of how it modified performance measures as a result of program
evaluations. However, we found that 38 of the 41 performance measures
discussed in the strategic plan did not mention how, if at all, HUD
revised and/or updated them as the result of program evaluations.
Elsewhere in the plan, HUD discussed program evaluation activities carried
out by its Office of Policy Development and Research; however, a
significant number of those evaluations will take place in the future and
there is no fixed timetable for when HUD will issue reports on its
findings.
3. HUD questioned an example we used to show that its strategic plan did
not always provide a clear picture of how it will be able to measure
progress toward its strategic goals. We chose this example because HUD
used the number of jobs created or retained to measure its progress in
achieving the results of the Community Development Block Grant (CDBG)
program. As HUD discusses in it strategic plan, there are factors external
to the CDBG program, such as broad macroeconomic trends and HUD's limited
control over how grant recipients use the funding, which can significantly
affect job creation in a community. Therefore, it is difficult to
establish the contribution of the CDBG program-apart from the other
factors-to HUD's stated goal.
1GAO/GGD-00-204.
Appendix X
Comments from the Department of Housing
and Urban Development
4. HUD also disagreed with our observation that in its annual performance
report it did not state the steps it would take to address unmet
performance goals. We recognize that in some instances HUD mentioned how
it would address unmet goals. GPRA requires that agencies explain and
describe, where a performance goal has not been met, why the goal was not
met, schedules for achieving the established performance goal, and whether
or not the performance goal is impractical or unfeasible. However, our
review of HUD's performance report found that of the 93 unmet performance
targets for fiscal year 2002, 74 lacked an explanation of how HUD would
address them in fiscal year 2003.
5. In commenting on our observation that HUD did not include an evaluation
of its fiscal year 2003 performance plan relative to the performance
attained by the department in fiscal year 2002, HUD mentioned that we
should consider the impact of the acceleration of reporting deadlines on
the department's ability to include an evaluation of the new fiscal year's
performance plan relative to the performance attained in the just
completed fiscal year and reasonable alternative actions to fulfill this
requirement. While we acknowledge that changes in the reporting deadlines
can create challenges for federal agencies, these deadlines are
governmentwide and not specific to HUD. In our review of agency plans we
found that some agencies, such as DOT, were able to collect performance
information for 95 percent of their performance indicators and were able
to predict future performance, despite not having complete performance
information and facing the same deadlines. DOT provided an evaluation of
whether or not fiscal year 2003 performance targets would be met for each
of its 40 performance goals based on fiscal year 2002 results. These
evaluations were included for the two performance goals for which data
were unavailable. For example, for the measure "Number of employment sites
(in the thousands) that are made accessible by Job Access and Reverse
Commute (JARC) transportation services," DOT could not characterize
performance since data had not yet been received from JARC grantees. The
2002 performance report stated that a new easier to use reporting system
is being implemented that should improve data gathering performance. The
report further stated that DOT would meet this target in fiscal year 2003.
6. HUD also disagreed with how we presented the performance information in
its summary report cards (see fig. 22). HUD noted that many of the results
were explained in the individual indicator write-ups
Appendix X
Comments from the Department of Housing
and Urban Development
that followed the summary information. Our review of HUD's reports
included, among other things, qualitative aspects of how the information
was presented, such as its usefulness to inform the average reader with
little to no exposure on the subject matter, and the extent to which it
presented summarized performance information that was complete and
user-friendly. Our analysis of HUD's performance information was largely
based on a review of the information and terms used in the performance
report cards. We characterized some of HUD's performance indicators as
being "undetermined," given that HUD did not clearly indicate whether or
not a goal was achieved. Instead, HUD provided footnotes, such as "results
too complex to summarize." We also characterized some performance targets
as having "no explanation," given that information was missing from the
report card to determine whether HUD had reached its desired target. To
develop the graphic summarizing HUD's performance information, we compiled
the results of HUD's performance indicators across all five report cards
contained in the report.
Appendix XI
Comments from the Social Security Administration
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix XI
Comments from the Social Security
Administration
See comment 1.
Appendix XI
Comments from the Social Security
Administration
See comment 2.
See comment 3.
See comment 4.
Appendix XI
Comments from the Social Security
Administration
See comment 5.
Appendix XI
Comments from the Social Security
Administration
Now on pp. 65 and 215.
Appendix XI
Comments from the Social Security
Administration
The following are our comments on SSA's letter dated January 16, 2004.
GAO Comments In general, SSA agreed with our conclusions. SSA also agreed
to incorporate the suggestions for improvement in its future planning
efforts. SSA made several points of clarification and disagreed with our
assessment in one area.
1.
2.
3.
In our draft report, we noted that SSA did not explicitly link external
factors that may affect its programs to its general goals and state how
these factors could affect goal attainment. SSA attests that its four
categories of environmental factors are discussed under each of the
strategic goals, as appropriate, and the relationship between these
factors and SSA's strategic priorities is described. This general
discussion of the environmental factors is useful in understanding the
challenges SSA faces in working toward its broad strategic goals. However,
SSA provides little or no discussion of these challenges in its discussion
of the agency's performance goals. Thus, the range of challenges facing
the agency in meeting each of its performance goals is not fully
explained.
In our draft report, we noted that SSA does not provide timetables or
schedules for achieving all the results in its strategic plan. SSA noted
that it expects to achieve its long-term outcomes within the 5-year period
covered by the strategic plan; in selected instances, shorter time frames
are specified. SSA noted that more detailed plans and timetables are
featured in its annual performance plan. GPRA requires agencies to furnish
a schedule of significant actions in their strategic plans; however, SSA
does not clearly articulate its timetables and schedules for achieving
each of its long-term outcomes in its strategic plan.
We noted that SSA's strategic plan could be improved by providing details
on how each performance and accountability challenge will be addressed.
SSA asserted that the strategic plan addresses some of the challenges, but
because the challenges are updated every year, they are more appropriately
addressed in the annual performance plan and performance and
accountability report. As noted in our discussion of the criteria used to
analyze agencies' strategic plans, it is particularly important that
agencies develop strategies that address management challenges that
threaten their ability to meet long-term strategic goals,
Appendix XI
Comments from the Social Security
Administration
as one of the purposes of GPRA is to improve the management of federal
agencies.
4. In our draft report, we observed that SSA's discussion of its
interactions with other agencies, especially those that serve the same
beneficiaries, was limited. SSA noted that such a discussion would be
useful, but is not an OMB requirement. While we agree that this is not an
OMB requirement, we have reported that given scarce resources and
competing priorities, it would be useful to identify agency efforts to
maximize its effect through cooperation and coordination across the
federal government. Better strategic plans not only identify the need to
coordinate with other agencies, but also discuss how agencies intend to
coordinate common or complementary goals and strategies with other
agencies.
5. With regard to SSA's performance and accountability report, we noted
that SSA did not clearly state how program evaluations were used to answer
questions about program performance and results and how those results can
be improved. SSA disagreed with our observation, stating that many of its
evaluations rely on surveys, and these surveys form the basis for its
efforts to deliver high-quality service. SSA also noted that it listed
other evaluations that are of great importance to its ongoing operations.
We do not discount the usefulness of SSA's surveys in assessing its
day-to-day management of programs. Rather, as we noted in the report, it
would be helpful for SSA to clearly identify the range of evaluations
conducted and how each of them contributed to improved program
performance. For example, we recently recommended that SSA evaluate a new
initiative to improve the integrity of Social Security number issuance to
noncitizens; the description of such an evaluation would be helpful for
SSA to determine how it can be best positioned to ensure the integrity of
its enumeration process.1 Additionally, our September 2000 report on
program evaluation states that GPRA recognizes the complementary nature of
program evaluation and performance measurement. Strategic plans are to
describe the program evaluations that were used in establishing and
revising goals and to include a schedule for future
1U.S. General Accounting Office, Social Security Administration: Actions
Taken to Strengthen Procedures for Issuing Social Security Numbers to
Noncitizens, but Some Weaknesses Remain, GAO-04-12 (Washington, D.C.: Oct.
15, 2003).
Appendix XI
Comments from the Social Security
Administration
program evaluations. Agencies are to summarize the findings of program
evaluations in their annual performance reports.
Additionally, SSA made technical comments that we incorporated into the
report, as appropriate.
Appendix XII
GAO Contact and Staff Acknowledgments
GAO Contact Patricia Dalton, (202) 512-6806
Acknowledgments In addition to the persons mentioned above, Thomas Beall,
Daniel Bertoni, Kay Brown, Joyce Corry, Elizabeth Curda, David Dornisch,
William Fenzel, Kimberly Gianopoulos, Evan Gilman, Katie Harris, Susan
Higgins, Benjamin Licht, William McKelligott, James Noel, Carol Petersen,
Carolyn Samuels, Teresa Spisak, Daren Sweeney, Carolyn Taylor, Michael
Volpe, Lynn Wasielewski, and Steven Westley made key contributions to this
report.
Related GAO Products
GPRA/Managing for Results
Results-Oriented Government: Using GPRA to Address 21st Century
Challenges, GAO-03-1166T (Washington, D.C.: Sept. 18, 2003).
Results-Oriented Management: Agency Crosscutting Actions and Plans in
Border Control, Flood Mitigation and Insurance, Wetlands, and Wildland
Fire Management, GAO-03-321 (Washington, D.C.: Dec. 20, 2002).
Results-Oriented Management: Agency Crosscutting Actions and Plans in Drug
Control, Family Poverty, Financial Institution Regulation, and Public
Health Systems, GAO-03-320 (Washington, D.C.: Dec. 20, 2002).
Performance and Accountability: Reported Agency Actions and Plans to
Address 2001 Management Challenges and Program Risks, GAO-03-225
(Washington, D.C.: Oct. 31, 2002).
Managing for Results: Next Steps to Improve the Federal Government's
Management and Performance, GAO-02-439T (Washington, D.C.: Feb. 15, 2002).
Managing for Results: Federal Managers' Views on Key Management Issues
Vary Widely Across Agencies, GAO-01-592 (Washington, D.C.: May 25, 2001).
Managing for Results: Federal Managers' Views Show Need for Ensuring Top
Leadership Skills, GAO-01-127 (Washington, D.C.: Oct. 20, 2000).
Managing for Results: Barriers to Interagency Coordination, GAO/GGD-00-106
(Washington, D.C.: Mar. 29, 2000).
Management Reform: Elements of Successful Improvement Initiatives,
GAO/T-GGD-00-26 (Washington, D.C.: Oct. 15, 1999).
Management Reform: Using the Results Act and Quality Management to Improve
Federal Performance, GAO/T-GGD-99-151 (Washington, D.C.: July 29, 1999).
Managing for Results: Opportunities for Continued Improvements in
Agencies' Performance Plans, GAO/GGD/AIMD-99-215 (Washington, D.C.: July
20, 1999).
Related GAO Products
Agency Performance Plans: Examples of Practices That Can Improve
Usefulness to Decisionmakers, GAO/GGD/AIMD-99-69 (Washington, D.C.: Feb.
26, 1999).
The Results Act: Assessment of the Governmentwide Performance Plan for
Fiscal Year 1999, GAO/AIMD/GGD-98-159 (Washington, D.C.: Sept. 8, 1998).
Managing for Results: An Agenda to Improve the Usefulness of Agencies'
Annual Performance Plans, GAO/GGD/AIMD-98-228 (Washington, D.C.: Sept. 8,
1998).
The Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans. GAO/GGD-10.1.20 (Washington, D.C.: April 1998).
Agencies' Annual Performance Plans Under the Results Act: An Assessment
Guide to Facilitate Congressional Decisionmaking, GAO/GGD/AIMD-10.1.18
(Washington, D.C.: February 1998).
Managing for Results: Agencies' Annual Performance Plans Can Help Address
Strategic Planning Challenges, GAO/GGD-98-44 (Washington, D.C.: Jan. 30,
1998).
Managing for Results: Building on Agencies' Strategic Plans to Improve
Federal Management, GAO/T-GGD/AIMD-98-29 (Washington, D.C.: Oct. 30,
1997).
Managing for Results: Critical Issues for Improving Federal Agencies'
Strategic Plans, GAO/GGD-97-180 (Washington, D.C.: Sept. 16, 1997).
Managing for Results: Using the Results Act to Address Mission
Fragmentation and Program Overlap, GAO/AIMD-97-146 (Washington, D.C.: Aug.
29, 1997).
Managing for Results: The Statutory Framework for Improving Federal
Management and Effectiveness, GAO/T-GGD/AIMD-97-144 (Washington, D.C.:
June 24, 1997).
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven, GAO/GGD-97-109 (Washington, D.C.: June 2,
1997).
Related GAO Products
Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate
Congressional Review, GAO/GGD-10.1.16 (Washington, D.C.: May 1997).
GPRA: Managerial Accountability and Flexibility Pilot Did Not Work as
Intended, GAO/GGD-97-36 (Washington, D.C.: Apr. 10, 1997).
Managing for Results: Enhancing the Usefulness of GPRA Consultations
Between the Executive Branch and Congress, GAO/T-GGD-97-56 (Washington,
D.C.: Mar. 10, 1997).
Managing for Results: Key Steps and Challenges in Implementing GPRA in
Science Agencies, GAO/T-GGD/RCED-96-214 (Washington, D.C.: July 10, 1996).
Executive Guide: Effectively Implementing the Government Performance and
Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996).
Strategic Human Capital Management
Human Capital: A Guide for Assessing Strategic Training and Development
Efforts in the Federal Government (Exposure Draft), GAO-03-893G
(Washington, D.C.: July 2003).
Results-Oriented Cultures: Creating a Clear Linkage between Individual
Performance and Organizational Success, GAO-03-488 (Washington, D.C.: Mar.
14, 2003).
Results-Oriented Cultures: Insights for U.S. Agencies from Other
Countries' Performance Management Initiatives, GAO-02-862 (Washington,
D.C.: Aug. 2, 2002).
Human Capital: Key Principles From Nine Private Sector Organizations,
GAO/GGD-00-28 (Washington, D.C.: Jan. 31, 2000).
Linking Resources to Performance Budgeting: Observations on the Use of
OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget,
GAO-04-174
Results (Washington, D.C.: Jan. 30, 2004).
Managing for Results: Efforts to Strengthen the Link Between Resources and
Results at the Nuclear Regulatory Commission, GAO-03-258 (Washington,
D.C.: Dec. 10, 2002).
Related GAO Products
Managing for Results: Efforts to Strengthen the Link Between Resources and
Results at the Administration for Children and Families, GAO-03-9
(Washington, D.C.: Dec. 10, 2002).
Managing for Results: Efforts to Strengthen the Link Between Resources and
Results at the Veterans Health Administration, GAO-03-10 (Washington,
D.C.: Dec. 10, 2002).
Managing for Results: Agency Progress in Linking Performance Plans With
Budgets and Financial Statements, GAO-02-236 (Washington, D.C.: Jan. 4,
2002).
Performance Budgeting: Fiscal Year 2000 Progress in Linking Plans With
Budgets, GAO/AIMD-99-239R (Washington, D.C.: July 30, 1999).
Performance Budgeting: Initial Experiences Under the Results Act in
Linking Plans With Budgets, GAO/AIMD/GGD-99-67 (Washington, D.C.: Apr. 12,
1999).
Performance Budgeting: Past Initiatives Offer Insights for GPRA
Implementation, GAO/AIMD-97-46 (Washington, D.C.: Mar. 27, 1997).
Measuring Performance
Program Evaluation: An Evaluation Culture and Collaborative Partnerships
Help Build Agency Capacity, GAO-03-454 (Washington, D.C.: May 2, 2003).
Managing for Results: Next Steps to Improve the Federal Government's
Management and Performance, GAO-02-439T (Washington, D.C.: Feb. 15, 2002).
Program Evaluation: Studies Helped Agencies Measure or Explain Program
Performance, GAO/GGD-00-204 (Washington, D.C. Sept. 29, 2000).
Managing for Results: Strengthening Regulatory Agencies' Performance
Management Practices, GAO/GGD-00-10 (Washington, D.C.: Oct. 28, 1999).
Managing for Results: Measuring Program Results That Are Under Limited
Federal Control, GAO/GGD-99-16 (Washington, D.C.: Dec. 11, 1998).
Related GAO Products
Grant Programs: Design Features Shape Flexibility, Accountability, and
Performance Information, GAO/GGD-98-137 (Washington, D.C.: June 22, 1998).
Program Evaluation: Agencies Challenged by New Demand for Information on
Program Results, GAO/GGD-98-53 (Washington, D.C.: Apr. 24, 1998).
Managing for Results: Regulatory Agencies Identified Significant Barriers
to Focusing on Results, GAO/GGD-97-83 (Washington, D.C.: June 24, 1997).
Managing for Results: Analytic Challenges in Measuring Performance,
GAO/HEHS/GGD-97-138 (Washington, D.C.: May 30, 1997).
Data Credibility Performance Reporting: Few Agencies Reported on the
Completeness and Reliability of Performance Data, GAO-02-372 (Washington,
D.C.: Apr. 26, 2002).
Managing for Results: Assessing the Quality of Program Performance Data,
GAO/GGD-00-140R (Washington, D.C.: May 25, 2000).
Managing for Results: Challenges Agencies Face in Producing Credible
Performance Information, GAO/GGD-00-52 (Washington, D.C.: Feb. 4, 2000).
Performance Plans: Selected Approaches for Verification and Validation of
Agency Performance Information, GAO/GGD-99-139 (Washington, D.C.: July 30,
1999).
Using Performance Managing for Results: Views on Ensuring the Usefulness
of Agency Performance Information to Congress, GAO/GGD-00-35 (Washington,
Information D.C.: Jan. 26, 2000).
Managing for Results: Using GPRA to Assist Congressional and Executive
Branch Decisionmaking, GAO/T-GGD-97-43 (Washington, D.C.: Feb. 12, 1997).
Related GAO Products
Managing for Results: Achieving GPRA's Objectives Requires Strong
Congressional Role, GAO/T-GGD-96-79 (Washington, D.C.: Mar. 6, 1996).
GAO's Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Service Requested
*** End of document. ***