Program Evaluation: Agencies Challenged by New Demand for Information on
Program Results (Letter Report, 04/24/98, GAO/GGD-98-53).
GAO reviewed federal agencies' efforts to provide information on federal
program results, focusing on: (1) the current resources and roles for
program evaluation in federal agencies; (2) the anticipated effects of
governmentwide reforms and other initiatives on evaluation of federal
programs; and (3) potential strategies for agencies to respond to the
anticipated effects and provide information on program results.
GAO noted that: (1) existing federal evaluation resources--at least as
currently configured and deployed--are likely to be challenged to meet
increasing demands for program results information; (2) agencies
reported devoting variable but relatively small amounts of resources to
evaluating program results; (3) morever, agencies reported that the
primary role of program evaluation was internally focused on program
improvement, rather than direct congressional or other external
oversight; (4) interest in the program by high-level officials was most
often cited as a criterion for initiating evaluation work; a small
portion of studies were said to be conducted for a congressional
committee or in response to a legislative mandate; (5) some of the
evaluation officials and experts that GAO interviewed anticipated not
only increased interest in learning the results of federal programs and
policies but also additional complications in obtaining that
information; (6) some evaluation officials from states with performance
measurement experience noted that effectiveness evaluations would
continue to be needed to assess policy impact and address problems of
special interest or larger policy issues, such as the need for any
government intervention at all in an area; (7) to meet the anticipated
increase in demand for program results information as well as the
associated technical challenges, some evaluation officials GAO
interviewed described efforts to leverage both federal and nonfederal
resources; (8) however, some agencies anticipated that major investments
in their data systems would be required to produce reliable data on
program outcomes; and, in a prior study, program officials were
concerned that reliance on less rigorous methods would not provide an
accurate picture of program effectiveness; (9) moreover, while some
federal evaluation officials envisioned providing increased technical
assistance to state and local evaluators, a few state evaluation
officials suggested an alternative strategy for the federal government;
(10) GAO drew several conclusions from its comparison of current federal
evaluation resources with the anticipated challenges to meeting
increased demand for information on program results; (11) federal
evaluation resources have important roles to play in responding to
increased demand for information on program results, but--at least as
currently configured and deployed--they are likely to be challenged to
meet that demand; and (12) in the future, carefully targeting federal
agencies' evaluation resources show promise for addressing key questions
about program results.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GGD-98-53
TITLE: Program Evaluation: Agencies Challenged by New Demand for
Information on Program Results
DATE: 04/24/98
SUBJECT: Strategic planning
Program evaluation
Reporting requirements
Information systems
Federal/state relations
Evaluation methods
Congressional oversight
IDENTIFIER: GPRA
Government Performance and Results Act
National Performance Review
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Committee on Governmental Affairs, U.S. Senate
April 1998
PROGRAM EVALUATION - AGENCIES
CHALLENGED BY NEW DEMAND FOR
INFORMATION ON PROGRAM RESULTS
GAO/GGD-98-53
Program Evaluation
(966704/973810)
Abbreviations
=============================================================== ABBREV
DOD - Department of Defense
FTE - full-time equivalent
FY - fiscal year
NPR - National Performance Review
OMB - Office of Management and Budget
Letter
=============================================================== LETTER
B-277820
April 24, 1998
The Honorable Fred D. Thompson
Chairman
The Honorable John Glenn
Ranking Minority Member
Committee on Governmental Affairs
United States Senate
Congressional and agency decisionmakers need evaluative information
about whether federal programs are working well or poorly, both to
manage programs effectively and to help decide how to allocate
limited federal resources. Increased interest in learning the
results of federal programs and activities is reflected in government
reforms, such as the Government Performance and Results Act of 1993
(GPRA or the Results Act), which institutes a governmentwide
requirement for agencies to, among other things, report on their
results in achieving their agency and program goals. However, other
recent reforms, such as reducing the size and authority of the
federal government while maintaining a level of services, has the
potential to hinder agencies' ability to obtain this information.
Because data on program results are typically more difficult and
resource intensive to obtain than data on program activities, limited
budget dollars mean that investing in obtaining information on
results may compete with spending on program activities. The proper
balance between the two spending priorities is essential, since
information on program results can contribute to deciding how to
allocate resources to activities to maximize program benefits.
Federal agencies are the primary source of evaluation information
about their programs. In past surveys of federal agencies, we found
limited (and diminishing) resources spent on formal studies of
program results, that is, program evaluation.\1 Because evaluation
can be vitally important in improving program results, we asked how
in a context of limited federal resources and responsibility, can
agencies support additional requests for program results information?
This report, which we prepared under our basic legislative
responsibilities, responds to that question by discussing the current
status of and future needs for program evaluation in federal
agencies. Because of your interest in improving the quality of
information on federal programs, we are addressing this report to
you. Our objectives were to identify (1) the current resources and
roles for program evaluation in federal agencies, (2) the anticipated
effects of governmentwide reforms and other initiatives on evaluation
of federal programs, and (3) potential strategies for agencies to
respond to the anticipated effects and provide information on program
results.
--------------------
\1 Program Evaluation Issues (GAO/OCG-93-6TR, Dec. 1992) and Federal
Evaluation: Fewer Units, Reduced Resources, Different Studies from
1980 (GAO/PEMD-87-9, Jan. 23, 1987).
BACKGROUND
------------------------------------------------------------ Letter :1
Program and policy decisions require a wide array of information that
answers various questions. For example, descriptive information
tells how a program operates--what activities are performed, who
performs them, and who is reached. In contrast, evaluative
information speaks to how well a program is working--such as whether
activities are managed efficiently and effectively, whether they are
carried out as intended, and to what extent the program is achieving
its intended objectives or results. There are a variety of methods
for obtaining information on program results, such as performance
measurement and program evaluation, which reflect differences in how
readily one can observe program results.
Performance measurement, as defined by the Results Act, is the
ongoing monitoring and reporting of program accomplishments,
particularly progress towards preestablished goals. It tends to
focus on regularly collected data on the type and level of program
activities (process), the direct products and services delivered by
the program (outputs), and the results of those activities
(outcomes). While performance may be defined more broadly as program
process, inputs, outputs, or outcomes, results usually refer only to
the outcomes of program activities. For programs that have readily
observable results, performance measurement may provide sufficient
information to demonstrate program results.
In some programs, however, results are not so readily defined nor
measured.\2 In such cases, program evaluations may be needed, in
addition to performance measurement, to examine the extent to which a
program is achieving its objectives. Program evaluations are
systematic studies conducted periodically to assess how well a
program is working. While they may vary in their focus, these
evaluations typically examine a broader range of information on
program performance and its context than is feasible in ongoing
performance measurement. Where programs aim to produce changes, as a
result of program activities, outcome (or effectiveness) evaluations
assess the extent to which those outcomes or results were achieved,
such as whether students increased their understanding of or skill in
the material of instruction. In cases where the program's outcomes
are influenced by complex systems or events outside the program's
control, impact evaluations use scientific research methods to
establish the causal connection between outcomes and program
activities, estimate what would have happened in the absence of the
program, and thus isolate the program's contribution to those
changes. For example, although outcome measures might show a decline
in a welfare program's caseload after the introduction of job
placement activities, a systematic impact evaluation would be needed
to assess how much of the observed change was due to an improved
economy rather than the new program.
In addition, a program evaluation that also systematically examines
how a program was implemented can provide important information about
why a program did or did not succeed and suggest ways to improve it.
For the purposes of this report, we used the definition of program
evaluation that is used in the Results Act, "an assessment, through
objective measurement and systematic analysis, of the manner and
extent to which federal programs achieve intended objectives."\3 We
asked about assessments of program results, which could include both
the analysis of outcome-oriented program performance measures as well
as specially conducted outcome or impact evaluations.
Two government initiatives could influence the demand for and the
availability and use of program evaluation information. The Results
Act seeks to promote a focus on program results, by requiring
agencies to set program and agency performance goals and to report
annually on their progress in achieving them (beginning with fiscal
year 1999). In addition to encouraging the development of
information on program results for activities across the government,
the Results Act recognizes the complementary nature of program
evaluation and performance measurement. It requires agencies to
include a schedule for future program evaluations in their strategic
plans, the first of which was to be submitted to Congress by
September 30, 1997. The Results Act also requires agencies to review
their success in achieving their annual performance goals (which are
set forth in their annual performance plans) and to summarize the
findings of program evaluations in their annual program performance
reports (the first of which is due by March 31, 2000). The National
Performance Review (NPR) led by the Vice President's office has asked
agencies to reexamine their policies, programs, and operations to
find and implement ways to improve performance and service to their
customers. Both of these initiatives--because of their focus on
program results--could be expected to increase the demand for and the
availability and use of program evaluation information.
Other recent governmentwide initiatives could have potentially
conflicting effects. In several program areas, devolution of program
responsibility from the federal level and consolidation of individual
federal programs into more comprehensive, multipurpose grant programs
has shifted both program management and accountability
responsibilities toward the states. These initiatives may thus make
it more difficult for federal agencies to evaluate the results of
those programs. In addition, efforts to reduce the growth of the
federal budget have resulted in reductions in both federal staff and
program resources in many agencies. The combination of these
initiatives raises a question: In an environment of limited federal
resources and responsibility, how can agencies meet the additional
needs for program results information?
To identify the roles and resources available for federal program
evaluation, in 1996, we conducted a mail survey of offices identified
by federal agency officials that were conducting studies of program
results or effectiveness in 13 cabinet-level departments and 10
independent executive agencies. Detailed information on program
evaluation studies refers to those conducted during fiscal year 1995
(regardless of when they began or ended). To identify how recent
reforms were expected to affect federal evaluation activities and
what strategies were available for responding to those changes, we
interviewed external evaluation experts and evaluation and other
officials at selected federal and state agencies. In this report, we
use the term "agency" to include both cabinet-level departments and
independent agencies.
--------------------
\2 For example, the intended results of a statistical program--valid,
reliable, and useful data--are typically measured through expert
judgments. See Martin & Straf, 1992.
\3 Although the term "program evaluation" is sometimes used to refer
to evaluation of program process (or implementation) without
examining results, this definition, which focuses on results, is also
used in the Office of Management and Budget's (OMB) requests for
budget justifications.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
Existing federal evaluation resources--at least as currently
configured and deployed--are likely to be challenged to meet the
increasing demand for program results information. Agencies reported
devoting variable but relatively small amounts of resources to
evaluating program results. Across the 13 departments and 10
independent agencies we surveyed, the resources that agencies
identified as being involved in assessing their programs' results
represented $194 million and 669 full-time equivalent staff (FTE) in
fiscal year 1995 and were unevenly distributed across the agencies.
We found 81 offices that reported conducting studies of program
results. Over half of the reporting offices were small, with 18 or
fewer FTEs.\4 Similarly, a majority (56 percent) of the 81 offices
reported conducting 5 or fewer studies in fiscal year 1995. The
evaluation studies were diverse as well--about half were handled
in-house, and almost half used existing program data.
Moreover, agencies reported that the primary role of program
evaluation was internally focused on program improvement, rather than
direct congressional or other external oversight. Interest in the
program by high-level agency officials was most often cited as a
criterion for initiating evaluation work; a small portion of studies
(17 percent) were said to be conducted for a congressional committee
or in response to a legislative mandate. The studies' primary
audiences were reported to be program managers and higher-level
agency officials. In addition, these offices reported that their
evaluation activities primarily consisted of conducting the studies
themselves or through contractors. Few offices reported frequent
efforts to extend the use of evaluation resources by providing
training in evaluation methods to others, such as federal program
staff or state and local evaluators.
Some of the evaluation officials and experts we interviewed
anticipated increased interest in learning the results of federal
programs and policies but also additional complications in obtaining
that information. For example, some said that devolving federal
program responsibility in the health care and welfare systems to the
states raises many new important questions about its effects, but as
programs become increasingly diverse, as expected, this will make
evaluating the effects of these reforms more difficult. Federal
funding reductions were said by some evaluation officials not only to
reduce the level of federal evaluation activity but also to diminish
agency technical capacity, through the loss of some of their most
experienced staff. Although federal agencies have limited experience
in meeting the Results Act's reporting requirements (the first annual
performance reports are not due until March 2000, for fiscal year
1999), some evaluation officials hoped the Results Act would increase
the use of evaluation results in decisionmaking, while others feared
that the large investments required to produce valid and reliable
outcome data across whole agencies would compete for funds currently
used for more in-depth evaluations of individual programs'
effectiveness. We previously reported that early agency efforts to
implement the Results Act have encountered numerous analytic
challenges, and agency officials found program evaluation expertise
helpful in their efforts to develop performance reporting systems.\5
In addition, some evaluation officials from states with performance
measurement experience noted that effectiveness evaluations would
continue to be needed to assess policy impact and address problems of
special interest or larger policy issues, such as the need for any
government intervention at all in an area.
To meet the anticipated increase in demand for program results
information as well the associated technical challenges, some
evaluation officials we interviewed described efforts to leverage
both federal and nonfederal resources. Suggestions included (1)
adapting existing information systems to yield data on program
results, (2) broadening the range of their work to include less
rigorous and less expensive methods, (3) devolving program evaluation
to federal (or state and local) program managers, and (4) developing
partnerships with others to integrate the varied forms of performance
information available on their programs. However, some agencies
anticipated that major investments in their data systems would be
required to produce reliable data on program outcomes; and, in a
prior study, program officials were concerned that reliance on less
rigorous methods would not provide an accurate picture of program
effectiveness. Moreover, while some federal evaluation officials
envisioned providing increased technical assistance to state and
local evaluators, a few state evaluation officials suggested an
alternative strategy for the federal government: (1) providing
evaluation leadership through establishing a catalog of tested
performance measures and (2) conducting impact evaluations to
supplement the states' performance measurement information.
We drew several conclusions from our comparison of current federal
evaluation resources with the anticipated challenges to meeting
increased demand for information on program results. First, federal
evaluation resources have important roles to play in responding to
increased demand for information on program results, but--at least as
currently configured and deployed--they are likely to be challenged
to meet that demand. Second, in the future, carefully targeting
federal agencies' evaluation resources and leveraging federal and
nonfederal resources show promise for addressing key questions about
program results. Some possible ways to target and leverage resources
include (1) assisting program managers to develop valid and reliable
performance reporting under the Results Act, and (2) planning
evaluation studies to fill the most important information gaps--such
as providing supplemental information on the reasons for observed
performance or examining policy issues that extend beyond program
borders. Third, one way to ensure that the results of diverse
evaluation activities can be synthesized to portray programs at the
national level is for federal evaluation staff to coordinate those
activities in advance.
--------------------
\4 We identified no offices conducting studies of program results in
two departments and four independent agencies. Some of these offices
noted that they assessed program outcomes by monitoring program
performance, or by analyzing efficiency or compliance, but not by
conducting evaluation studies per se. It is also possible that
offices, including those without programs per se, did assess the
effectiveness of federal activities, but they did not complete the
survey for this work.
\5 Managing for Results: Analytic Challenges in Measuring
Performance (GAO/HEHS/GGD-97-138, May 30, 1997).
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
We distributed surveys in 1996 regarding federal evaluation
activities within 13 cabinet level departments and 10 independent
executive agencies in the federal government. We excluded the
Department of Defense from our survey of evaluation offices because
of the prohibitively large number of offices it identified as
conducting assessments of effectiveness. Although we asked agency
officials to be inclusive in their initial nominations of offices
that conducted evaluations of program results, some offices that
conducted evaluations may have been overlooked and excluded from our
survey. However, many offices that were initially identified as
having conducted evaluations later reported that they had not done
so. In our survey, we asked each office about the range of its
analytic and evaluation activities and about the length, cost,
purpose, and other characteristics of the program evaluation studies
they conducted during fiscal year 1995. (See appendix I for more
details on the scope and methodology of the survey.)
Between 1996 and 1997, we conducted interviews of program evaluation
practitioners selected to represent divergent perspectives. We asked
what had been or were expected to be the effects of various
government changes and reforms on federally supported and related
program evaluation activities and strategies for responding to those
effects. We identified individuals with evaluation knowledge and
expertise from a review of the survey responses, the evaluation
literature, and our prior work; they were from an array of federal
and state agencies and the academic and consulting communities. We
then judgmentally selected 18 people to interview to reflect (1) a
mix of different program types and diverse amounts of experience with
program evaluation and (2) experience with some of the reforms at the
state or federal level. Those selected included nine evaluation
officials (six from offices in federal agencies and three from state
legislative audit agencies) and seven external evaluation experts
(four from private research organizations and three from
universities). In addition, we interviewed an OMB official and one
official from a state executive branch agency, and we also asked the
officials from the state legislative audit agencies about their
experiences with state performance reporting requirements.
We conducted our review between May 1996 and July 1997 in accordance
with generally accepted government auditing standards. However, we
did not independently verify the types of studies conducted, other
information reported by our respondents, nor information gained from
interviewees.
FEDERAL AGENCIES DEVOTE
VARIABLE BUT SMALL AMOUNTS OF
RESOURCES TO EVALUATING PROGRAM
RESULTS, PRIMARILY FOR INTERNAL
USE
------------------------------------------------------------ Letter :4
The resources allocated to conducting systematic assessments of
program results (or evaluation studies) were small and unevenly
distributed across the 23 agencies (departments and independent
agencies) we surveyed. We found 81 offices that reported expending
resources--funds and staff time--on conducting program effectiveness
studies in fiscal year 1995. Over half of those offices had 18 or
fewer full-time equivalent staff FTEs, while only a few offices had
as many as 300 to 400 FTEs. (See figure 1.)
Figure 1: Offices That
Conducted Program Evaluations
in FY 1995 (Office size in
FTEs)
(See figure in printed
edition.)
Note: No data on seven offices.
Source: GAO survey on program evaluation in federal agencies
governmentwide.
Moreover, about one-third of the offices reported spending 50 percent
or more of their time on evaluation activities (including development
of performance measures and assessments of program effectiveness,
compliance, or efficiency), since program evaluation was only one of
these offices' many responsibilities. (See survey question 9 in
appendix I.) Two of the 3 largest offices (over 300 FTEs) spent about
10 percent of their staff time on program evaluation activities.
Thus, the estimated staff and
budget resources that the 81 offices actually devoted to evaluation
activities totaled 669 FTEs and at least $194 million across the 23
agencies surveyed.\6
In addition, most (61 of 81) offices reported also conducting
management analysis activities; the most frequent activities were
conducting management studies, developing strategic plans, and
describing program implementation. Of those offices that could
estimate their staff time, about half reported spending less than 25
percent of their time on management analysis. Similarly, many
offices reported conducting policy planning and analysis, but most of
them reported spending less than 25 percent of their time on it.
Thus, a majority of the offices (45 of the 81 identified) conducted
few evaluation studies (5 or less in fiscal year 1995), while 16
offices--representing 7 agencies--accounted for two-thirds of the 928
studies conducted. (See table 1.)
Table 1
Distribution of Program Evaluations
Conducted in FY 1995 Across the Offices
Surveyed
Studies
conducted
--------------
Number
of
office Percen
Number of studies reported per office s Number t
---------------------------------------------- ------ ------ ------
1-5 45 111 12%
6-17 20 193 21
20-79 16 624 67
======================================================================
Total 81 928
----------------------------------------------------------------------
Source: GAO survey.
Finally, 6 of the 23 agencies we surveyed did not report any offices
conducting evaluation studies in fiscal year 1995. A few of these
agencies indicated that they analyzed program accomplishments or
outputs or conducted management reviews to assess their programs'
performance but did not conduct an evaluation study per se. Some of
the 6 agencies also reported conducting other forms of program
reviews that focused on assessing program compliance or efficiency
rather than program results.
Offices conducting program evaluations were located at various levels
of federal agencies. A few of the 81 offices were located in the
central policy or administrative office at the highest level of the
organization (5 percent) or with the Inspector Generals (5 percent);
many more were located in administrative offices at a major
subdivision level (43 percent) or in program offices or technical or
analytic offices supporting program offices (30 and 16 percent,
respectively). (See table 2.)
Table 2
Organizational Location of Offices
Conducting Evaluations for the Total
Sample and Offices Conducting 20 or More
Studies, in FY 1995
Number of offices
Percent of that
offices in reported
each conducting 20
Organizational location location or more studies
-------------------------------------- ---------- ------------------
Inspector General 5% 2
Central administrative office 5 2
Central administrative 43 6
office for a major
operating subdivision of
department or agency
Program office 30 4
Technical or analytical 16 2
unit supporting a
program office
Other 1 0
======================================================================
Total offices 81 16
----------------------------------------------------------------------
Source: GAO survey.
Four of the 23 agencies surveyed had offices at all 3 levels (agency,
division, and program), and over half the agencies (14 of 23) had
conducted evaluations at the program level. The 16 offices
conducting 20 or more studies were more likely to be centralized at
the agency or division level than at the program level.
A diverse array of evaluation studies were described in the surveys.
Just over half of the studies for which we have such information were
conducted in-house (51 percent), and 27 percent lasted under 6
months. But the studies that were contracted out tended to be larger
investments--almost two-thirds of them took over a year to complete,
and over half cost between $100,000 and $500,000. Moreover, almost a
third of all the studies lasted more than 2 years, reflecting some
long-term evaluations. (See table 3.)
Table 3
Reported Percentage of Duration and Cost
of Evaluation Studies Conducted in FY
1995, by Study Locus\a
Study Locus
----------------------------------------
Jointly with
another
federal,
In- Contracted state, or
house out local agency Total
---------------------------- ------ ---------- ------------ ------
Duration
Less than 6 49% 5% 2% 27%
months
6 to 12 months 29 31 <1 25
13 to 24 months 10 28 11 17
Over 2 years 12 36 86 31
======================================================================
Total number of studies 418 281 124 823
Cost\b
Less than 65% 14% 23% 39%
$100,000
Between 30 57 33 41
$100,000 and
$499,000
Between 2 13 21 9
$500,000 and
$999,000
Over $1 million 3 16 24 11
======================================================================
Total number of studies 319 263 106 688
----------------------------------------------------------------------
Note: Percents may not total 100 due to rounding.
\a Respondent information on study length and cost was incomplete.
Information was provided on study length for 89 percent of the 928
studies respondents reported were conducted in fiscal year 1995,
while information was provided on cost for only 74 percent of the 928
studies. Many studies from offices reporting no cost information at
all lasted less than 6 months.
\b Study cost was the total cost regardless of funding source or
fiscal year in which funds were obligated. For contracted-out
studies, the amount of the contract or grant itself as well as staff
costs associated with issuing and monitoring the contract or grants
were included.
Source: GAO survey.
For example, a study of the impact of a medical treatment program,
which used an experimental design with a complex set of Medicare
program and clinical data from thousands of patients on numerous
outcomes (for both patients and program costs), took over 2 years and
cost over $1 million to complete.\7
Many of the 1995 studies reportedly used relatively simple designs or
research methods, and many relied on existing program data. The two
most commonly reported study designs were judgmental assessments (18
percent) as well as experimental designs employing random assignment
(14 percent). (See table 4 for a list of designs ranging from the
most to least amount of control over the study conditions.)
Table 4
Research Methods Used for the Evaluation
Studies Conducted in FY 1995
Number
of
studie Percen
s t of
using all
method studie
Research method \a s\b
------------------------------------------------------ ------ ------
Experimental and control groups randomly assigned: 129 14%
after-intervention measures
Some experimental control: before-and after- 66 7
intervention outcome measures with some
statistical controls
Time series: many repeated measures taken before 56 6
and after intervention
Matched pairs or cross-sectional studies: outcomes 39 4
measured after intervention with statistical controls
Panel studies: several repeated measures taken 21 2
during and after intervention
Statistical modeling or simulation: program 59 6
outcomes compared to predicted (control) results
Simple before-and after-studies: outcomes 114 12
measured before and after intervention
One time survey of outcomes 96 10
Judgmental assessment: use of outside expert, 170 18
program administrator, or participant judgment of
program effects
Other\c 79 8
----------------------------------------------------------------------
\a An individual study could have been reported as using more than
one research method.
\b These percents are based on all 928 studies that respondents
reported were conducted in fiscal year 1995; however, responses to
this question were not provided on all studies. For example, 6
offices did not answer this question for any of the 117 studies they
reported having conducted. Therefore, these percents may
underestimate the actual level of use of these methods.
\c The other methods used included citation analysis, cost benefit
analysis, and regression analysis.
Source: GAO survey.
Many of the studies (over 70 of the 129) that used experimental
designs were evaluations of state demonstration programs, which were
required by law to use such methods, and were conducted out of one
office. Experimental designs and designs using statistical controls
are used to identify a program's net impact on its objectives where
external factors are also known to affect its outcome. However,
without knowing the circumstances of many of the programs being
evaluated, it is impossible for us to determine the adequacy of the
designs used to assess program effectiveness.
At least 40 percent of the studies employed existing program records
in their evaluations, while about one-quarter employed special
surveys or other ad hoc data-collection methods specially designed
for the studies. Just under half (40 percent) of the studies used
data from program administrative records that were produced and
reported at the federal level; almost a third (28 percent) used data
from routinely produced, but not typically reported, program records;
5 percent of the studies used data from administrative records of
other federal agencies; and 14 percent used administrative records
from state programs. Some studies may have used many types of data
sources, which would suggest a heavy reliance on administrative and
other program-related data. (See table 5.)
Table 5
Data Sources Reported for the Evaluation
Studies Conducted in FY 1995
Number Percen
of t of
studie studie
Data sources s\a s\b
------------------------------------------------------ ------ ------
Generated by the program
Program administrative records routinely produced and 369 40%
reported federally
Program administrative records routinely produced but 257 28
not typically reported or aggregated centrally
Recurring surveys of program participants 111 12
Special surveys or other ad hoc data collections on 254 27
all or
part of the program or in selected locales
Generated by another agency
Administrative records or routinely aggregated data 46 5
from
other federal programs (e.g., Medicare Claims and
Utilization Files)
Administrative records or routine data from state 133 14
programs or agencies (e.g., Federal Accident
Reporting
System (FARS))
Federally sponsored multipurpose national surveys 44 5
(e.g.,
the decennial census)
Other\c 73 9
----------------------------------------------------------------------
\a An individual study could have been reported as using more than
one data source.
\b These percents are based on all 928 studies that respondents
reported were conducted in fiscal year 1995; however, responses to
this question were not provided on all studies. For example, 5
offices did not answer this question for any of the 81 studies they
reported having conducted. Therefore, these percentages may
underestimate the actual level of use of these data sources.
\c The other data sources included private industry, medical records
abstracts, and special studies generated by other agencies.
Source: GAO survey.
--------------------
\6 Since several offices did not report either their 1995 funding
level or the proportion of funding (or staffing resources) spent on
evaluation, this may underreport actual expenditures. By comparison,
we conducted a similar survey in 1984 that found that evaluation
resources represented roughly comparable proportions of agency
funding and staff resources. See Federal Evaluation: Fewer Units,
Reduced Resources, Different Studies from 1980 (GAO/PEMD-87-9, Jan.
23, 1987).
\7 For comparison, our 1984 survey of offices conducting program
evaluations found that half of all the studies lasted under 6 months,
and the majority of the studies that were contracted out lasted 1
year or less.
STUDIES PRIMARILY SERVED
INTERNAL PROGRAM IMPROVEMENT
PURPOSES, NOT CONGRESSIONAL
OVERSIGHT
---------------------------------------------------------- Letter :4.1
The primary reported purpose of the studies was to evaluate ongoing
programs, either on an office's own initiative or at the request of
top agency officials. In the survey, most officials conducting
evaluations reported having a formal and an ad hoc planning process
for deciding what evaluation work they would do. Many criteria were
indicated as being used to select which studies to do (such as a
program office request, congressional interest, or continuation or
follow-up of past work), but the criterion most often cited was the
interest of high-level agency officials in the program or subject
area. Moreover, about one-fourth of the studies were requested by
top agency officials. About one-fourth of the studies were indicated
to be self-initiated. Most offices were not conducting studies for
the Congress or as the result of legislative mandates; only 17
percent of the studies were reported to have been requested in those
ways. (See table 6.)
Table 6
Sources of Request or Mandate for
Evaluation Studies Conducted in FY 1995
Number
of Percen
report t of
ed all
studie studie
Source of mandate or request s\a s\b
------------------------------------------------------ ------ ------
Self-initiated 224 24%
Top agency officials 224 24
Program personnel 186 20
Legislation or congressional committee 154 17
OMB or executive order 26 3
Other\c 125 13
----------------------------------------------------------------------
\a An individual study could have been reported as being requested or
mandated by more than one source.
\b These percents are based on all 928 studies that respondents
reported were conducted in fiscal year 1995; however, responses to
this question were not provided on all studies. For example, 6
offices did not answer this question for any of the 105 studies they
reported having conducted. Therefore, these percents may
underestimate the actual level of study sources.
\c Other sources of study requests included advisory boards,
reengineering directives, NPR efforts, and state governments.
Source: GAO survey.
For those offices reporting that they conducted studies, about half
of the 570 studies for which we have information evaluated ongoing
programs.\8 Ongoing programs of all sizes were evaluated, ranging in
funding from less than $10 million to over $1 billion. About
one-third of these studies evaluated demonstration programs and many
of them cost less than $10 million. In contrast, few reported
evaluations of new programs and many of these new programs reportedly
were small (with funding under $10 million).
Program evaluation was reported to be used more often for general
program improvement than for direct congressional oversight. Their
primary uses most often were said to be to improve program
performance (88 percent), assess program effectiveness (86 percent),
increase general knowledge about the program (62 percent), and guide
resource allocation decisions within the program (56 percent). (See
table 7.) Accordingly, these offices overwhelmingly (over
three-fourths of respondents) reported program managers and
higher-level agency officials as the primary audience of their
studies. (See table 8.)
Table 7
Primary Uses Reported for the Results of
Evaluation Studies Conducted in FY 1995
Percent of
offices
reporting
as
Uses of study results primary\a
---------------------------------------------------------- ----------
Identify opportunities to improve program performance 88%
Ascertain the extent of the program's effectiveness 86
Increase general knowledge about the program or topic 62
Guide decisions on resource allocation within the program 56
Support program budget requests 31
Ascertain the success of corrective actions 30
Support program reauthorization 20
Other 14
----------------------------------------------------------------------
Note: Percents do not add to 100 because respondents could give
multiple responses.
\a Percents are based on a total of 81 offices reporting that they
conducted evaluation studies in fiscal year 1995. However, responses
to this survey question were not provided by all offices, so these
percents may somewhat underrepresent the various uses.
Source: GAO survey.
Table 8
Primary Audience Reported by Offices for
the Results of Evaluation Studies
Conducted in FY 1995
Percent of
Primary audience offices\a
---------------------------------------------------------- ----------
Program managers 85%
Higher-level agency officials 78
Agency evaluation staff 37
Office of Management and Budget 25
Congress, in general 32
Appropriations committees 22
Legislative committees 20
Program partners in private sector 26
Outside professional audience (experts, researchers, or 28
analysts)
General public or others 25
----------------------------------------------------------------------
Note: Percents do not add to 100 because respondents could give
multiple responses.
\a Percents are based on a total of 81 offices reporting that they
conducted evaluation studies in fiscal year 1995. However, responses
to this survey question were not provided by all offices so these
percents may somewhat underrepresent the various uses.
Source: GAO survey.
About one-third of the offices reported support for budget requests
as a primary use and one-third reported congressional audiences were
primary users for their studies. Fewer respondents (20 percent)
reported program reauthorization as a primary use of the study
results. (See tables 7 and 8.)
Program evaluation was not the primary responsibility for most of
these offices and the offices often reported "seldom, if ever"
performing the program evaluation roles we asked about. The role
most likely to be characterized as `most often performed' was
conducting studies of programs administered elsewhere in their
agency. (See table 9.)
Table 9
Roles Offices Played in Conducting
Evaluation Studies in FY 1995
Percent Reported
-----------------------------------------
Often
Seldom (about Always
, 1/2 or Number
if the Very almost report
Offices' roles in evaluations ever Sometimes time) often always ed
----------------------------- ------ --------- ------ ------ ------ ------
Conduct evaluation studies of 38% 34% 10% 7% 10% 68
programs that are run by
your office
Conduct evaluation studies of 26 29 10 17 17 69
programs administered
elsewhere in agency
Design studies to be 67 25 5 3 0 60
performed by others (other
than your own contractors)
Monitor evaluations conducted 34 45 3 8 9 64
by others
Provide technical or design 24 50 8 12 6 66
assistance to others'
evaluations
Approve plans for studies 59 23 3 10 5 61
conducted by others
Conduct joint or cooperative 22 52 13 9 4 69
studies with other units
Train other units (e.g., 61 26 3 7 3 61
federal, state, or local
government, private agencies
or firms) in research or
evaluation methods
Other activity/role 17 33 17 8 25 12
--------------------------------------------------------------------------------
Source: GAO survey.
About one-half of those who responded reported "sometimes" providing
technical or design assistance to others or conducting joint studies,
while a few offices saw their role as training others in research or
evaluation methods. One office dealing with an evaluation mandate
conducted work sessions with state and local program managers and
evaluators as well as provided training to enhance state evaluation
capabilities. Two-thirds of the offices seldom, if ever, designed
evaluations conducted by other offices or agencies, trained others in
research or evaluation methods, or approved plans for studies by
others.
--------------------
\8 Information was provided on program dollar size and maturity for
only 61 percent of the 928 studies that respondents reported were
conducted. Eighteen offices did not specifically answer this
question for any of the 280 studies they reported having conducted.
REFORMS MAY INCREASE INTEREST
IN, BUT COMPLICATE THE PROCESS
OF, OBTAINING RESULTS
------------------------------------------------------------ Letter :5
Some of our interviewees thought that recent governmentwide reforms
would increase interest in learning the results of federal programs
and policies but would also complicate the task of obtaining that
information. Devolution of federal program responsibility in the
welfare and health care systems has increased interest in evaluation
because the reforms are major and largely untested. However, in part
because programs devolved to the states are expected to operate quite
diversely across the country, some evaluation officials noted that
evaluating the effects of these reforms was expected to be more
difficult. In addition, federal budget reductions over the past few
years were said by some not only to have reduced the level of federal
evaluation activity but also to have diminished agency technical
capacity through the loss of some of their most experienced staff.
Because implementation of the Results Act's performance reporting
requirements is not very far along (the first annual reports on
program performance are not due until March 2000), several of our
interviewees thought it was too early to estimate the effect of the
Results Act. Some hoped the Act would increase the demand for
results information and expand the role of data and analysis in
decisionmaking. One interviewee thought it would improve the focus
of the evaluations they now conduct. A few evaluation officials were
concerned that a large investment would be required to produce valid
and reliable outcome (rather than process) data. A few also noted
that resources for obtaining data on a greatly expanded number of
program areas would compete for funds used for more in-depth
evaluations of program impact. Other evaluators noted that changes
in the unit of analysis for performance reporting from the program
level to budget account or organization might make classic program
evaluation models obsolete.
As we previously reported,\9 the federal program officials who have
already begun implementing performance measurement appeared to have
an unusual degree of program evaluation support and found it quite
helpful in addressing the analytic challenges of identifying program
goals, developing measures, and collecting data. Many of these
program officials said they could have used more of such assistance;
but, when asked why they were not able to get the help they needed,
the most common response was that it was hard to know in advance that
evaluation expertise would be needed. In addition to using program
evaluation techniques to clarify program goals and develop reliable
measures, several of these program officials saw the need for impact
evaluations to supplement their performance data. Their programs
typically consisted of efforts to influence highly complex systems or
events outside government control, where it is difficult to attribute
a causal connection between their program and its desired outcomes.
Thus, without an impact evaluation or similar effort to separate the
effects of their programs from those of other external events or
factors, program officials from the previous study recognized that
simple examination of outcome measures may not accurately reflect
their programs' performance.
Some states' experiences with performance measurement suggested that
performance measurement will take time to implement, and the federal
experience suggests that it will not supplant the need for
effectiveness evaluations. Two state officials described a multiyear
process to develop valid and reliable measures of program performance
across the state government. While performance measures were seen as
useful for program management, some state agency and legislative
staff also saw a continuing need for evaluations to assess policy
impact or address problems of special interest or "big-picture"
concerns, such as whether a government program should be continued or
privatized.
NPR was seen by several of those we interviewed as not having much of
an effect on efforts to evaluate the results of their programs beyond
increasing the use of customer surveys. This may have been because
it was seen as primarily concerned with internal government
operations, or because, as one agency official reported, its effect
was most noticeable in only a few areas: regulatory programs and
other intergovernmental partnerships. However, one agency official
said that NPR had a big impact on reorienting their work toward
facilitating program improvement, while two others felt that it
reaffirmed changes they had already begun.
--------------------
\9 GAO/HEHS/GGD-97-138.
AGENCIES SUGGESTED VARIED
STRATEGIES FOR OBTAINING
PROGRAM RESULTS INFORMATION
------------------------------------------------------------ Letter :6
LEVERAGING LIMITED RESOURCES
---------------------------------------------------------- Letter :6.1
Given constraints on federal budgets, some officials we interviewed
in general did not expect federal evaluation resources to rise to
meet demand, so they described efforts to leverage and prioritize
available resources. While an evaluation official reported
supplementing his evaluation teams with consultants, concern was also
expressed that staff reductions in their unit had left the technical
expertise too weakened to competently oversee consultants' work.
Another evaluation official explained that they responded to the
increasing demand for information by narrowing the focus and scope of
their studies to include only issues with major budget implications
or direct implications for agency action. Both a state official and
two external evaluation experts felt that states grappling with new
program responsibilities would have difficulty evaluating them as
well, so that continued federal investment would be needed. A
federal official, however, noted that private foundations could fund
the complex rigorous studies needed to answer causal questions about
program results.
Some of the evaluators we interviewed expected that fewer impact
studies would be done. Some expected that the range of their work
may broaden to rely on less rigorous methods and include alternatives
such as monitoring program performance and customer satisfaction.
From our interviews, we learned that a few agencies have devolved
responsibility for evaluations to the program offices, which may have
more interest in program improvement. Another agency reported that
it had built evaluation into its routine program review system, which
provides continuous information on the success of the program and its
outcomes, noting that it thereby reduced the need for special
evaluation studies. One evaluation official reported that by having
redefined evaluation as part of program management, program
evaluation became more acceptable in his agency because it no longer
appeared to be overhead.
A few agencies reported that they were adapting the elements of their
existing program information systems to yield information on program
results. But in other agencies, evaluation officials and external
experts thought that their systems were primarily focused on program
process, rather than results. The evaluation official said that
structural changes to, and a major investment in, their data systems
will be required to provide valid and meaningful data on results.
CREATING NEW EVALUATION
DESIGNS AND PARTNERSHIPS
---------------------------------------------------------- Letter :6.2
As program responsibility shifts to state and local entities,
evaluation officials and others we interviewed described the need for
study designs that can handle greater contextual complexity, new ways
to measure outcomes, and the need to build partnerships with the
programs' stakeholders. One of the officials saw classical
experimental research designs as no longer feasible in programs,
which, due to increased state flexibility in how to deliver services,
no longer represented a discrete national program or were unlikely to
employ rigorous evaluation techniques that entailed random assignment
of particular program services to individuals. Others noted the need
to develop evaluation designs that could reflect the multiple levels
on which programs operate and the organizational partnerships
involved. To address some of these complexities, federal offices
with related program interests have formed task groups to attempt to
integrate their research agendas on the effects of major changes in
the health and welfare systems. Similarly, a few federal evaluation
officials reported an interest in consulting with their colleagues in
other federal offices to share approaches for tackling the common
analytic problems they faced.
In other strategies, federal evaluation officials described existing
or planned efforts to change the roles they and other program
stakeholders played in conducting evaluations. One agency has
arranged for the National Academy of Sciences to work with state
program officials and the professional communities involved to help
build a prototype performance measurement system for federal
assistance to state programs. One evaluation office expects to shift
its role toward providing more technical assistance to local
evaluators and synthesizing their studies' results. Another federal
office has delegated some evaluation responsibility to the field
while it synthesizes the results to answer higher level policy
questions, such as which types of approaches work best.
The Results Act recognizes and encourages the complementary nature of
program evaluations and performance measures by asking agencies to
provide a summary of program evaluation findings along with
performance measurement results in their annual performance reports.
One federal evaluation official said his agency had efforts under way
to "align" program evaluation and performance measurement through,
for example, planning evaluations so that they will provide the
performance data needed. But, the official also expressed concern
about how to integrate the two types of information. Officials from
states that had already begun performance measurement and monitoring
said they would like to see the federal government provide more
leadership by (1) providing a catalog of performance measures
available for use in various program areas and (2) funding and
designing impact evaluations to supplement their performance
information.
CONCLUSIONS
------------------------------------------------------------ Letter :7
Seeking to improve government performance and public confidence in
government, the Results Act has instituted new requirements for
federal agencies to report on their results at the same time that
other management reforms may complicate the task of obtaining such
information. Comparison of current federal program evaluation
resources with the anticipated challenges leads us to several
conclusions.
First, federal agencies' evaluation resources have important roles to
play in responding to increased demand for information on program
results, but--as currently configured and deployed--they are likely
to be challenged to meet these future roles. It is implausible to
expect that, by simply conducting more program evaluation studies
themselves, these offices can produce data on results across all
activities of the federal government. Moreover, some agencies
reported that they had reduced their evaluation resources to the
point that the remaining staff feel unable to meet their current
responsibilities. Lastly, the devolution of some program
responsibilities to state and local governments has increased the
complexity of the programs they are being asked to evaluate, creating
new challenges.
Second, in the future, carefully targeting and reshaping the use of
federal evaluation resources and leveraging federal and nonfederal
resources show promise for addressing the most important questions
about program results. In particular, federal evaluators could
assist program managers to develop valid and reliable performance
reporting by sharing their expertise through consultation and
training. Early agency efforts to meet the Results Act's
requirements found program evaluation expertise helpful in managing
the numerous analytical challenges involved, such as clarifying
program goals and objectives, developing measures of program
outcomes, and collecting and analyzing data. In addition, because
performance measures will likely leave some gaps in needed
information, strategic planning for future evaluations might strive
to fill those gaps by focusing on those questions judged to have the
most policy importance. In many programs, performance measures alone
are not sufficient to establish program impact or the reasons for
observed performance. Program evaluations can also serve as valuable
supplements to program performance reporting by addressing policy
questions that extend beyond or across program borders, such as the
comparative advantage of one policy alternative to another.
Finally, without coordination, it is unlikely that the increasingly
diverse activities involved in evaluating an agency's programs will
efficiently supplement each other to meet both program improvement
and policymaking information needs. As some agencies devolve some of
the evaluations they conducted in the past to program staff or state
and local evaluators, they run the risk that, due to differences in
evaluation resources and questions, data from several studies
conducted independently may not likely be readily aggregated. Thus,
in order for such devolution of evaluation responsibility to better
provide an overall picture of a national program, those evaluations
would have to be coordinated in advance. Similarly, as federal
agencies increasingly face common analytic problems, they could
probably benefit from cross-agency discussion and collaboration on
approaches to those problems.
AGENCY COMMENTS
------------------------------------------------------------ Letter :8
The Director of OMB commented on a draft of this report and generally
agreed with our conclusions. OMB noted that other countries are
experiencing public sector reforms that include a focus on results
and increasing interest in program evaluation. OMB also provided
technical comments that we have incorporated as appropriate
throughout the text. OMB's comments are reprinted in appendix II.
We are sending copies of this report to the Chair and Ranking
Minority Member of the House Committee on Government Reform and
Oversight, the Director of OMB, and other interested parties. We
will also make copies available to others on request.
Please contact me or Stephanie Shipman, Assistant Director at (202)
512-7997 if you or your staff have any questions. Major contributors
to this report are listed in appendix III.
Sincerely yours,
Susan S. Westin
Associate Director, Advanced Studies
and Evaluation Methodology
SCOPE AND METHODOLOGY OF SURVEY
=========================================================== Appendix I
DEPARTMENTS AND INDEPENDENT
AGENCIES SURVEYED
--------------------------------------------------------- Appendix I:1
The 23 federal executive agencies (13 cabinet-level departments and
10 independent agencies) that we surveyed are listed as follows.
Departments Independent Agencies
Agriculture Agency for International Development
Commerce Environmental Protection Agency
Education Federal Emergency Management Agency
Energy General Services Administration
Health and Human Services National Aeronautics and Space Administration
Housing and Urban National Science Foundation
Development
Interior National Research Council
Justice Office of Personnel Management
Labor Small Business Administration
State Social Security Administration
Transportation
Treasury
Veterans Affairs
--------------------------------------------------------------------------------
These represent 23 of the 24 executive agencies (we excluded the
Department of Defense) covered by the Chief Financial Officer's Act.
The 24 represent about 97 percent of the executive branch's full-time
staff and cover over 99 percent of the federal government's outlay
for fiscal year 1996.
IDENTIFYING OFFICES
CONDUCTING EVALUATION
STUDIES
------------------------------------------------------- Appendix I:1.1
To identify the roles and resources expended on federal program
evaluation, we surveyed all offices (or units) in the 23 executive
branch departments and independent agencies that we identified as
conducting evaluation in fiscal year 1995. We defined evaluation as
systematic analysis using objective measures to assess the results or
the effects of federal programs, policies, or activities. To
identify these evaluation offices, we (1) began with the list of
evaluation offices that we surveyed in 1984 (2) added offices based
on a review of office titles implying analytical responsibilities and
discussions with experts knowledgeable about evaluation studies, and
(3) talked with our liaison staff and other officials in the federal
departments and agencies to ensure broad yet appropriate survey
coverage.
In some instances, the survey was distributed to offices throughout
an agency by agency officials, while in other instances we sent the
survey directly to named evaluation officials. We attempted to
survey as many evaluation offices as possible; however, in some
cases, we may not have been told about or directed to all such
offices. Therefore, we cannot assume that we have identified all
offices that conducted program evaluation studies in fiscal year
1995. Overall, we received about 160 responses, of which 81 were
from offices that conducted such studies.
The survey was directed toward results-oriented evaluation studies,
such as formal impact studies, assessments of program results, and
syntheses or reviews of evaluation studies. We sought to exclude
studies that focused solely on assessing client needs, describing
program operations or implementation, or assessing fraud, compliance,
or efficiency. However, we allowed the individual offices to (1)
define "program" since a federal program could be tied to a single
budget account, represent a combination of several programs, or
involve several state programs and (2) determine whether or not they
did this type of study and, if not, they could exempt themselves from
completing the survey. We did not verify the accuracy of the
responses provided by evaluation units. We also had some information
on fiscal year 1996 activities but did not report those results since
they were comparable to the fiscal year 1995 results.
RESPONDENT REPORTING
------------------------------------------------------- Appendix I:1.2
Some respondents were unable to complete different parts of the
survey. About one-third of the respondents did not report either the
office's budget, its number of full-time equivalent staff (FTE), cost
information about studies, or the sources of data used in the
studies. For some questions, respondents were asked to answer in
terms of the number of studies conducted, and we used the total
number of studies indicated by all respondents to the question as the
denominator when computing percents. However, when the level of
nonresponse to individual survey questions was above 20 percent or
was unclear due to incomplete information on how many studies had
been reported on, we used the full complement of 928 studies to
provide a conservative estimate.
SELECTED SURVEY QUESTIONS
--------------------------------------------------------- Appendix I:2
The questions for which we reported results are reproduced on the
following pages.
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)Appendix II
COMMENTS FROM THE OFFICE OF
MANAGEMENT AND BUDGET
=========================================================== Appendix I
MAJOR CONTRIBUTORS
========================================================= Appendix III
GENERAL GOVERNMENT DIVISION
Elaine Vaurio, Evaluator-in-Charge
Joseph Wholey, Senior Advisor for Evaluation Methodology
BIBLIOGRAPHY
========================================================== Appendix IV
Committee on Governmental Affairs, United States Senate. "Government
Performance and Results Act of 1993." Report No. 103-58, June 16,
1993.
Evaluation Practice. "Past, Present, Future Assessments of the Field
of Evaluation." Entire Issue. M.F. Smith, ed., Vol. 15, #3, Oct.
1994.
Martin, Margaret E., and Miron L. Straf (eds.). Principles and
Practices for a Federal Statistical Agency. Washington, D.C.:
National Academy Press, 1992.
National Performance Review. "Mission-Driven, Results-Oriented
Budgeting." Accompanying Report of the National Performance Review of
the Office of the Vice President, Sept. 1993.
New Directions for Program Evaluation. "Evaluation in the Federal
Government: Changes, Trends, and Opportunities." Entire issue. C.G.
Wye and R. Sonnichsen, eds. #55, Fall 1992.
New Directions for Program Evaluation. "Progress and Future
Directions in Evaluation: Perspectives on Theory, Practice, and
Methods." Entire issue. Debra Rog and Deborah Fournier, eds. #76,
Winter 1997.
Office of Evaluation and Inspections. Practical Evaluation for
Public Managers: Getting the Information You Need. Washington,
D.C.: Office of Inspector General, Department of Health and Human
Services, 1994.
Public Law 103-62, Aug. 3, 1993, "Government Performance and Results
Act of 1993."
Wargo, Michael J. "The Impact of Federal Government Reinvention on
Federal Evaluation Activity." Evaluation Practice, 16(3) (1995), pp.
227-237.
RELATED GAO PRODUCTS
The Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans (GAO/GGD-10.1.19, Mar. 1998).
Balancing Flexibility and Accountability: Grant Program Design in
Education and Other Areas (GAO/T-GGD/HEHS-98-94, Feb. 11, 1998).
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).
Managing for Results: Analytic Challenges in Measuring Performance
(GAO/HEHS/GGD-97-138, May 30, 1997).
Block Grants: Issues in Designing Accountability Provisions
(GAO/AIMD-95-226, Sept. 1995).
Program Evaluation: Improving the Flow of Information to the
Congress (GAO/PEMD-95-1, Jan. 30, 1995).
Management Reform: Implementation of the National Performance
Review's Recommendations (GAO/OGC-95-1, Dec. 5, 1994).
Public Health Service: Evaluation Set-Aside Has Not Realized Its
Potential to Inform the Congress (GAO/PEMD-93-13, Apr. 1993).
Program Evaluation Issues (GAO/OCG-93-6TR, Dec. 1992).
"Improving Program Evaluation in the Executive Branch." A Discussion
Paper by the Program Evaluation and Methodology Division
(GAO/PEMD-90-19, May 1990).
Program Evaluation Issues (GAO/OCG-89-8TR, Nov. 1988).
Federal Evaluation: Fewer Units, Reduced Resources, Different
Studies from 1980 (GAO/PEMD-87-9, Jan. 23, 1987).
(966704/973810)
*** End of document. ***