Program Evaluation and Methodology Issue Area Plan--Fiscal Years 1995-97
(Letter Report, 06/01/95, GAO/IAP-95-12).
GAO provided information on its Program Evaluation and Methodology
Issue Area Plan for fiscal years 1995 through 1997, focusing on: (1)
program effectiveness and quality; (2) government evaluation capability
and performance; and (3) methodology review and development.
GAO plans to: (1) identify key executive branch programs that are
ineffective, produce unintended effects, and are of poor quality; (2)
ensure that state and federal agencies are capable of evaluating their
programs in an unbiased and technical manner; (3) make program
evaluation within state and federal sectors an integral part of
government reform; (4) evaluate the methodological soundness of various
executive branch assessment methods; (5) identify policies based upon
flawed analyses; and (6) determine whether new and innovative evaluation
methods can be used to determine program effectiveness.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: IAP-95-12
TITLE: Program Evaluation and Methodology Issue Area Plan--Fiscal
Years 1995-97
DATE: 06/01/95
SUBJECT: Program evaluation
Agency missions
Research programs
Federal programs
Evaluation methods
Program management
Total quality management
Cost effectiveness analysis
Investigations into federal agencies
State-administered programs
IDENTIFIER: Medicare Program
TQM
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Program Evaluation and Methodology Division
June 1995
PROGRAM EVALUATION AND METHODOLOGY
ISSUE AREA PLAN
FISCAL YEARS 1995-97
GAO/IAP-95-12
Abbreviations
=============================================================== ABBREV
FOREWORD
============================================================ Chapter 0
As the investigative arm of Congress and the nation's auditor, the
General Accounting Office is charged with following the federal
dollar wherever it goes. Reflecting stringent standards of
objectivity and independence, GAO's audits, evaluations, and
investigations promote a more efficient and cost-effective
government; expose fraud, waste, abuse, and mismanagement in federal
programs; help Congress target budget reductions; assess financial
and information management; and alert Congress to developing trends
that may have significant fiscal or budgetary consequences. In
fulfilling its responsibilities, GAO performs original research and
uses hundreds of databases, or creates its own when information is
unavailable elsewhere.
To ensure that GAO's resources are directed toward the most important
issues facing Congress, each of GAO's 35 issue areas develops a
strategic plan that describes the significance of the issues it
addresses, its objectives, and the focus of its work. Each issue
area relies heavily on input from congressional committees, agency
officials, and subject-matter experts in developing its strategic
plan.
The Program Evaluation and Methodology issue area is a technical area
of work implemented within GAO. Because of the growing need of
Congress to understand the impact or effects of federal programs, the
issue area was developed to use innovative research methodologies for
evaluating federal and related programs and activities.
Consequently, the work is characterized by its interdisciplinary
approach. Projects are conducted by sociologists, psychologists,
engineers, economists, and statisticians as well as other staff with
technical backgrounds, most of whom hold doctorates. Our projects
are designed to address evaluation and methodological questions that
require complex evaluation approaches and extensive data analyses.
To address these questions, research methods are used to design the
evaluations, collect and analyze data, and interpret the analytical
results. These evaluations are conducted across a number of
substantive areas. They include defense, education, agriculture,
aging, environment, health, public management, transportation, and
welfare. Consequently, the program evaluation and methodological
work crosscuts other substantive work areas within the agency.
The work emphasizes three issues:
1. The program effectiveness and quality issue focuses on executive
branch programs, including research work conducted by or for federal
agencies.
2. The government evaluation capability and performance issue
focuses on state and federal agencies.
3. The methodological review and development issue focuses on three
areas: the soundness of statistical and measurement methods used
within the executive branch; the soundness of agencies' testing and
assessment methods; the contributions of new and innovative
evaluation and analytical approaches to evaluating program
effectiveness.
Our issue area planning process is used to develop preliminary ideas
concerning the focus of our work, and then identify high-priority
projects. In order to accomplish this, we consult with Members of
Congress and their staffs as well as key industry and agency
officials. In addition, we convene an advisory panel of nationally
recognized technical experts to review our project proposals and
provide recommendations concerning the focus and scope of the
evaluation methods we plan to use. In the sections that follow, we
characterize the work we plan to conduct for fiscal years 1995, 1996,
and 1997. If you have any questions or suggestions concerning this
plan, please call me at (202) 512-2900.
Terry E. Hedrick
Assistant Comptroller General for
Program Evaluation and Methodology
CONTENTS
============================================================ Chapter 1
FOREWORD
---------------------------------------------------------- Chapter 1:1
1
TABLE I: KEY ISSUES
---------------------------------------------------------- Chapter 1:2
4
TABLE II: PLANNED MAJOR WORK
---------------------------------------------------------- Chapter 1:3
6
TABLE III: GAO CONTACTS
---------------------------------------------------------- Chapter 1:4
7
TABLE I: KEY ISSUES
============================================================ Chapter 2
Issue Significance
--------------------------------------- ---------------------------------------
Program effectiveness and quality: What It is important to understand how much
is the effectiveness or quality of executive branch programs, including
executive branch programs? research programs, are actually
achieving their desired results,
especially if there are questions about
whether they are cost-effective. It is
also important to understand whether
these programs are having unintended
effects, both positive and negative. In
some cases, studies focus on assessing
the quality of a program rather than
its results.
Government evaluation capability and Title VII of the GAO authorizing
performance: What is the evaluation legislation mandates the assessment of
capability and performance of state and the executive branch's program
federal agencies? evaluation capability and performance.
The Government Performance and Results
Act emphasizes that executive branch
agencies should measure performance.
State governments currently implement
most social programs, and they may take
over even more programs. Assessing
these agencies' capability to conduct
evaluations and examining the quality
of that work will help ensure that
programs are being designed and
implemented in a cost-effective and
efficient manner.
Methodology review and development: Are Increasingly, federal agencies are
analytical methods used within the required to use sophisticated
executive branch sound and can new and analytical methods in conducting their
innovative methods contribute to an work. This includes statistical,
understanding of program effectiveness? measurement, indicator, testing, and
assessment methods. The application of
these methods and the analytical
results they produce influence both
policy and program decisions.
Consequently, it is important to
determine whether these analytical
methods are sound. In addition, new and
innovative methods are being developed
that may have applications for a broad
range of evaluations or analytical
assessments. It is therefore important
to examine their potential, especially
if they could be applied to programs
that have not been assessed because of
the shortcomings of traditional
methods.
--------------------------------------------------------------------------------
Objectives Focus of work
--------------------------------------- ---------------------------------------
--Identify key executive branch --The effectiveness of federal
programs, including research programs, regulatory health protection programs.
that are not effective.
--Identify executive branch programs --The effectiveness of cost containment
where unintended effects occur that are approaches in health care.
unacceptable.
--Identify executive branch programs, --Cost-effectiveness and resource
including research programs, whose allocations in defense systems.
quality is unacceptable.
--Determine whether the effectiveness --Effectiveness of medical training,
or quality of the programs can be technologies, and clinical research.
improved and, if not, whether the
program is viable. --The effectiveness of productivity and
quality-enhancing programs in the
federal government.
--Viability of technologies for
immigration control and for
transportation.
--Ensure that state and federal --Identification of information and
agencies either have or are developing methods required to evaluate
the capability to evaluate their legislatively mandated federal
programs. programs.
--Ensure that program evaluations are
unbiased and technically sound. --The adequacy of evaluations assessing
--Establish program evaluation within immigration issues and programs.
the state and federal sectors as an
important component of reforming --Capability of federal and state
government. agencies to conduct program
evaluations.
--Evaluate the methodological soundness --Adequacy of models and measurement
of statistical, measurement, testing, methods applied in the agricultural and
and assessment methods used by environmental protection areas.
executive branch agencies to support
major policy and program decisions. --The quality of measures used to
--Identify policies or programs that assess social conditions.
are based upon flawed analyses and
identify the consequences of these --The acceptability of methods used to
programs. design and implement defense programs.
--Determine whether the methodological
problems can be corrected and, if not, --The feasibility of using innovative
whether the affected policies or applications of analytical methods to
programs are viable. assess medical technologies.
--Determine whether new and innovative
methods can be applied to programs to
determine their effectiveness.
--------------------------------------------------------------------------------
TABLE II: PLANNED MAJOR WORK
============================================================ Chapter 3
Issue Planned major job starts
---------------------------------- ----------------------------------
Program effectiveness or quality --Federal responses to
environmental health risks
--Nutritional training
requirements for primary care
physicians
--Government support of super-car
development
--Cost-effectiveness of
alternative fuels
--Cost-effectiveness of focused
medical review
--Effects of drug utilization
review on Medicaid drug
prescription
--Utilization and effectiveness of
expert systems
--Cost-effectiveness of defense
sensor technologies
--Impact of structural variation
in managed care on health care
systems
--The effects of regulatory
requirements on clinical research
--Alzheimer's disease research
--Medicare Part B prescreening
effectiveness
--The accuracy and cost-
effectiveness of medical tests
--Effectiveness of Immigration and
Naturalization Service border
control
Government evaluation capability --Information necessary and
and performance available for evaluating the
Individuals with Disabilities
Education Act
--The use of program evaluation in
implementing the Government
Performance and Results Act
--Immigration and Naturalization
Service evaluation of worker
identity systems
--Federal and state government
program evaluation capabilities
Methodology review and development --Adequacy of models and
indicators for measuring
agricultural sustainability
--Validity of alternative measures
for welfare dependency
--Methodological foundations of
the DOD science technology
program
--Biases in the armed forces
selection testing
--Factors that contribute to poor
performance on educational
assessment
--The application of spatial
analysis to environmental equity
issues
--Assessment of hormone
replacement therapy for menopause
--Meta-analysis of data supporting
applications for drug approval by
the Food and Drug Administration
----------------------------------------------------------------------
TABLE III: GAO CONTACTS
============================================================ Chapter 4
ASSISTANT COMPTROLLER
GENERAL
-------------------------------------------------------- Chapter 4:0.1
Terry E. Hedrick, (202) 512-2900
DIRECTOR OF PLANNING AND
REPORTING
-------------------------------------------------------- Chapter 4:0.2
Joseph F. Delfico, (202) 512-2900
ASSISTANT DIRECTOR, PLANNING
AND REPORTING
-------------------------------------------------------- Chapter 4:0.3
Boris Kachura
DIRECTOR OF OPERATIONS
-------------------------------------------------------- Chapter 4:0.4
Franklin Frazier, (202) 512-2900
DIRECTOR OF PROGRAM
EVALUATION IN THE PHYSICAL
SYSTEMS AREA (PEPSA)
-------------------------------------------------------- Chapter 4:0.5
Kwai-Cheung Chan, (202) 512-3092
ASSISTANT DIRECTORS, PEPSA
-------------------------------------------------------- Chapter 4:0.6
Marcia Crosse
John Oppenheim
Sushil Sharma
Winslow Wheeler
Robert White
DIRECTOR OF PROGRAM
EVALUATION IN THE HUMAN
SERVICES AREA (PEHSA)
-------------------------------------------------------- Chapter 4:0.7
Robert York, (202) 512-5885
ASSISTANT DIRECTORS, PEHSA
-------------------------------------------------------- Chapter 4:0.8
Judy Droitcour
Patrick Grasso
Gail MacColl
Stephanie Shipman
George Silberman
Boris Kachura
*** End of document. ***