Program Evaluation and Methodology Issue Area Plan--Fiscal Years 1995-97
(Letter Report, 06/01/95, GAO/IAP-95-12).

GAO provided information on its Program Evaluation and Methodology
Issue Area Plan for fiscal years 1995 through 1997, focusing on: (1)
program effectiveness and quality; (2) government evaluation capability
and performance; and (3) methodology review and development.

GAO plans to: (1) identify key executive branch programs that are
ineffective, produce unintended effects, and are of poor quality; (2)
ensure that state and federal agencies are capable of evaluating their
programs in an unbiased and technical manner; (3) make program
evaluation within state and federal sectors an integral part of
government reform; (4) evaluate the methodological soundness of various
executive branch assessment methods; (5) identify policies based upon
flawed analyses; and (6) determine whether new and innovative evaluation
methods can be used to determine program effectiveness.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  IAP-95-12
     TITLE:  Program Evaluation and Methodology Issue Area Plan--Fiscal 
             Years 1995-97
      DATE:  06/01/95
   SUBJECT:  Program evaluation
             Agency missions
             Research programs
             Federal programs
             Evaluation methods
             Program management
             Total quality management
             Cost effectiveness analysis
             Investigations into federal agencies
             State-administered programs
IDENTIFIER:  Medicare Program
             TQM
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Program Evaluation and Methodology Division

June 1995

PROGRAM EVALUATION AND METHODOLOGY
ISSUE AREA PLAN

FISCAL YEARS 1995-97

GAO/IAP-95-12



Abbreviations
=============================================================== ABBREV


FOREWORD
============================================================ Chapter 0

As the investigative arm of Congress and the nation's auditor, the
General Accounting Office is charged with following the federal
dollar wherever it goes.  Reflecting stringent standards of
objectivity and independence, GAO's audits, evaluations, and
investigations promote a more efficient and cost-effective
government; expose fraud, waste, abuse, and mismanagement in federal
programs; help Congress target budget reductions; assess financial
and information management; and alert Congress to developing trends
that may have significant fiscal or budgetary consequences.  In
fulfilling its responsibilities, GAO performs original research and
uses hundreds of databases, or creates its own when information is
unavailable elsewhere. 

To ensure that GAO's resources are directed toward the most important
issues facing Congress, each of GAO's 35 issue areas develops a
strategic plan that describes the significance of the issues it
addresses, its objectives, and the focus of its work.  Each issue
area relies heavily on input from congressional committees, agency
officials, and subject-matter experts in developing its strategic
plan. 

The Program Evaluation and Methodology issue area is a technical area
of work implemented within GAO.  Because of the growing need of
Congress to understand the impact or effects of federal programs, the
issue area was developed to use innovative research methodologies for
evaluating federal and related programs and activities. 
Consequently, the work is characterized by its interdisciplinary
approach.  Projects are conducted by sociologists, psychologists,
engineers, economists, and statisticians as well as other staff with
technical backgrounds, most of whom hold doctorates.  Our projects
are designed to address evaluation and methodological questions that
require complex evaluation approaches and extensive data analyses. 

To address these questions, research methods are used to design the
evaluations, collect and analyze data, and interpret the analytical
results.  These evaluations are conducted across a number of
substantive areas.  They include defense, education, agriculture,
aging, environment, health, public management, transportation, and
welfare.  Consequently, the program evaluation and methodological
work crosscuts other substantive work areas within the agency. 

The work emphasizes three issues: 

1.  The program effectiveness and quality issue focuses on executive
branch programs, including research work conducted by or for federal
agencies. 

2.  The government evaluation capability and performance issue
focuses on state and federal agencies. 

3.  The methodological review and development issue focuses on three
areas:  the soundness of statistical and measurement methods used
within the executive branch; the soundness of agencies' testing and
assessment methods; the contributions of new and innovative
evaluation and analytical approaches to evaluating program
effectiveness. 

Our issue area planning process is used to develop preliminary ideas
concerning the focus of our work, and then identify high-priority
projects.  In order to accomplish this, we consult with Members of
Congress and their staffs as well as key industry and agency
officials.  In addition, we convene an advisory panel of nationally
recognized technical experts to review our project proposals and
provide recommendations concerning the focus and scope of the
evaluation methods we plan to use.  In the sections that follow, we
characterize the work we plan to conduct for fiscal years 1995, 1996,
and 1997.  If you have any questions or suggestions concerning this
plan, please call me at (202) 512-2900. 

Terry E.  Hedrick
Assistant Comptroller General for
 Program Evaluation and Methodology


CONTENTS
============================================================ Chapter 1


   FOREWORD
---------------------------------------------------------- Chapter 1:1

1


   TABLE I:  KEY ISSUES
---------------------------------------------------------- Chapter 1:2

4


   TABLE II:  PLANNED MAJOR WORK
---------------------------------------------------------- Chapter 1:3

6


   TABLE III:  GAO CONTACTS
---------------------------------------------------------- Chapter 1:4

7


TABLE I:  KEY ISSUES
============================================================ Chapter 2

Issue                                    Significance
---------------------------------------  ---------------------------------------
Program effectiveness and quality: What  It is important to understand how much
is the effectiveness or quality of       executive branch programs, including
executive branch programs?               research programs, are actually
                                         achieving their desired results,
                                         especially if there are questions about
                                         whether they are cost-effective. It is
                                         also important to understand whether
                                         these programs are having unintended
                                         effects, both positive and negative. In
                                         some cases, studies focus on assessing
                                         the quality of a program rather than
                                         its results.










Government evaluation capability and     Title VII of the GAO authorizing
performance: What is the evaluation      legislation mandates the assessment of
capability and performance of state and  the executive branch's program
federal agencies?                        evaluation capability and performance.
                                         The Government Performance and Results
                                         Act emphasizes that executive branch
                                         agencies should measure performance.
                                         State governments currently implement
                                         most social programs, and they may take
                                         over even more programs. Assessing
                                         these agencies' capability to conduct
                                         evaluations and examining the quality
                                         of that work will help ensure that
                                         programs are being designed and
                                         implemented in a cost-effective and
                                         efficient manner.

Methodology review and development: Are  Increasingly, federal agencies are
analytical methods used within the       required to use sophisticated
executive branch sound and can new and   analytical methods in conducting their
innovative methods contribute to an      work. This includes statistical,
understanding of program effectiveness?  measurement, indicator, testing, and
                                         assessment methods. The application of
                                         these methods and the analytical
                                         results they produce influence both
                                         policy and program decisions.
                                         Consequently, it is important to
                                         determine whether these analytical
                                         methods are sound. In addition, new and
                                         innovative methods are being developed
                                         that may have applications for a broad
                                         range of evaluations or analytical
                                         assessments. It is therefore important
                                         to examine their potential, especially
                                         if they could be applied to programs
                                         that have not been assessed because of
                                         the shortcomings of traditional
                                         methods.
--------------------------------------------------------------------------------
Objectives                               Focus of work
---------------------------------------  ---------------------------------------
--Identify key executive branch          --The effectiveness of federal
programs, including research programs,   regulatory health protection programs.
that are not effective.
--Identify executive branch programs     --The effectiveness of cost containment
where unintended effects occur that are  approaches in health care.
unacceptable.
--Identify executive branch programs,    --Cost-effectiveness and resource
including research programs, whose       allocations in defense systems.
quality is unacceptable.
--Determine whether the effectiveness    --Effectiveness of medical training,
or quality of the programs can be        technologies, and clinical research.
improved and, if not, whether the
program is viable.                       --The effectiveness of productivity and
                                         quality-enhancing programs in the
                                         federal government.

                                         --Viability of technologies for
                                         immigration control and for
                                         transportation.

--Ensure that state and federal          --Identification of information and
agencies either have or are developing   methods required to evaluate
the capability to evaluate their         legislatively mandated federal
programs.                                programs.
--Ensure that program evaluations are
unbiased and technically sound.          --The adequacy of evaluations assessing
--Establish program evaluation within    immigration issues and programs.
the state and federal sectors as an
important component of reforming         --Capability of federal and state
government.                              agencies to conduct program
                                         evaluations.



--Evaluate the methodological soundness  --Adequacy of models and measurement
of statistical, measurement, testing,    methods applied in the agricultural and
and assessment methods used by           environmental protection areas.
executive branch agencies to support
major policy and program decisions.      --The quality of measures used to
--Identify policies or programs that     assess social conditions.
are based upon flawed analyses and
identify the consequences of these       --The acceptability of methods used to
programs.                                design and implement defense programs.
--Determine whether the methodological
problems can be corrected and, if not,   --The feasibility of using innovative
whether the affected policies or         applications of analytical methods to
programs are viable.                     assess medical technologies.
--Determine whether new and innovative
methods can be applied to programs to
determine their effectiveness.



--------------------------------------------------------------------------------

TABLE II:  PLANNED MAJOR WORK
============================================================ Chapter 3

Issue                               Planned major job starts
----------------------------------  ----------------------------------
Program effectiveness or quality    --Federal responses to
                                    environmental health risks
                                    --Nutritional training
                                    requirements for primary care
                                    physicians
                                    --Government support of super-car
                                    development
                                    --Cost-effectiveness of
                                    alternative fuels
                                    --Cost-effectiveness of focused
                                    medical review
                                    --Effects of drug utilization
                                    review on Medicaid drug
                                    prescription
                                    --Utilization and effectiveness of
                                    expert systems
                                    --Cost-effectiveness of defense
                                    sensor technologies
                                    --Impact of structural variation
                                    in managed care on health care
                                    systems
                                    --The effects of regulatory
                                    requirements on clinical research
                                    --Alzheimer's disease research
                                    --Medicare Part B prescreening
                                    effectiveness
                                    --The accuracy and cost-
                                    effectiveness of medical tests
                                    --Effectiveness of Immigration and
                                    Naturalization Service border
                                    control

Government evaluation capability    --Information necessary and
and performance                     available for evaluating the
                                    Individuals with Disabilities
                                    Education Act
                                    --The use of program evaluation in
                                    implementing the Government
                                    Performance and Results Act
                                    --Immigration and Naturalization
                                    Service evaluation of worker
                                    identity systems
                                    --Federal and state government
                                    program evaluation capabilities

Methodology review and development  --Adequacy of models and
                                    indicators for measuring
                                    agricultural sustainability
                                    --Validity of alternative measures
                                    for welfare dependency
                                    --Methodological foundations of
                                    the DOD science technology
                                    program
                                    --Biases in the armed forces
                                    selection testing
                                    --Factors that contribute to poor
                                    performance on educational
                                    assessment
                                    --The application of spatial
                                    analysis to environmental equity
                                    issues
                                    --Assessment of hormone
                                    replacement therapy for menopause
                                    --Meta-analysis of data supporting
                                    applications for drug approval by
                                    the Food and Drug Administration
----------------------------------------------------------------------

TABLE III:  GAO CONTACTS
============================================================ Chapter 4


      ASSISTANT COMPTROLLER
      GENERAL
-------------------------------------------------------- Chapter 4:0.1

Terry E.  Hedrick, (202) 512-2900


      DIRECTOR OF PLANNING AND
      REPORTING
-------------------------------------------------------- Chapter 4:0.2

Joseph F.  Delfico, (202) 512-2900


      ASSISTANT DIRECTOR, PLANNING
      AND REPORTING
-------------------------------------------------------- Chapter 4:0.3

Boris Kachura


      DIRECTOR OF OPERATIONS
-------------------------------------------------------- Chapter 4:0.4

Franklin Frazier, (202) 512-2900


      DIRECTOR OF PROGRAM
      EVALUATION IN THE PHYSICAL
      SYSTEMS AREA (PEPSA)
-------------------------------------------------------- Chapter 4:0.5

Kwai-Cheung Chan, (202) 512-3092


      ASSISTANT DIRECTORS, PEPSA
-------------------------------------------------------- Chapter 4:0.6

Marcia Crosse
John Oppenheim
Sushil Sharma
Winslow Wheeler
Robert White


      DIRECTOR OF PROGRAM
      EVALUATION IN THE HUMAN
      SERVICES AREA (PEHSA)
-------------------------------------------------------- Chapter 4:0.7

Robert York, (202) 512-5885


      ASSISTANT DIRECTORS, PEHSA
-------------------------------------------------------- Chapter 4:0.8

Judy Droitcour
Patrick Grasso
Gail MacColl
Stephanie Shipman
George Silberman
Boris Kachura

*** End of document. ***