Program Evaluation: Improving the Flow of Information to the
Congress (30-JAN-95, PEMD-95-1).
Pursuant to a congressional request, GAO provided information on:
(1) the kinds of information that would be useful to the Senate
Committee on Labor and Human Resources for program oversight and
reauthorization reviews; (2) the extent to which agencies
currently collect and report such information; and (3) a strategy
the Committee could use to improve its access to agency
information.
-------------------------Indexing Terms-------------------------
REPORTNUM: PEMD-95-1
ACCNO: 153296
TITLE: Program Evaluation: Improving the Flow of Information to
the Congress
DATE: 01/30/1995
SUBJECT: Agency reports
Authorizing committees
Communication
Congressional oversight
Congressional/executive relations
Evaluation methods
Program evaluation
Reporting requirements
Statistical data
Dept. of Education Chapter 1 Program for
Educationally Disadvantaged Children
Dept. of Education National Assessment
of Educational Progress
Head Start Program
HHS Community and Migrant Health Centers
Program
HHS Comprehensive Child Development
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
PEMD-95-1
Program Evaluation and Methodology Division
B- 257897 January 30,1995 The Honorable Edward M. Kennedy Ranking Minority
Member Committee on Labor and Human Resources United States Senate
Dear Senator Kennedy: This report responds to your request that we assist
the Committee in identifying and obtaining the information needed to oversee
and evaluate the programs under its jurisdiction. We review the questions
that the Committee is likely to find useful for evaluating programs of
different types. We then explore the extent to which three programs collect
information pertinent to these questions and report it to the Committee.
Finally, we suggest a strategy that the Committee might use to identify
evaluation questions of interest for a given program and to request
information in advance from agency officials for later use in
reauthorization or oversight hearings.
We will send copies of this report to the Secretaries of Education and
Health and Human Services and to interested congressional committees. Copies
will also be made available to others upon request.
If you have any questions or would like additional information, please call
me at (202) 512- 2900 or Robert L. York, Director of Program Evaluation in
Human Services Areas, at (202) 512- 5885. Major contributors are listed in
appendix VI.
Sincerely yours, Terry E. Hedrick Assistant Comptroller General
Executive Summary Purpose Committees of the Congress need evaluative
information to help them
make decisions about the programs they oversee- information that tells them
whether, and in what important respects, a program is working well or
poorly. Concerned that the information it receives from administrative
agencies is often insufficient, the Senate Committee on Labor and Human
Resources asked GAO to (1) identify the kinds of information that would be
useful for oversight and reauthorization review of the types of programs
under its jurisdiction; (2) examine the extent to which agencies collect and
report such information; and (3) propose a strategy the Committee could use
to improve its access to agency information.
Background The Senate Committee on Labor and Human Resources oversees most
of the programs administered by the Departments of Education, Labor, and
Health and Human Services (HHS). These departments fund a wide variety of
programs, from basic medical research to education services for
disadvantaged children. The Committee can request information in various
ways, but it does not automatically receive evaluative information about
each program as it comes up for legislative action.
GAO drew from the literature of program evaluation and consulted experts and
Committee staff to identify categories of information and core questions
within each category that would be useful for oversight and reauthorization
review of various programs. Statistics, research, demonstration, service,
and regulatory programs were included in the review. The information
categories covered included descriptive information about the programs and
evaluative information about program implementation, targeting, impact, side
effects, and comparative advantage. GAO then examined documents and
interviewed officials of three programs to ascertain which core questions
were pertinent to each program, what information relevant to those questions
was available, what information had been requested by and reported to the
Committee, and how requests and responses had been communicated. The three
programs, selected with the Committee's agreement, were
l the Comprehensive Child Development Program (CCDP), which aims to
demonstrate that providing very young children and their families with
educational, health, and social services contributes to child development
and family self- sufficiency; l the Community Health Centers (CHC) program,
which provides clinical
services to medically underserved areas or populations; and Page 2 GAO/
PEMD- 95- 1 Improving the Flow of Program Information to the Congress
Executive Summary GAO's Analysis Committee staff accorded good,
appropriately disaggregated descriptive
information high priority and indicated that it would answer many of their
questions about a program. Information about implementation is critical for
oversight of new programs and provisions and for evaluation of programs
whose activities must be implemented in accordance with professional
standards if they are to succeed. The relevance of questions about program
targeting, impact, side effects, and comparative advantage reflects features
specific to each program and is best determined on a case- by- case basis.
The programs included in this study offer contrasting examples of
information requested, collected, and received. The CCDP was designed with
evaluation in mind and collected a wealth of descriptive information on
clients and services, as well as data that could answer evaluative questions
about program implementation and effects. Although the mandated evaluation
report on the program was still in preparation, some emerging findings about
CCDP had reached the Committee through various means in 1993 and early 1994.
The CHC program collected information for program management purposes that
could be organized to answer questions about service quality, program
targeting, and other relevant issues. Agency staff used this information to
answer specific questions from the Committee, but available information had
not been synthesized and much of it remained unreported. The Chapter 1 ESEA
program collected little management information from grantees but drew on
data from a wide range of studies that were summarized in a mandated
evaluation report in 1993.
In each of these cases, the Committee obtained the program information it
requested. It asked for and received information on particular aspects of
the CHC program. The Congress mandated studies on each of the other two
programs, and the Committee received reports accordingly. However, the
information the Committee had initially asked for- once received- did not
always meet its current needs. For example, information on discrete aspects
of the CHC program did not convey an overall picture of the program. Both
the CCDP and the Chapter 1 ESEA evaluation mandates as expressed in the
statute presented feasibility problems or design constraints that limited
the usefulness of results. (For example, information on program impact was
to be presented before new provisions could have been fully implemented.) In
both cases, the policy issues had also changed somewhat by the time the
report was due.
Page 4 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Executive Summary . the Chapter 1 Elementary and Secondary Education Act
(ESEA) program of
grants to states, which in turn fund school districts to provide
compensatory education services in schools serving low- income students.
Drawing on the broad review of information needs and the case studies, GAO
framed a strategy the Committee could use to request information from an
agency in order to evaluate whether a program is adequately serving its
intended purpose. Of course, the purpose itself may need to be reevaluated.
However, such reevaluations draw on information from many . sources, not
primarily from the administering agency, and thus fall outside
the scope of this report. Results in Brief GAO concluded that a brief list
of descriptive and evaluative questions
could capture the agency information most useful for program evaluation.
Questions can be selected and adapted from this list to fit the program and
the Committee's purpose in seeking information. Descriptive information is
essential to the Committee and should be disaggregated to show how
activities, settings, and clients vary within a program. For oversight
purposes in the years between reauthorizations, it is important that the
Committee receive information on progress in implementing new provisions and
notification of new developments that may require future attention.
Reauthorization decisions may call for systematic answers to evaluative
questions about program implementation, targeting, impact, and side effects.
When associated with policy review, such decisions may also call for
comparison of the current program to various alternatives.
Each of the programs GAO studied collected a great deal of useful
information. However, much of this information was not requested and did not
reach the Committee. Information that was specifically requested did reach
the Committee, but much of it was difficult to digest, too highly
aggregated, or received too late to be useful. Communication between
Committee and agency staff on information issues was limited and afforded
little opportunity to build a shared understanding of the Committee's needs
and how to meet them.
GAO concluded that obtaining useful information involves selecting pertinent
descriptive and evaluative questions, explicitly requesting a response, and
communicating with agency staff to ensure mutual understanding of what is
needed.
Page 3 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Executive Summary GAO observed that contacts between Committee and agency
staff about these programs consisted of a series of one- way communications
(from the Committee to the agency or the reverse), with little opportunity
for direct discussion. There appeared to be no current mechanism for .
adjusting requests in the interest of technical soundness or new priorities,
or of adjusting reporting plans and formats to increase the timeliness and
accessibility of the information reported. Communications between the
executive and legislative branches on evaluative questions are policy-
sensitive and follow formalized procedures. However, increased communication
about information needs could be accommodated within these procedures.
Matters for Congressional Consideration
This report proposes a strategy for obtaining information to assist with
program oversight and reauthorization review that the Committee may wish to
adopt. The three components of this strategy are (1) selecting and adapting,
from a core list, the descriptive and evaluative questions to be asked about
a program in interim years and at reauthorization; (2) arranging explicitly
to obtain timely oversight information in interim years as well as to
receive results of evaluation studies at reauthorization; and (3) providing
for increased communication with agency program and evaluation staff to help
ensure that information needs are understood and that requests and reports
are suitably framed and are adapted as needs evolve. This strategy can be
adapted to take institutional realities into account. For example, in view
of the many demands on its attention, the Committee might select future
reauthorization questions for some programs and invite agencies to propose
questions for others.
Agency Comments Responsible officials from the Departments of Education and
HHS provided written comments on a draft of this report. (See appendixes IV
and V.)
Officials from both agencies concurred with GAO'S suggestion that the
dialogue between the Committee and the agencies be strengthened at critical
points in the evaluation process. While generally supporting GAO'S approach,
agency officials were concerned that it be consistent with the requirements
of the Government Performance and Results Act (GPRA) and ‘ not
constitute an added burden. GAO considers its approach to be compatible with
GPRA and foresees little, if any, added burden on the annual reporting of
program performance data When the Committee requests information about other
aspects of a program, such as side effects or comparative advantage,
consultation should ensure that burden is taken into account as evaluations
are planned.
Page 6 GAOfPEMD- 95- 1 Improving the Flow of Program Information to the
Congress
' Contents Executive Summary 2 Chapter 1 Introduction
8 Background Objectives, Scope, and Methodology Types of Programs the
Committee Oversees Case Study Programs Reviewed Strengths and Limitations of
Our Study Organization of This Report
8 9 10
12 13 14
Chapter 2 What Information Would Be Most
Categories of Information and Related Questions Different Forms of Program
Information and Why They Are
Needed 15
16 23
Useful? Summary Observations 24 Chapter 3 27 What Information Is
Introduction 27 Available? Three Case Demonstration Program: CCDP 27
Service Programs: CHC and Chapter 1 ESEA 31 Studies Communicating About
Information Needs 37
Concluding Observations 39 Chapter 4 41 A Strategy for Identifying and
Adapting Core Questions 41
Arranging for Responses 45 Requesting Evaluation Communicating About
Information Needs 47 Information Feasibility of Our Approach 48
Agency Comments and Our Response 49 Appendixes Appendix I: Comprehensive
Child Development Program
Appendix II: Community Health Centers Appendix III: Chapter 1 Elementary and
Secondary Education
Act 52
58 64
Appendix IV Comments From the Department of Education Appendix V Comments
From the Department of Health and
Human Services 72
74 Appendix VI: Major Contributors to This Report 77
Page 6 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Contents Bibliography 78 Related GAO Products 84 Tables Table 2.1: Core
Questions by Category of Information
Table 3.1: Core Questions Adapted to CHC and Chapter 1 ESEA Programs
26 32
Table 4.1: Guide to Identifying Needed Information Table I. 1: CCDP
Evaluation Questions and Related Information Table II. 1: CHC Evaluation
Questions and Related Information Table III. 1: Chapter 1 ESEA Evaluation
Questions and Related
Information 44
55 61
68 Abbreviations ACYF Administration on Children, Youth, and Families AFDC
Aid to Families With Dependent Children BCRR Bureau's Common Reporting
Requirements BPHC Bureau of Primary Health Care CCDP Comprehensive Child
Development Program CHC Community Health Centers ESEA Elementary and
Seconds@ Education Act GPFU Government Performance and Results Act HHS
Department of Health and Human Services HMO Health maintenance organization
HRSA Health Resources and Services Administration NACHC NationaI Association
of Community Health Centers PHS Public Health Service
Page 7 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
I Chapter 1
Introduction Congressional committees need information to help them make
decisions about the programs they oversee- information that tells them
whether, and in what important respects, a program is working well or
poorly. Executive agencies provide eva+ ations, studies of the
implementation of programs, and other reports that can contain information
helpful to congressional committees. But authorizing committees may not
automatically receive such information about an existing program as it comes
up for reauthorization or oversight review and thus may receive insufficient
program information.
The then- Chairman of the Senate Committee on Labor and Human Resources
expressed concern that the Committee was not getting the evaluative
information it needs to carry out its oversight responsibilities and asked
us to ascertain what information would be most useful and how this
information could be obtained from the responsible program agencies.
Background This authorizing committee oversees programs administered by the
Departments of Education, Labor, and Health and Human Services. These
programs perform functions as diverse as funding education for disadvantaged
children, providing medical care, conducting basic medical research,
collecting data on American jobs, and enforcing occupational safety
standards.
The Committee has used a variety of strategies to request or obtain
information on these varied programs. For some programs, it has proposed
(and the Congress has adopted) legislation that identifies specific
questions to be answered in an evaluation study or annual report. For
others, legislation has mandated an evaluation study or report of
activities, but in very general terms. Another legislative strategy has been
to direct an agency to set aside some portion of its appropriation for
evaluation, yet permit the department to decide which programs to evaluate
as well as which questions to ask. The Committee also obtains program
information by asking questions as they arise or in connection with
oversight and appropriations processes.
The Committee can also draw information from reports that all agencies must
provide, such as appropriation request documents, the Chief Financial
Officers Act report, and the Federal Managers' Financial Integrity Act
report. In the future, the Congress will also receive reports on program
performance and outcome measures under the Government Performance and
Results Act. Yet despite these different ways of obtaining
Page 8 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 1 Introduction
information, the Committee finds that it does not necessarily get the
information it needs, when it needs it, in a usable form.
Objectives, Scope, and Methodology
Objectives and Scope The then- Chairman of the Senate Committee on Labor and
Human Resources asked us to help the Committee learn more about how to
evaluate the programs under its jurisdiction. F'irst, the Committee
requested that we examine the effectiveness of a specific evaluation
provision, one permitting a set- aside of program funds for evaluation of
public health programs. Our 1993 report found that these funds were not
fully utilized and that evaluation results were not synthesized or regularly
communicated to the Congress, suggesting that other approaches for getting
information to the Congress were needed. l Second, the Committee asked us to
identify information needed to evaluate the types of programs it oversees
and to suggest how it might request such information from the agencies.
This report addresses the second task. Based on the letter of request and
discussions with Committee staff, we identified three report objectives: (1)
to identify the kinds of information that would be useful to the Committee
for oversight and reauthorization review of the various types of programs
under its jurisdiction, (2) to examine the extent to which the agencies
currently collect and report such information, and (3) to propose a strategy
the Committee could use to improve its access to useful information from the
agencies.
Our objective was to help the Committee obtain agency information about the
performance of existing programs, to assist them with decisions such as how
to refocus or improve an existing program that meets an evident need. Of
course, decisions about a program's future may involve fundamental policy
questions- such as the question of whether program continuation is warranted
in light of social and demographic changes, current budget conditions, and
policy priorities. Information to resolve such questions comes to the
Committee from a wide range of sources, not primarily from the administering
agency. Since our concern was with how
%ee Public Health Service: Evaluation Set- Aside Has Not Realized Its
Potential to Inform the Congress (GAO/ PEMD- 93- 13; Apr. 8, 1993).
Page 9 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 1 Introduction
to obtain information from administrative agencies, we touch on the
fundamental policy questions only briefly. However, readers should bear in
mind that broad- based policy evaluation often takes place prior to or in
tandem with the evaluation of program performance.
Methodology To identify the kinds of information that would be most useful,
we (1) consulted the literature and experts familiar with the evaluation of
public programs of various types, (2) asked Committee staff members what
information they and Committee members would most like to have for oversight
and reauthorization, (3) reviewed statutory and other requests for
information about our case- study programs, and (4) compiled a list of core
questions drawing from all of the foregoing. In this way, we identified
categories of information that are useful for evaluation purposes as well as
for ongoing oversight activities.
To learn how much of this information agencies collect and report, we
narrowed our scope to conduct in- depth case studies of three programs
selected after consultation with the Committee. (These programs and the
selection criteria are described in more detail later in the report.) For
each program, we reviewed documents and conducted interviews with agency
staff to inventory what information was available to answer core questions
pertinent to that program. We also ascertained from Committee aud agency
staff what information the Congress had requested about each program, what
Committee and agency communication had occurred around these requests, and
what the Committee had received. We did not assess the quality of the
agencies' data.
To frame a strategy the Committee could use to request information, we drew
upon our observations concerning major gaps in the information the Committee
receives and shortcomings in communications between the Committee and
program agencies. Responsible officials from the Departments of Education
and HHS commented on our findings at exit conferences and provided written
comments on a draft of this report. (See appendixes IV and V.) We conducted
our work from July 1993 to October 1994 in accordance with generaLly
accepted government auditing standards.
Types of Programs the Our first objective required that we classify the
programs under the Committee Oversees Committee's jurisdiction by program
type, using a typology that had
relevance for evaluation. Our review of public policy and program Page 10
GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the Congress
Chapter 1 Introduction
evaluation literature found that a program's purpose is central to any
consideration of how to evaluate its effectiveness or worth. 2 Five major
program purposes- statistics, research, demonstration, service, and
regulation- appeared relevant to the work of the Committee. We describe the
five here as ideal types.
Statistics programs compile and analyze data and disseminate information,
typically on a recurring basis. They develop indicators from data collected
for administrative, enforcement, or statistical analysis purposes relevant
to public policy issues such as health care, employment, or education.
Programs of this type under the Committee's jurisdiction include the Bureau
of Labor Statistics and the National Center for Education statistics.
Research programs primarily aim to develop new knowledge or increase
understanding of the subject studied. Research may be conducted in- house by
agency personnel or by individuals, research institutions, or other
organizations supported by grants or contracts. Examples of research
programs are the National Institute on Drug Abuse and the Research and
Demonstration Center program of the Office of Educational Research and
Improvement.
Demonstration programs are defined here as those that aim to provide
evidence of the feasibility or effectiveness of a new approach or practice.
They may be undertaken to learn about the suitability of a particular
approach under real- world conditions, to try out a variety of approaches
and examine their effects, or to develop a new approach. 3 Examples are the
Jobstart demonstration program and the Comprehensive Child Development
Program.
Service programs provide services, directly or indirectly, to users or
clients. Not only are there many different kinds of service (human,
organizational, informational, fmancial), but they are delivered and funded
through a variety of mechanisms, making service the most varied of the
program types. Some programs deliver a specific service (such as student
loans for postsecondary education) while others (such as the Community
2Some public policy literature classifies programs sccording to the
“tool” through which they are carried out (for example, through
a government agency, project grant, formula grant, loan, loan guamntee, or
regulation) or by their maturity (whether the program is new or
establiihed). Program purpose is the basis for our typology, but the other
dimensions are discussed where relevant
3The term “demonstration” may also refer to activities whose
purpose is simply to provide an example of an approach In our view, such
purely illustrative demonstrations are most appropriately classified (for
these purposes) as service pro- since they mainly do not result in
evaluative conclusions about sn approach
Page 11 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 1 Introduction
Services Block Grant program) support a broad range of services that may
vary across providers or locations. Government agencies may make grants or
loans directly to recipient individuals or organizations, to service
providers, or to states (which in turn make subgrants to providers).
Regulatory programs are intended to protect people from harm or from a
violation of their rights by influencing the behavior of individuals or
organizations. Some regulations prescribe actions (such as requiring
employers to follow certain workplace safety practices) and others prohibit
actions (such as employer discrimination against employees on the basis of
age). Federal agencies (directly or through the states) adopt rules or
standards of behavior and compel or coax public compliance with the
regulation. Examples are the Occupational Safety and Health Administration
and the Department of Education's Office of Civil Rights programs.
We reviewed programs overseen by the Committee and found that we could
classify them according to these types.* To do so, we examined the programs'
actual functions, not simply the title or label given to a program by
enabling legislation. (For example, if a “demonstration” program
focused on providing services rather than on gathering evidence about what
was demonstrated, we treated it as a service program.) In addition, we
observed that some programs have a secondary purpose that complements their
primary purpose, such as a statistics program that is intended not only to
produce data, but also to perform the service of disseminating data to
clients. Such programs would need to be evaluated in terms of each relevant
purpose.
Case Study Programs To address our second objective and assess the extent to
which potentially Reviewed useful descriptive and evaluative information was
collected and reported
by federal agencies, we conducted case studies of three programs the
Committee oversees. The following three programs were selected, in
consultation with the Committee staff, from a list of programs due to be
reauthorized:
. the Comprehensive Child Development Program, which aims to demonstrate the
effectiveness of various approaches to providing very young children and
their families with educational, health, and social services;
40ur typology is comprehensive for a. ll programs under this Committee's
jurisdiction. It does not cover other possible program types, such as cash
assistance programs, entitlement programs (such as Social Security), or tax-
expenditure programs.
Page 12 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congresr!
Chapter 1 Introduction
l the Community Health Centers program, which funds centers that provide
clinical services to medically underserved areas or populations; and l the
Chapter 1 Elementary and Secondary Education Act program of grants
to states, which in turn fund school districts to provide compensatory
education services in schools serving low- income students.
The Chapter 1 ESEA program is administered through the Department of
Education, and the other two programs are administered through HHS. (For
program details, see appendixes I- III.)
This portfolio of one demonstration and two service programs covers only two
of our five program types. However, our three cases offer the following
advantages:
. All three areas- health, education, and early childhood development- are
high on the current policy agenda . Information about a demonstration
program and its evaluation is
particularly important, since a demonstration (such as CCDP) provides the
Congress with an opportunity to learn the effects of a proposed new program
on a small scale before committing to major implementation. l Service
programs constitute much of the Committee's workload and
funding authority. Our cases represent two major variants- one provides
federal grants to the states, which in turn fund local providers of services
(Chapter 1 ESEA), and the other provides grants from the federal government
directly to service providers (CHC).
Strengths and Limitations of Our Study
The major strength of our study is that, in identifying this Committee's
evaluation information needs, we considered the structure of the legislative
process, the nature of the programs, and the agencies' program
administration practices. Both the Congress and federal agencies could find
this perspective useful in developing both better information and a better
match between evaluative needs and the information provided.
Another strength of the study is our examination of the communication
process between this Committee and the three agencies it oversees, including
how that process affected what information was reported and when it was
received. Our report also provides detailed information on the three
programs, information for which there is both an immediate and a longer term
need.
Page 13 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Organization of This Report
Chapter 1 Introduction
A limitation of this report is that the three programs, while illustrative,
are not representative of all programs. The case studies exemplify the
importance of, and barriers to, gathering (or not getting) program
information but do not, of course, reflect all possible types of problems.
In addition, observations based on the study of a particular Senate
authorizing committee at a particular point in time may not reflect the
information needs and procedures of other Senate or House authorizing
committees or of appropriations committees. However, we are confident that
the general strategy we propose could be readily adapted to cover other
policy mechanisms (such as tax incentives) and evaluation questions (such as
whether there is still a demonstrable need for the program) that we did not
specifically address.
Chapter 2 describes the kinds of information we found to be of most
potential use to the Committee, as well as how the core questions to be
asked vary by program type. Chapter 3 summarizes the information available
for each of our three case programs and examines congressional requests for
information and how agencies responded to them. Chapter 4 proposes a
strategy the Committee staff could use to obtain useful evaluative
information in the future.
Appendixes I- III contain detailed information on the three programs
reviewed in chapter 3. Each appendix includes a brief description of the
program and a table listing evaluation questions, specific information
needed to answer those questions, and an assessment of whether the needed
information was available. Appendixes IV and V reproduce the comments from
the Departments of Education and HFIS on a draft of this report.
Page 14 GAOREMD- 95- 1 Improving the Flow of Program Information to the
Congresa
Chapter 2 What Information Would Be Most Useful?
Our lit- t task was to determine what kinds of information would be most
useful to the Committee as it evaluates various types of programs for
possible reauthorization and oversees their progress during interim years.
Our approach to this task included reviewing the literature on public
program evaluation and congressional oversight and interviewing experts
familiar with the evaluation questions associated with different types of
programs as well as congressional requests for information about such
programs1 We discussed the results of this inquiry and their own views with
senior members of the Committee staff with responsibilities in each of
several program areas and obtained their assessments of the kinds of
information they and Committee members would most like to receive.
From this approach, we identified priority categories of evaluative
information, compiled a list of core questions connected with each category,
and noted the kinds of data or other material that would be needed to
respond to these questions. We examined the relevance of the various core
questions to programs of different types. Finally, we identified different
forms of information that are useful for different purposes, such as
oversight, reauthorization, and policy change.
Our inquiry focused primarily on agency- provided information that could
help the Committee evaluate how well an existing program is working. Such
information is helpful in deciding whether a program is adequately serving
its intended purpose and if so, what level of funding is appropriate. Of
course, the purpose itself may need to be reevaluated: changing conditions
may have rendered it irrelevant or decreased the need for the program.
Because such reevaluations draw on information from outside the program
agency, they fall outside the scope of this report and we do not discuss
them specifically. However, guidelines to identifying the information needed
for such studies can be found in the general literature of evaluation and in
other GAO reports2 The general strategy we outline in chapter 4 should
assist the Committee in obtaining information concerning the need for a
program, as well as the other categories of information outlined in the
remainder of this chapter.
‘We interviewed program specialists from the four congressional
support agencies Congressional Research Service, Office of Technology
Assessment, Congressional Budget Office, and GAO.
2For a discussion of criteria for evaluating the need for a program, see
Children's Programs::‘ A Comparative Evaluation Framework and Five
Illustrations (GAO/ PEMlN? Z- 28BR; Aug. 341988).
Page 15 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 2 What Information Would Be Most Useful?
Categories of Information and Related Questions
The main congressional evaluation questions we derived can be broadly
categorized as seeking descriptive information about the program,
information that can be used to evaluate its implementation, and information
about various effects. These three broad categories of information are
generally applicable to all five of the program types we identified.
However, the relative priority of different categories and questions varied
somewhat from type to type as did the particular manner in which our core
questions were phrased. We describe our information categories and core
questions next.
Program Description Descriptive information tells what the program is and
what it does. Descriptive questions include the following: What activities
does the program support? Toward what end or purpose? Who performs these
activities? How extensive and costly are the activities, and whom do they
reach? Are conditions, activities, purposes, and clients fairly similar
throughout the program, or is there substantial variation across program
components, providers, or subgroups of clients?
The descriptive information that such questions elicit is not itself
evaluative and receives relatively little emphasis in the program evaluation
literature. It is important for evaluation, nonetheless, and was accorded
high priority by the Committee staff we interviewed. Descriptive narration
and statistics provide the foundation for identifying key evaluation
questions and interpreting the evaluative information concerning the
program. Better descriptive information would answer many of the Committee's
questions about the program and thus would enable the Committee to focus its
oversight and evaluation efforts more effectively.
Committee staff indicated that they typically do not get sufficient
descriptive information to understand the variety of conditions under which
a program actually operates and how federal funds are actually, being used.
(General summaries of activities and programwide totals of actions performed
or clients served, such as accompany appropriation requests, do not meet
this need.) Nor do they typically receive site- specific observational data
to convey the “flavor” of a program as participants see it (for
example, to help them understand how a student's experiences in federally
supported bilingual education classes relate to the rest of his or her
school day). Such information, they told us, is essential for understanding
how well a program is working and thus is relevant to the Committee even
when site- level conditions are a state or local responsibility.
Page 16 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 2 What Information WouM Be Most Useful?
Our review and interviews suggested that descriptive information is useful
for every type of program, be it concerned with statistics, research,
demonstration, service, or regulation. The nature and amount of information
needed vary with such program features as internal diversity and contact
with the public. For example, the typical summary description may provide a
sufficient basis for understanding a statistics program or a demonstration
that performs a uniform set of activities in a uniform way in . one or a few
sites. Much more information, appropriately disaggregated, is
needed to understand a demonstration, service, or regulatory program that
operates through diverse providers who perform different mixes of activities
under diverse conditions (for example, in schools, clinics, or workplaces
ranging from the very large and urban to the very small and IWd).
Implementation Our second category concerns information about program
implementation- specifically, about how and to what extent activities have
been implemented as intended and whether they are targeted to appropriate
populations or problems.
Extent and Nature of Implementation Questions about the extent and nature of
program implementation are
concerned with procedural issues- that is, with how program activities are
carried out. Relevant questions include the following: (1) Are mandated or
authorized activities actually being carried out? (2) Are the activities in
accordance with the purpose of the law and implementing regulations? (3) Do
they conform to the intended program model or to professional standards of
practice, if applicable? (4) Are program resources efficiently managed and
expended? 4
These implementation questions typically arise in the years immediately
after a new program or provision is authorized, as well as at
reauthorization. With respect to new provisions, our respondents were
interested in learning whether acceptable progress had been made in putting
new activities into practice, whether significant feasibility or management
problems had arisen, and whether program modifications were needed as a
result. Information obtained through the administering
3Thii and the preceding question refer to what is sometimes called
“program fidelity” in the evaluation literature.
4F'rogram evaluations and reauthorization discussions typically do not focus
on management issues unless there is evidence of problems with the program
in question. Our review accordingly did not focus on such issues or the
information needed to address them However, we consider it useful for the
Committee to ask whether there is evidence of such problems and, if so, to
request follow- up information.
Page 17 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
Congress
Chapter 2 What Information Would Be Most Useful?
agency's standard reporting and monitoring procedures may be sufficient to
answer these interim- year questions and to identify feasibility or
management problems that arise in connection with ongoing activities. For
reauthorization, interest focused on questions concerning conformance to
program purposes, model specifications, or professional standards.
Descriptive data obtained through routine reporting, if designed with this
purpose in mind and known to be sufficiently reliable, may provide a
sufficient basis for comparing program activities against program
requirements5 However, special data collection- obtained through program
monitoring or through external studies- may be necessary to ensure the
objectivity and detail required to explore issues of program quality.
Our review indicated that most of the implementation questions we have
discussed are pertinent to all of the types of programs we considered.
However, there are some differences with respect to the question of
conformance to professional standards. This question not only applies but
also is critical to the evaluation of two of the types of programs we
reviewed: statistics and research. Well- established professional standards
apply to these types of programs, and procedures and products that do not
meet such standards will not be considered credible. 6 Applying these
standards, an evaluation of the National Center for Education Statistics
should ask whether the education indicators generated were free from bias
and technically sound and whether data collection methods, analysis methods,
and limitations of the data were fully disclosed. Expert judgments commonly
are used to provide the needed evaluative information.
Obtaining information on the extent to which prescribed procedures were
followed is also especially important for demonstration projects. If a
prescribed approach was to be demonstrated, the Committee will want to know
whether the approach was implemented as planned and, if not, how
5Client counts and other monitoring data, dependent as they are on the
limited resources that program offices or grantees typically can devote to
data collection, may well be imprecise. The cost of achieving precision is
considerable, and often outweighs the benefits. In such instances, the
agency should make a systematic estimate of the extent of uncertainty in
their numbers and then present this estimate along with the numbers
themselves.
‘% 3iteria to be met by federal statistical agencies have been set
forth by the National Research Council. See Margaret E. Martin and Miron L
Straf (eds.), Principles and Practices for a Federal Statistical Agency
(Washington, D. C: National Academy Press, 1992). The Council's report
emphasizes that statistical agencies' work will have the necessary
credibility only if it ls impartial, tech& ally sound, and fully described.
Similar professional criteria apply to research.
Page 18 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Targeting Chapter 2
What Information Would Be Most Useful? it was modified and why modifications
were deemed necessary. Similarly, determining how activities and
arrangements differ across projects is necessary in order to assess those
demonstration programs designed to encourage and compare alternate
approaches.
For regulatory and service programs, the applicability of models or
standards and the importance of evaluating conformance to them must be
examined on a case- by- case basis. Some regulatory and service programs
emphasize the importance of following specific procedures (such as medical
procedures or procedures for gathering evidence of compliance with
regulatory requirements), while other programs involve activities for which
standards are not available or have not been specified. 7 (See our case
studies of service programs in chapter 3.)
Targeting has to do with the objects (problems or populations) at which a
program's activities are directed and the coverage that these activities
achieve. Many programs under the Committee's jurisdiction are targeted to
particular populations or problems (such as migrant students or unsound
management of pension plans). Others are directed to a broad area (such as
research on infectious diseases), from which the agency is expected to
select high- priority problems or clients (such as a particular disease that
endangers children or the elderly). In either case, it is useful for the .
Committee to know what kind of coverage a targeted program has achieved.
Relevant questions include the following: (1) Have program efforts focused
on appropriate problems? (2) Has the program reached the appropriate people
or organizations? (3) Do current resources and targeting practices leave
significant needs unmet (that is, priority problems unaddressed or priority
clients unserved)?
Assessing a program's targeting success typically requires not only good
measures of program activities and data on the clients served, but also
information about the size and distribution of the eligible population (for
target groups) or problem area as a whole. Thus, this assessment may require
a special data collection activity or access to sources of data outside the
program (such as the decennial census).
The question of whether appropriate problems are targeted is critical to
statistics programs (which should address policy- relevant issues) and
research programs (which should address important gaps in knowledge). For
other types of programs, the importance of targeting and the particular
‘Our focus here is on standards for conducting the service or
regulatory activities central to the purpose of the program. Federally
funded programs of every type are expected to meet financial and certain
other management standards.
Page 19 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 2 What Information Would Be Most Useful?
questions to be asked must be determined on a case- by- case basis.
Questions for programs of different types are illustrated below.
. Statistics: Are Bureau of Labor Statistics indicators relevant to current
issues of public policy? To what extent have they reached key federal,
state, and private sector policymakers? Are there important issues or
populations on which labor statistics are not being collected? * Service:
What proportion of the students served by the migrant education
program are currently migrant (that is, have moved within the past 5 years)?
What proportion of recent migrants are not served? 0 Regulatory: Are
Occupational Safety and Health Administration
enforcement efforts targeted to the safety problems that pose the greatest
dangers or that endanger the greatest number of people?
Program Effects We found it useful to distinguish three different aspects of
program effects: (1) whether the program is achieving its intended purposes
or outcomes (impact), (2) whether it has other important effects that relate
to congressional concerns (side effects), and (3) how it compares with
alternative strategies for achieving the same ends (comparative advantage).
We address each of these aspects below.
Impact Impact questions center on whether program activities actually
resulted in the improvements the program was designed to produce, as well as
on what progress was made toward achieving the program's goals. Where
programs or demonstrations are designed to produce changes as a result of
program activities, assessing a program's impact is often central to
reauthorization deliberations. For example, a job training program is
expected to show that participation in the program led to significantly
higher income or more stable employment. Where impact was expected, our
respondents were interested in learning, for reauthorization purposes, about
(1) the aggregate impact of the program; (2) how impact or outcomes varied
across participants and approaches; and (3) how impact varied across
providers- specifically, whether the program was supporting providers whose
performance was consistently weak.
Good evaluative information about program effects is difficult to obtain.
Each of the tasks involved- measuring outcomes, ensuring the consistency and
quality of data collected at various sites, establishing the causal
connection between outcomes and program activities, and separating out the
influence of extraneous factors- raises formidable technical or logistical
problems. Thus, evaluating program impact
Page 20 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
Congress
Chapter 2 What Information Would Be Most Useful?
generally requires a planned study and, frequently, considerable time and
expense. Program features affect the relative difficulty of getting good
impact information. The more varied the program activities, and the less
direct the connection between the provider and the federal agency, the
greater the likely difficulty of getting comparable, reliable data
concerning clients and services. For example, a federal agency whose own
employees deliver a specified service is likely to be able to obtain impact
data more easily than one that administers grants that states then pass on
to a variety of local entities to be used in any of several ways. Also, due
to the absence of a contrasting comparison group, it is practically
impossible to estimate the impact of a long& anding program that covers all
who are eligible.
It is critical to obtain impact information from demonstration programs
intended to test whether an approach (or any of several approaches) can
obtain results, as well as from regulatory or service programs that are
intended to produce specified kinds of outcomes or changes. However, impact
questions are not commonly directed at statistics programs (which are
designed to produce a product rather than a change). Nor are they directed
at programs that regulate or offer a service when it is considered certain
that the desired outcomes will occur as a result of program activities (for
example, that a vaccine, properly administered, will produce immunization or
that safety regulations, properly enforced, will reduce workplace injuries).
Information showing that activities have been soundly implemented and have
reached a high proportion of the relevant client population may be
sufficient to justify a decision to continue such programs.
The purpose of research programs is to produce new knowledge, either basic
or applied. The impact of basic research programs depends on whether the
projects, in the aggregate, have added to the knowledge in their fields by
eliminating or confirming hypotheses (for example, hypotheses concerning
brain chemistry or human perception and cognition). Applied research, on the
other hand, is expected to result in the development of new or improved
products (such as better medications or new approaches to the teaching of
reading). If it achieves its intended immediate impact, either type of
research program may ultimately have a much broader social impact (such as a
reduction in the incidence of alcoholism or a rise in educational
achievement levels). However, it would be inappropriate to evaluate either
type of program primarily in terms of such ultimate impacts. The influence
of basic research on either product development or practice is hard to trace
and may not be evident for many
Page 21 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 2 What Information Would Be Most Useful?
years, and products or practices may fail to gain widespread use for reasons
that are beyond their developers' control.*
Side Effects Public programs of any type may have important (and sometimes
unforeseen) effects beyond those they were intended to produce. These side
effects, which were of interest to our Committee respondents, should be
noted in any comprehensive evaluation. For example, a program might have
unforeseen effects- either positive or negative- on the problem or clients
it was designed to address. Or, a new program might have an effect on other
programs aimed at a similar problem or population.
Information to identify unforeseeable side effects is hard to plan for
except by encouraging reflective observation and maintaining open channels
of communication between program administrators and clients. However, some
kinds of side effects can be foreseen- indeed, possible side effects may
well come up in debate when a program is authorized- and attention to them
built into the program's data collection plans. For example, a program might
arrange from the outset to collect data that would enable it to determine
whether its procedures were unintentionally resulting in unequal treatment
of similarly situated individuals. Or, a service program that refers clients
to another program might arrange to monitor the level of demand placed on
the other program in order to ascertain whether meeting this demand was
impeding that program's ability to meet its own goals.
Comparative Advantage Finally, evaluation may focus on the comparative
advantage of continuing the current program. Typical comparative advantage
questions include the following: (1) are the effects gained through the
program worth its financial and other costs? and (2) taking both costs and
effects into account, is the current program superior to alternative
strategies for achieving the same goals? Options against which the current
program might be compared could include:
l discontinuing the program; 9 consolidating it with other federal programs
that serve a similar purpose; 0 utilizing a different type of program (such
as tax incentives rather than
regulation) to address that purpose; . transferring responsibility for
program decisions to the state or local level
of government (for example, by replacing a federal categorical service
program with block grants to support locally determined services); or
*For a comprehensive recent treatment of the subject, see Barry Bozeman and
Julia Melkers (eds.), Evaluating R& D Impacts: Methods and Practice (Boston:
Kluwer Academic, 1993).
Page 22 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 2 What Information Would Be Most Useful?
l transferring the function to the private sector. Comparative advantage
questions typically arise when programs are up for reauthorization. Our
Committee respondents indicated that such questions were of special interest
in the current climate of budget pressure (which may force choices) and of
major policy change (which may affect all programs in a part&& r policy
area).
Comparison of the current program to alternatives raises special challenges
for information collection. It requires not only good, comprehensive data on
the current program, but also truly comparable data concerning alternatives
or policy options. Simulation data or other forms of data based on
projection may be needed when considering options for which there is no
directly relevant experience. Because of their complexity, comparative
studies are likely to require considerable lead time, careful planning, and
special arrangements for implementation.
Different Forms ,of We observed from our interviews that program information
serves several Program Information different functions for the Committee-
functions that call for the receipt
of this information at different times and in different forms. The primary
and Why They Are functions of program information are to (1) signal the need
for attention to Needed a program in the interim years between
reauthorizations, (2) guide
reauthorization decisions, and (3) assist the Committeein deciding whether a
major policy change should be recommended. We refer to the corresponding
forms of information as (1) notification, (2) answers, and (3) evidence of
comparative advantage.
Notification That Attention Our Committee respondents emphasized that it was
very important that Is Needed they be kept informed of significant
developments on an ongoing basis and
particularly that they receive early notification when a problem or the need
for change in a program becomes evident (for example, notification that a
new provision is having unwelcome side effects or that shifting needs have
rendered program targeting obsolete). Such notification helps the Committee
focus its efforts on the programs that need attention, avoid unpleasant
surprises, and recognize successful program sites or practices.
Our interviews with Committee and agency staff revealed that developments
that may merit notification come to their attention in a number of ways,
often as a by- product of program management. Initial clues to such
developments can emerge from observations made during
Page 23 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 2 What Information Would Be Most Useful?
site visits, from the review of applications or of program performance
measures, from concerns that site officials raise in meetings, from
conversations with clients, or from media coverage of a local program. Since
a single clue may reflect erroneous information or represent an isolated
instance, other evidence is needed to verify the existence and importance of
the development suggested by the clue. The program managers we interviewed
could draw on a variety of resources- networks and data sources, as well as
their own expert knowledge of the program- to make such a verification and
to judge whether there was sufficient evidence to justify notice to the
Committee.
The varied evidence that supports managers' judgment that something worthy
of notice has occurred is not generally thought of as a form of evaluation
information. This may be because the evidence on which notice is based is
rarely sufficient to support evaluative conclusions about a program.
However, this form of evidence is sufficient to identify evaluation
questions to which the Committee will want answers in the future. We
therefore inquired about information and notification concerning current
developments in our case studies, and we report our findings in chapter 3.
Answers to Guide At times when major reauthorization decisions are being
made, the Reauthorization Decisions Committee needs answers concerning
evaluation questions such as those
just discussed. Answers arise out of systematic evaluation studies. These
studies must be planned and budgeted for in advance, information relevant to
the questions the Committee will face must be gathered, and findings must be
reported in time to inform deliberation.
Comparative Advantage Information for Policy Decisions
At times, the program under review concerns a policy area that is being
reexamined. In this event, the Committee may need information that evaluates
the current program against alternative strategies (including dropping the
program if it cannot be justified in terms of cost- effectiveness or need).
As previously noted, such comparative studies require data beyond that which
the administering agency can provide and raise complex analytic (as well as
policy) issues. Thus, these studies should be planned and arranged for well
in advance.
s= w Observations Our interviews and reading of the literature led us to
three conclusions
concerning what information would be useful, Page 24 GAO/ PEMD- 95- 1
Improving the Flow of Program Information to the Congress
Chapter 2 What Information Would Be Most Usefbl?
1. Descriptive information is more useful to the Committee than has
generally been realized. Such information should be sufficiently specific,
detailed, and disaggregated to convey an in- depth understanding of the
program.
2. In the interim years between reauthorizations, it is useful to the
Committee to obtain information on progress and problems in implementing new
provisions, as well as to be notified of developments that raise new
evaluation questions.
3. Reauthorization decisions (including those associated with major changes
of policy) call for information that provides answers to evaluation
questions concerning program implementation and effects and the comparative
advantage of continuing the program or pursuing other options. Advance
planning is generally needed to identify questions relevant to each program
and collect the data needed to answer them.
We further observed that the questions that it might be useful to ask,
expressed in general form, comprise a relatively short list. We have listed
these questions- grouped according to whether they concern program
description, implementation, targeting, impact, side effects, or comparative
advantage-- in table 2.1. For any given program, some of these questions
(but probably not all) will be relevant. Knowledge of program type is of
some help in narrowing the list, but features specific to the program (such
as expectations concerning targeting) must also be taken into account.
Page 25 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 2 what Information Would Be Most Useful?
Table 2.1: Core Questions by Category of Information Category of
information Core auestion Description Overall, what activities are
conducted? By whom? How extensive
and costly are the activities, and whom do they reach? If conditions,
activities, and purposes are not uniform throughout the program, in what
significant respects do they vary across croaram conaoonents, oroviders, or
subaroucs of clients?
implementation What progress has been made in implementing new provisions? a
Have feasibility or management problems become evident? a
Targeting If activities and products are expected to conform to professional
standards or to program specifications, have they done so? Have program
activities or products focused on appropriate issues or problems?
To what extent have they reached the appropriate people or organizations?
Do current targeting practices leave significant needs unmet (problems not
addressed, clients not reached)?
Impact Overall, has the program led to improvements consistent with its
purpose?
If impact has not been uniform, how has it varied across program components,
approaches, providers, or client subgroups?
Are there components or providers that consistently have failed to show an
imoact?
Side effects Comparative advantage
Have program activities had important positive or negative side effects,
either for program participants or outside the program? a
Is this program's strategy more effective in relation to its costs than
others that serve the same rourroose?
alnformation important for interim years. We used these core questions to
categorize and summarize the information that was available from the three
programs that we used as case studies, as reported in chapter 3 and in the
case study appendixes. In chapter 4, we suggest how the Committee might draw
from this list to request interim- year information and to frame evaluation
questions for reauthorization review of programs of different types and
features.
Page 26 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Chanter 3 What Information Is Available? Three Case Studies
Introduction The second objective of this study was to determine to what
extent information is actually available to answer oversight and
reauthorization
questions pertinent to various programs. To address this objective, we used
one demonstration program and two contrasting service programs as case
studies.
We identified core questions (described in chapter 2) pertinent to each
program based on our interviews and our review of enabling legislation and
congressional requests. Next, we reviewed all the information available on
the programs (from site reports and management information systems to formal
studies) to determine what information was available on core evaluation
questions (as adapted to fit each program), how much of this information
made its way to the Committee staff, and when and in what form it had
reached the staff. We also interviewed Committee and agency staff to learn
how adequately the Committee members and staff felt that they had been
informed, and what additional communication occurred as the agency sought to
interpret and respond to the Committee's requests.
Our three cases offer richly contrasting examples of information requested,
available and reported. The Comprehensive Child Development Program is a new
demonstration program whose design included explicit provision for the
collection of descriptive and evaluative information, Chapter 1 of the
Elementary and Secondary Education Act is a long- standing education service
program that collects minimal data from grantees but has been studied
extensively through several mandated evaluations, the most recent of them
reported to the Congress in 1993. The Community Health Centers program
maintains extensive data on program operations but has not been asked for,
nor has it provided the Committee with, comprehensive evaluation or program
reports for years. Taken together, the three cases offer considerable
insight into what information is collected and how it reaches (or fails to
reach) the Committee.
Demonstration Program: CCDP
Core Questions About CCDP
The Comprehensive Child Development Program, which originated in 1988, was
designed to demonstrate whether providing comprehensive and continuous
services to very young low- income children, their parents, and
Page 27 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available7 Three Case StUdieS
family members enhances the child's well- being as well as his or her
physical, emotional, and intellectual development. ' The various sites
funded through the program offer certain services in common- parent
education and training, case management, early childhood development
intervention, health care, child care, and family and child needs
assessment. However, sites vary considerably in the way they deliver these
services and in the specific objectives emphasized. (For example, some sites
focus on helping clients with day- today basic survival, while others also
provide preventive counseling.)
The descriptive, implementation, and effects questions for oversight and
reauthorization of the CCDP program are as follows: What is the basic model
or approach tested here? How are the centers different or similar in terms
of the local population, geographic location, and service provision? What
are some of the start- up and implementation difficulties the centers faced?
What are the principal program impacts? Have there been beneficial or
detrimental side effects to this program or any aspect of it? 2
Available Information Relevant to These Questions
On the whole, we found that CCDP was designed (through legislation and
administration) to collect the categories of information critical to the
evaluation of this demonstration program, and it has done so. The
Administration for Children, Youth, and Families (ACYF), which administers
the program, has descriptive and evaluative information to answer the core.
questions we identified.
Many sources provide the agency with useful information about the CCDP.
Between the program's management information system and observational
reports, the agency has the raw material from which to construct both
quantitative and qualitative descriptions of the CCDP approach as it is
demonstrated across the centers, as well as to measure CCDP'S impact and
document other effects. (See table I. 1 for more details on the data
available.)
Description ACYF has rich descriptive information on program operations,
clients, and settings for the approximately 34 centers. The agency's
information system includes standardized information from each center on
client needs as well as on what services are offered and how often they are
provided. Information is also collected on client and community
characteristics-
‘Although the enabling legislation did not refer to CCDP as a
demonstration program, the agency has collected implementation and impact
data to meet the information needs of a demonstration program
%ee appendix I for more information on the program and what specific
information is needed. Page 28 GAO/ PEMD- 96- 1 Improving the Flow of
Program Information to the Congress
Chapter 3 What Information Is Available? Three Case StlldieS
whether the center is located in a rural or urban area and what the center's
philosophy and staffing patterns are- through center- written progress
reports, program- staff site visits, and ethnographers' reports. ACYF thus
has information that can communicate a “feel” for the challenges
providers face in delivering a comprehensive set of services and how clients
see the program as affecting their lives.
Implementation We found much useful information on CCDP implementation
processes and feasibility issues, as well as evaluative information on
whether the program is being implemented as intended. ACYF has qualitative
and quantitative information that is useful for both compliance assessment
and service improvement. The ethnographers' reports from the first year of
the program included specific information on stat+ up problems. They also
described how site officials completed such necessary steps as setting up
interagency agreements for service delivery and how grantees' approaches to
service delivery were adapted as programs took shape. The grant application,
as well as the ethnographer, site visit, and progress reports, provided
additional insights into planned implementation.
Impact The agency has data on participating children's developmental
progress based on the results of standardized tests, as well as on whether
parents gained parenting skills. Measures of short- term changes in parental
income and employment status, as well as of general parental well- being and
level of education, are also available. The agency has gathered similar
information on a nonparticipating control group of families to compare with
clients to determine whether any observed changes in the program families
were a consequence of the demonstration program. Anecdotal information from
participating family members (in progress or ethnographer reports) about how
the program has improved their lives has also been collected. These data
should be sufficient to support conclusions concerning the program's impact
on low- income families and children.
Side Effects and Comparative Advantage The agency has diverse sources of
information that would capture CCDP'S
side effects (intended or not) on the local community and its social
services agencies. CCDP evahmtion studies have focused on this program alone
and were not designed to compare CCDP with similar or different programs. A
study comparing programs or models would need CCDP program data as well as
information on other programs, and it would need to be arranged for
separately. In addition, cross- site comparisons of effectiveness may be
difficult to make since CCDP centers differ in many ways. If the Committee
wants a comparison of the CCDP approach with
Page 29 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case StUdieS
I another, or an assessment of which CCDP features are most needed, it
should request such a study.
What the Congress Requested and What It Received
Enabling legislation for the CCDP mandated a study that was to report on
results of the program by October 1,1993. The legislation set out evaluation
objectives for a study (describe service delivery mechanisms and assess
project impacts), specified that project impacts be measured with control
group comparisons, and provided for an evaluation of program successes, but
allowed the agency to select its evaluation design.
In response to this mandate, the agency provided the Committee with a report
on program impacts on families and communities, an analysis of program
implementation, and a description of families and projects3 It reported
short- term effects on children, families, and mothers in terms of their
education, income, health, and parenting shills. Because it was due before
the demonstration was complete, the report provided only interim findings.
This mandated report had both strengths and limitations. It provided needed
outcome conclusions, useful feasibility information, and a separate,
readable executive summary. Programwide impact results (and answers to
feasibility and process evaluation questions) were reported succinctly and
clearly, but disaggregated statistics would have shown the Committee the
range of performance and reflected possibly important different features of
the CCDP model. The feasibility information would have been useful during
the deliberations concerning reauthorization of the Head Start Program and
its inclusion of ccrAike services, but the report came out in May 1994 just
as final action on the legislation was occurring.
In the interim, the Committee obtained program information through three
other means (apart from normal ad hoc information requests). In December
1991, the program agency sent the Committee copies of a report it had
prepared to inform the public and other agencies about CCDP. Although
intended for another audience, the report contained information on program
operations and differences among sites and also evaluative information on
program feasibility of use to the Committee in its CCDP oversight and Head
Start reauthorization activities.
sACYF', “Comprehensive Child Development Program-- A National Family
Support Demonstration,” interim report to the Congress, May 6,1994.
Page 30 GAO/ PEMD- 951 Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case Stlldi~
In addition, the agency provided information on CCDP to a special commission
(consisting of congressional committee staff- including representatives from
this Committee- as well as agency officials and outside experts) formed to
make recommendations concerning the Head Start Program. The CCDP program
officer provided information to assist the commission in understanding the
possible effects of extending the Head Start Program to include ccm4ike
services to very young children. Finally, a member of the Committee staff
requested and obtained current data on program impact- data more recent than
the figures to be summarized in the CCDP interim report- from child
development center directors.
ACYF plans to provide a final report to the Congress in 1996. It will be
based on the entire 5 years of experience with the program and is expected
to provide an assessment of the overall impact of CCDP.
Service Programs: CHC and Chapter 1 ESEA
Core Questions About The first of our service programs, the CHC program,
funds medical centers These Programs to provide outpatient care (and
supporting services such as patient
transportation) to populations that otherwise would lack access to such
health care. Some community health centers serve remote rural areas that
lack medical facilities, and others serve urban populations that do not have
access to care through other providers. (For a description of the CHC
program, see appendix II.) The Chapter 1 ESEA program provides financial
assistance (through grants to states) to local education agencies to meet
the special needs of educationally deprived children who live in areas with
high concentrations of low- income families. Student demographics,
educational achievement levels, and the services delivered vary considerably
from school to school. For example, some schools assist
I low- performing students within the regular classroom, while others pull
these students out to a special setting for assistance. (For a description
of
! I the Chapter 1 ESEA program, see appendix III.) Table 3.1 summarizes the
core questions adapted to fit the type and nature of each program. The two
programs have certain questions in common. Both operate in diverse settings,
so it is important to ask how services and conditions vary as welI as to
secure a good summary description. Both are
Page 31 GAO/ PEMD- 951 Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case StUdieS
targeted to needy populations, so it is relevant to ask what proportion of
eligible clients were served and whether significant numbers of eligible
clients remained unserved. (For Chapter 1 ESEX, attention has focused
particularly on eligible students in the neediest schools- that is, schools
in which a high percentage of students come from low- income families.)
Table 3.1: Core Questions Adapted to CHC and Chapter 1 ESEA Programs
Category of
information Questions for CHC Questions for Chapter 1 ESEA Description
implementation Targeting
Overall, what services are Overall, what services are provided and to whom?
How do provided and to whom? How do services, center locations, and services
and school and student mix of clients vary from site to characteristics vary
from site to site? site?
Do clinical practices meet professional quality standards?
What fraction of those who What fraction of the neediest would otherwise
lack access to schools and students care does the CHC program participate?
How does this serve? How does this fraction fraction vary across sites? How
vary across grantees? What many sites lack services? significant areas,
health problems, or populations are not covered?
Impact Has the long- term health status In the aggregate, has of CHC
patients improved as a disadvantaged students' result of program services?
achievement improved? Has
individual participants' achievement'? How does impact differ across
approaches and across schools? Do some schools or approaches consistently
fail to produce gains?
Side effects Comparative advantage
What are the effects on other Does the program fragment health care
providers (if any) in responsibility for ensuring that the CHC communities?
students' progress improves?
What are the comparative advantages of shifting to schoolwide, rather than
individually targeted, aooroaches?
Beyond the common ground delineated in table 3.1, the two programs are
dissimilar. For the CHC program, the question of whether the clinicaI
services provided meet current standards of good medical practice is
fundamental. In terms of program effect, the simple provision of quality
clinical services is an important program outcome. The broader medical
Page 32 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case Studies
research question of whether these services influence a patient's health
status in the long run is pertinent to any health care service system (not
just this program) and thus needs to be answered in a broader context. The
Chapter 1 ESEA program, by contrast, focuses on questions of impact, such as
whether the provided services produced increases in student achievement.
Defining and evaluating the quality of implementation of Chapter 1 ESEA
services is a state and local responsibility. However, the Congress has an
interest in learning whether some of the general . approaches used are more
advantageous than others.
Available Information Relevant to These Questions
The agencies that administer the service programs we reviewed collect a
considerable amount of information potentially relevant to the foregoing
questions. HHS' Health Resources and Services Administration (HRSA) receives
narrative information in center applications; collects data on CHC program
services, costs, and clients; and maintains records of quality reviews. When
drawn together, this information could address many of the questions we have
identified. The numerous evaluation studies concerning the Chapter 1 ESEA
program similarly include information that covers many of our questions.
Moreover, HRSA and the Department of Education's Office of Compensatory
Education receive additional useful information- including clues to emerging
concerns- through various program management activities as well as through
active networking with service providers. In the sections below, we
summarize what is available with respect to each of the information
categories and questions listed in table 3.1.
Description Both of these programs have gathered information from grantees
and have used this information to provide aggregate descriptions and
statistics, rather than to describe the range of variation to be found
within the program. For Chapter 1 ESEA, data on clients, services, and
student achievement are initially collected at the school or center level
but are subsequently aggregated and reported at the state and national
levels. CHC reports data only for the program as a whole. HR, SA could piece
together information from site visit reports and from its routine project
management activities to describe for the Committee how community health
centers and patients' experiences within them differ. Descriptive
information of this kind is available from school case studies conducted in
connection with the recent Chapter 1 ESEA evaluation. 4 The summary report
of the evaluation included vignettes drawn from these studies and
%am Stringfield et al., Urban and SuburbanRural Special Strategies for
Educating Disadvantaged Students- Fht Year Report (Washington, D. C.: U. S.
Department of Education, Office of the Under secretary, 1994).
Page 33 GAOREMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case Studies
also presented data that was disaggregated to describe differences between
high- poverty and low- poverty schools. 6
Implementation Information about how the CHC clinics implement recommended
medical procedures is based on site visits, HRSA'S quality review, and
federal and regional officials' routine program monitoring. As mentioned
previously, Chapter 1 ESEA has considered the quality of instructional
practices to be a state and district responsibility, Federal program
monitoring thus has not included the gathering of data on instructional
quality. However, there are plans to make observation of instructional
practices a larger part of federal monitoring in the future.
Targeting The CHC program generates data about the numbers of clients served
in its centers, and HRSA could develop population estimates to calculate the
proportion of the targeted population that is served by each center and by
the program as a whole in areas already designated as medically underserved.
HRSA has relied on applicants to make known unmet needs in areas not
currently included. (Community groups or local health organizations often
apply for a designation of need.) In the past, it has not initiated its own
assessment of medical need for areas from which there has been no
application. However, based on Bureau of Primary Health Care (BPHC)
estimates of areas with high priority for federal intervention, HRSA could
determine the areas or people with the greatest general health care need and
estimate the amount of unmet need in areas not now covered by CHCS. 6
The 1993 Chapter 1 ESEA evaluation included a special study that examined
the degree to which program funds were reaching the schools with the
greatest concentrations of disadvantaged students, as well as the number of
such schools that remain unserved. The question of whether funds are going
to the most educationally needy students cannot be answered from grantee
reports, but a study currently being conducted (the “Prospects”
study) will shed light on this issue. (The final report from that study is
due in January 1997.)
Ykhools in which 75 percent or more of the students came from low- income
families were designated high- poverty schools; low- poverty schools were
those in which 0 to 19 percent of the students came from such families.
%RSA is developing a proactive method of estimating the amount of need for
primary care that does not restrict the focus to currently designated ~IW.
S. Some recent data on the amount of unmet medically underserved needs are
available from the National Association of Community Health Centers.
Page 34 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case Studies
Impact Side Effects and Comparative Advantage
As noted previously, the primary purpose of the CHC program is to provide
medically underserved people with access to medical services. Program data
have focused on numbers served and service quality. 7 ‘I'he express
purpose of Chapter 1 ESEA, however, is to improve student achievement.
Traditionally, two sources of information have been used to evaluate program
impact. Results from the National Assessment of Educational Progress-- which
periodically tests a national sample of 4th, 8th, and 12th grade students in
reading, writing, and mathematics- have been used to gauge whether
disadvantaged students' achievement is improving overall. The program has
also required states to report year- to- year gains in participating
students' scores (aggregated to the state level) on nationally normed
standardized tests. .
These gain- score data give some idea of whether students in the aggregate
are making progress, but they do not show the extent to which individual
participants are benefiting from the program to the extent that they no
longer need compensatory services. Nor do these aggregated data illuminate
variations in impact across schools or subgroups of students. Attempts to
use gain scores to identify schools that consistently fail to get results
have been problematic, in part because of inconsistency in reporting and
problems of data reliability.
In the future, the aforementioned “Prospects” study will fill in
some of the gaps left by traditional information sources. That study will
provide more differentiated information on services received and on
achievement gains of Chapter 1 ESEA participants in comparison with other
students. As of July 1,1995, states will no longer be required to report
nationally normed test score data, Rather, each state will assess student
progress and adequacy of school performance in terms of its own standards
for content coverage and student performance, using achievement tests and
other instruments aligned to those standards. lt is not yet clear what
information will be reported to the federal program office.
An issue for Chapter 1 ESEA has been whether providing compensatory
education services through separate teachers or in separate settings has had
undesirable side effects, such as impeding coordination between regular and
compensatory instruction or encouraging classroom teachers to disclaim
responsibility for Chapter 1 students' progress. While the program has data
on how many schools use “pull- out” and other approaches and how
much time students spend in each, these data do not
%FBA collects some health outcome data and has recently contracted for an
impact study of the CHC program. It has also supported efforts to develop
health outcome measures pertinent to any primary care program.
Page 35 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case StUdieS
illustrate how commonly used versions of each approach actually operate in
relation to the regular classroom, how quality varies within each, and how
variations in quality and approach are linked to student outcomes, The
program has only begun to gather information at the level of detail needed
to shed light on these questions and to compare the advantages of various
models of delivering compensatory educational services. For the CHC program,
information is available on program side effects. The comparative advantage
of the CHC program over other programs has not been an evaluation issue,
although it may be in the future.
What the Congress Requested and What It Received
The Congress did not mandate an evaluation study or reports on the CHC
prbgram, It did encourage evaluation of programs under the Public Health
Service, which would include CHC, through legislation that set aside funds
for that purpose. However, the set- aside provision did not require the
agency to synthesize information and report it to the Congress. Reports have
been provided when specificzilly requested, but this approach has left much
useful information unreported, as the CHC case illustrates. 8
The Committee informally requested studies and reports about particular
aspects of the CHC program (such as the costs and effects of the
Comprehensive Perinatal Care Program, quality of care, and capital
improvement needs), and the agency provided them. In the absence of a
request for a summary evaluation of the program, however, the bulk of the
agency's information resources remained untapped. The only overall report
the Committee received on the CHC program was the annual appropriations
request- which stated the program's purpose, noted some of the services
offered and the number of users, and reported previous program funding
levels. However, this annual request, in keeping with the standard format
for such requests (which are focused on appropriations issues), described
the program in very broad terms. Committee staff commented that such
general, highly aggregated information was not particularly useful for
overseeing or evaluating the CHC program.
The Chapter 1 ESEA program offers a contrasting case in which the Committee
has requested and received a great deal of information, both in interim
years and for reauthorization. Soon after maor changes were made to the
program in 1988, congressional and agency officials became
sFor wonnation on the use of the PHS set- aside, see our earlier report
entitled Public Health Service: Evaluation Set- Aside Has Not Realiied Its
Potential to Inform the Congress, GAO/ PEMD- 93- 13 (Washington, D. C.: Apr.
8,1993). The Preventive He& h Amendments of 1993 (P. L. 103- 183) amended
the Public Health Service Act to establish a requirement for annual
reporting of findings of evaluations conducted under the set- aside.
Page 36 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case StUdie. 9
aware of implementation problems with the “maintenance of
effort” provision for schoolwide programs? The two sides communicated,
and the problems were remedied. (We were told that this kind of interim-
year problem identification and correction are common.)
The Congress mandated two major evaluation efforts for Chapter 1 ESEA: (1) a
longitudinal study of a national sample of students (the
“Prospects” study) and (2) an evaluation of the implementation
and the effects of changes made in 1988. It also established a special panel
to examine the use of standardized tests to assess student achievement.
In the course of its reauthorization deliberations in 1993 and 1994, the
Committee received reports from the longitudinal study, the assessment
study, and the numerous special studies that comprised the mandated
evaluation. There was also a summary report that drew together findings and
derived recommendations from all of these efforts. The plethora of study
reports provided more information than the Committee could comfortably
process. However, the summary report- which was organized around policy
questions and provided easy- to- find summaries of main points,
disaggregated data, and illustrative vignettes on case study school- was
viewed as useful.
The Congress also requested, and the Committee has received, a summary of
Chapter 1 ESEA program participation and achievement data for each year-
unfortunately, not until a year or more after the end of the school year for
which data are collected. lO General descriptive material and summary data
are included in appropriation requests and in the Department of Education's
annual summary of evaluations.
Communicating About In each of the cases we studied, we observed very
limited communication Information Needs between the parties concerning the
Committee's questions and how the
agency might best respond to them. Ad hoc inquiries aside, congressional
requests for evaluative information took the form of statutory mandates.
These mandates were quite specific, setting forth design details and report
dates as well as objectives for the evaluation. The specifics were worked
out in Committee deliberations with relatively little input from agency
staff. In one of our cases, agency officials reviewed and suggested changes
sThis provision concerns expenditures per pupil under a schoolwide program
compared to expenditures in the same schools in the previous fmcal year.
‘Drhe agency has established new procedures and, ss a result, expects
to make data available more promptly in the future.
Page 37 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case Studies
in an early draft that was under House committee review, but they did not
discuss their concerns with committee representatives.
Once the evaluation mandates became law, agency officials in the Department
of Education and HHS took them as a given. In preparing to implement the
mandates, they went to considerable effort to interpret the interests and
concerns expressed in the written record, but they did not confirm their
interpretation with the Committee. Education Department officials did seek
to obtain a sense of the relative priority of the questions listed and to
discuss what they judged to be unworkable deadlines or design
specifications, but Committee staff informed us that such behavior was the
exception.” (In another case, the program manager did not see
discussion of concerns with the Committee as a realistic option.) We found
that, for the most part, agency officials simply did their best to do what
was mandated.
Interchange was similarly limited as evaluations were conducted and
reported. For example, Committee staff attended some of the meetings of the
advisory group to the Chapter 1 ESEA evaluation, but as observers rather
than as spokespersons for the Committee's information needs. Department of
Education officials briefed the Committee on the forthcoming report, but
Committee staff perceived that these briefings were designed to inform the
Committee of what the Department had done rather than as an occasion for
dialogue about priorities or about the best way to present the information.
Finally, we observed that the Committee rarely provided feedback to the
agency concerning the reports it received. Committee staff did recall
telling Department of Education officials that the clear organization and
policy focus of the Chapter 1 report had been helpful. But they indicated
that the Committee was unlikely to provide feedback on reports that were
unsatisfactory- that is, that either failed to provide relevant information
or buried it in an undigested mass of facts and observations.
The literature on public policy and the legislative process suggests that
the communication patterns we observed are to be expected. The executive and
legislative branches of government have different constituencies,
institutional perspectives, and interests. Evaluative information often has
policy implications that could have important consequences for the agency
and department.
IThe National Assessment of Chapter 1 was mandated in May 1990, with a final
report due at the end of 1992. Meeting this schedule posed major challenges
for the Department.
Page 38 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case Studies
In view, of their importance, policy- sensitive communications with the
Congress are channeled through central congressional liaison offices, and
contacts with committees are subject to high- level policy review and
clearance. Discussions with our Committee respondents and agency officials,
together with. our review of agency policy, revealed that the liaison
officer's chief role is to facilitate the exchange of information as
legislation is drafted and enacted. Liaison officers also pass along ad hoc
committee requests for information and arrange the briefings that precede
the issuance of an evaluation study report. But they typically do not bring
agency and committee staff together for evaluation planning.
Concluding Observations
Each of the three programs we studied collected a great deal of information
relevant to questions of interest to the Committee. True, there were gaps in
the information, and some useful questions could not be answered from
available sources. However, we observed that lack of information does not
appear to be the main problem. Rather, the problem seems to be that
available information is not organized and communicated effectively. Much of
the available information did not reach the Committee, or reached it in a
form that was too highly aggregated to be useful or that was difficult to
digest.
Second, we observed that the Committee was given no more than it asked for.
It asked for no summary evaluation report on the CHC program, and it
received none- either in interim years or at reauthorization. It mandated
studies on a host of implementation and evaluation questions concerning the
Chapter 1 ESEA program, and it received a report on every one. It mandated a
report evaluating the CCDP, and it received such a report- although the
report results could only be based on information from a few years of
experience.
However, receiving what it initially asked for did not always meet the
Congress' current needs, since these evolved over time (whereas statutory
mandates remained fixed). For example, the Chapter 1 ESEA mandate called for
a study of the implementation of services to children in private schools,
but implementation was no longer a major issue by the time the report was
due. The reauthorization and consideration of expanding Head Start services
provided another example: it created a need for an earlier- than- scheduled
report on emerging findings (on feasibility and implementation issues, in
particular) from the CCDP program 0ffice. l'
%ome of the emerging findings on implementation drawn from the report could
have been fruitfully shared in a letter or briefing to the Committee while
report editing and clearance proceeded as usual.
Page 39 GAO/ PEMD- 95- 1 improving the Flow of Program Information to the
Congress
Chapter 3 What Information Is Available? Three Case StUdieS
F'inally, we observed that communication between the Committee and agency
staff knowledgeable about program information was limited and comprised a
series of one- way communications (from the Committee to the agency or the
reverse) rather than joint discussion. This pattern of communication, which
was reinforced by departmental arrangements for congressional liaison,
affords little opportunity to build a shared understanding about the
Committee's needs and how to meet them.
Taking into consideration what we found and reported in chapter 2, our case
studies led us to conclude that obtaining timely and useful information for
oversight and reauthorization requires not only knowing what questions to
ask, but also ensuring that the information is in fact requested when needed
and communicating with the agency to establish a mutual understanding of
information needs and how they can be met. In chapter 4, we propose a
strategy that incorporates each of these features.
Page 40 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
Our final task for this report was to propose a strategy that the Committee
might use to request information- one that would take into account the
various factors that have contributed to unsatisfactory results in the past.
The Committee's request letter recognized one such factor: the difficulty of
knowing what questions to ask about programs of various types. Our strategy
addresses that factor as well as two others that came to light through our
case studies: agencies' disposition to provide information only when they
have been asked to do so, and insufficient communication between the
Committee and the agency with respect to information needs. We thus propose
a strategy that attends not only to the content of information requests, but
also to communication practices that can help ensure satisfactory responses
to those requests.
The strategy we suggest for the Committee's consideration includes the
following three components:
l selecting and adapting, from a core list, the descriptive and evaluative
questions to be asked about a program in interim years and at
reauthorization; . arranging explicitly to obtain timely oversight
information in interim years
as well as to receive results of evaluation studies at reauthorization; and
l providing for increased communication with agency program and
evaluation staff to help ensure that information needs are understood and
that requests and reports are suitably framed and are adapted as needs
evolve.
Identifying and We begin by proposing guidance to assist the Committee in
identifying the Adapting Core questions to ask concerning a given program.
In chapter 2, we outlined a
set of core questions that might be asked about programs of various types.
Questions In the section that follows, we suggest how the Committee, drawing
on its
sense of the policy issues and of the history of a particular program, can
select the core questions most important to that program, restate them in
program- specific terms, and outline the forms of information needed to
answer them. The Committee may elect to select and state the questions
initially (as in the process of framing the questions for a mandated
evaluation) and then request comments from the agency. Or, it may ask the
agency to perform this initial step and to conlirm the questions with the
Committee.
I In either case, we suggest that the Committee begin by reviewing the
particular program's purpose and history, current policy issues regarding
Page 41 GAOREMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
this program, and the policy context. This initial review will help to
determine which of the core questions listed in table 2.1 apply, what
additional questions are of interest, and whether a full evaluation is
needed. (It would probably be useful to ask the agencies to review the list
also, as a first step in building a common framework for thinking about
evaluation information.) For example, if policy review determines that the
program's function could now be performed at the state level or that its
dwindling population could be covered under a similar federal program, there
might be little reason to seek additional agency information about program
implementation or impact.
As noted in chapter 2, knowledge of a program's type or purpose is helpful
in identifying the categories of information that are likely to be most
critical to its evaluation. For example, the evaluation of demonstration
programs (other than those that merely seek to provide an example) generally
involves questions concerning implementation and impact, Statistics and
research programs virtually always involve questions of implementation in
accordance with professional standards and targeting to relevant problems.
Beyond this, programs must be reviewed on a case- by- case basis to
determine which core questions are most pertinent. For example, the
Committee or agency might ask itself the following questions:
0 Do conditions within this program vary so substantially that the Committee
will need disaggregated descriptive information? 0 Are there new program
aspects or provisions that are so significant that
the Committee will want information on their implementation in interim
years? 0 Is targeting a critical issue for this program? If so, which of the
targeting
questions apply? 0 Have possible side effects been identified that warrant a
specific question?
In most cases, the Committee or the agency should restate each relevant core
question so that it fits the program- for example, to identify the
particular target population as “recently migrant students”
rather than as “the appropriate people.” The case studies
reported in chapter 3 and in the appendixes provide examples of such
restatements.
The Committee should also consider what form of information it needs in
response to each question, and When and for what purpose (oversight or
reauthorization) the information is needed. This is not to say that the
Page 42 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
Committee should spell out in detail the specific data to be submitted.
Rather, it should let the agency know its intent with regard to such
questions as the following:
Is the information needed for the next reauthorization or at an interim date
before that time? Is an empirically precise answer- and hence time and
resources to support a planned study- needed?
Examining its information needs in this way will help the Committee focus on
the central issues, express its intent clearly, and avoid elaborate and
expensive data collection when an estimate based on existing evidence would
do as well. (For example, rather than developing an exact count of the
number of sites that have implemented a new provision, program managers
might estimate based on recent monitoring visits that about two thirds of
the sites had done so.) This outline will form the basis for framing
information requests that make the Committee's objectives clear, while
avoiding detailed specifications that may prove to be unworkable. (Under our
strategy, the agency is responsible for working out the details but must
communicate with the Committee as it does so to ensure that the resulting
plan meets the Committee's needs.)
To assist the Committee in creating such an outline, we have drawn on our
observations from this study to create a guidance table (table 4.1). This
table contains a row for each of our categories of information: description,
implementation, targeting, impact, side effects, and comparative advantage.
In addition, column one of the table lists our core questions in abbreviated
form, column two notes the kinds of information that may be needed for each
question, and column three notes the types of programs to which the question
is likely to apply.
Page 43 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
Table 4.1: Guide to Identifying Needed information Information needed to
Question respond Applicability Description Overall, what activities are
Narratives and summary All programs conducted? By whom? How data that
describe the extensive and costly are the scope and extent of the
activities, and whom do they program reach?
If conditions, activities, and Narratives and Programs that feature purposes
are not uniform disaggregated data that varied activities, clients,
throughout the program, in show the extent of variation and settings what
significant respects do within the program they vary across program
components, providers, or subgroups of clients?
Implementation What progress has been made in implementing new provisions? a
Notification of early All demonstration programs; difficulties; descriptive
data other types of programs on the extent of when new or when
implementation at specified important new features are intervals; answers
introduced concernina feasibilitv issues
Have feasibility or Same as above Any program management problems become
evident? a
If activities and products are Answers concerning All statistics and
research expected to conform to whether and to what extent programs; other
types of professional standards or to criteria have been met; programs when
criteria are program specifications, have notification of evidence that
specified they done so? standards or criteria may be
inappropriate or outdated Targeting Have program activities or Description
of the issues or Statistics, research, service, products focused on problems
targeted and and regulatory programs appropriate issues or evidence that
supports their problems? selection; if different
components of the program target different problems, report them separately
To what extent have they reached the appropriate people or organizations?
Answers showing how Demonstration, service, and many of each relevant
regulatory programs; category of person or statistics and research
organization were reached; programs whose objectives if targeting differs by
site or include increasing the by program component, participation of or
report separately for each disseminating results to
certain populations (continued)
Page 44 GAO/ PEMD- 95- 1 Improving the Flow of Pronram Infcwmatinn t. n the
Cnnano~
A Sirategy for Requesting Evaluation Information Chauter 4
Question Information needed to resoond Aoolicabilitv . . . Do current
targeting Answers that show the Service or regulatory practices leave
significant nature and extent of critical programs intended to cover needs
unmet (problems not unmet needs a specified population fully; addressed,
clients not research and statistics reached)? programs intended to cover
a problem area fully . Overall, has the program led Answers concerning All
demonstrations; to improvements consistent changes in outcomes and research,
regulatory, and with its purpose? how they are related to service programs
that are
program activities intended to produce specified results
If impact has not been Answers concerning Demonstration, service, and
uniform, how has it varied variations in conditions and regulatory programs
that across program components, program impact under each deal with varied
clients or approaches, providers, or condition varied conditions of client
subgroups? implementation
Are there components or Answers concerning Same as above providers that
consistently variation in impact across have failed to show an providers or
components impact? and extent of turnover in the
low- performance category Side effects Have program activities had
Notification of possible Any program important positive or negative effects;
answers if evidence side effects, either for shows need for further program
participants or investigation outside the program?=
Is this program's strategy Comparative analysis of Any program when more
effective in relation to effects of this program and alternatives and
evidence its costs than others that of alternative approaches in concerning
them are serve the same purpose?, relation to costs
%formation important for interim years. available -
Arranging for Responses
Once questions have been identified, the Committee will need to arrange
explicitly to obtain responses to them. Based on our observations, we
suggest that the Committee set up procedures for reporting in interim years
as well as arranging for program or comparative evaluations to be conducted
in connection with reauthorization and policy review decisions.
Page 46 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
Soliciting Interim- Year Information
Currently, there is no standard mechanism for prompting agencies to report
information the Committee would find useful for oversight in interim years-
that is, time- sensitive information that would keep the Committee informed
of progress in implementing new provisions and of significant new evidence
concerning program activities and effects. (As noted previously, neither
budget submissions nor annual reports- as currently configured- are designed
to serve this purpose.)
We propose that the Committee initiate a mechanism for prompting agencies
(or a subset of agencies the Committee selects) to respond to program
oversight questions each year. Agencies already prepare responses to
questions about their budget requests each year, and they could provide
responses to the authorizing committee on a similar schedule.
In the interest of evoking responses that are brief and up- todate, we
suggest that the Committee express its request for interim- year information
through a letter (transmitted to the program agency through established
departmental channels) rather than more formally through legislation. 1 The
annual letter could include questions tailored to a particular program (such
as progress in remedying a problem that was identified in a previous
evaluation or oversight review). However, we believe that many of our core
questions pertinent to interim years (for example, “what progress has
been made in implementing signiiicant new provisions” or “have
feasibility problems become evident?“) could be asked in general form
in cases where the Committee's specific concerns have been communicated in
earlier discussions with the agency. 2
Agency officials indicated that they can respond to telephone or letter
requests for time- sensitive factual information relatively quickly.
However, responses to statutory reporting mandates or to Committee requests
for information that raise budget or policy issues or that involve changes
in legislation take time to develop. Such responses often involve policy or
management concerns-- concerns that go beyond the immediate program and that
program staff are not authorized to address. Department officials
appropriately require that these responses be centrally reviewed so that
‘We expect that currently established departmental procedures would be
followed in transmitting Committee requests to program agency staff and in
approving and transmitting the latter's response.
‘The questions to be included in the letter, and the timing of the
requests, should be reviewed when the system of annual performance reports,
established through the Government Performance and Results Act of 1993 (P.
L. 103- G!), is implemented in the year 2000. The fit between our approach
and the act's ls discussed in more detail later in this chapter.
Page 46 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
these broader concerns can be addressed and input from different units
coordinated.
It may be helpful if, along with stating the Committee's questions, the
request letter or other communication included such language as:
“Please contact (Committee staff person) if you have questions; we
welcome the opportunity to discuss this request.” This would encourage
contact from the agency in cases where clarification was needed and would
also give the department an opportunity to indicate which questions could be
answered quickly and directly and which raised complex issues that would
take more time to resolve.
Preparing for Reauthorization or Policy Review
As discussed in chapter 2, answering the evaluation questions that arise
during reauthorization often requires data or narrative information beyond
what the program routinely collects. We urge the Committee to notify
agencies well in advance when it has such questions in mind so that they can
plan (and budget) to obtain and analyze the necessary material in time. Such
advance notice would help the agency set priorities for using the resources
available to support evaluations. Comparative studies also require
considerable advance preparation.
Communicating About The foregoing leads us to the final element of our
strategy: increased Information Needs committee- agency communication
concerning information needs. As
stated in chapter 3, present practice affords little opportunity for joint
discussion and gives agency program and evaluation staff little basis for
understanding what the Committee wants to know and what method of
“packaging” the information would be most useful to it. We agree
with other authors that increasing the opportunities for discussion would
likely lead to more satisfactory results.
Increased communication is critical at two points and would be useful at
several others. The critical points occur
l when the Committee frames a request for information, to ensure that the
agency understands what is wanted and thus can alert the Committee to issues
of content or feasibility that need resolution, and l as report drafting
begins, to assist the agency in understanding the issues
that will be before the Committee and what kind of presentation format is
thus likely to be most useful.
Page 47 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
These points are critical because unverilied agency assumptions or
misinterpretations at either point can cause even the best- intentioned
agency response to be off the mark.
The additional points at which we suggest communication occur l when the
agency develops the design of a major study, to verify that it is
on target from the Committee's perspective; . midway through the period
between reauthorizations, to consider whether
issues and priorities have changed and to involve any new Committee staff;
and 0 after a report is submitted, to convey feedback concerning its
usefulness
to the Committee. Communication at these points might not be necessary in
cases where the task before the agency is clear- cut, the issues are
unlikely to change, and the report is well received. However, in less
favorable circumstances, these communications points would likely be useful.
Additional communication can be facilitated by using organizational
structures and procedures that are already in place. It is important for the
Committee to initiate requests and to give agencies a clear signal that it
welcomes discussion with the agency, as previously outlined. The
congressional liaison officer for the agency can then assist by providing
agency program and evaluation staff with background information about the
Committee's intent. He or she can also bring agency and Committee
representatives together so that issues can be resolved and mutual
understanding achieved.
Feasibility of Our Approach
During the course of our study, we discussed our proposed strategy with
Committee staff and with officials in the agencies that served as case
studies. Both groups found our core questions useful and acknowledged the
potential benefits and even a common interest in moving in the direction we
propose.
Both groups also noted factors (ranging from demands on Committee time to
potential conflicts over where to draw the line between legislative
oversight and executive responsibility) that create obstacles to the kinds
of collaboration the strategy envisions. Given the different constituencies,
institutional perspectives, and interests of the Congress and the executive
Page 48 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
departments, these obstacles are to be expected. However, we were impressed
by the commonality of interest that we observed between the two groups. We
believe that it constitutes a solid basis for moving forward to improve the
usefulness of evaluation information.
Agency Comments and Our Response
In written comments on a draft of this report, the Departments of Education
and HHS agreed to the benefits of increased consultation between the
Committee and the agency regarding evaluative information needs. However,
they were concerned that our proposed strategy be consistent with the
requirements of the Government Performance and Results Act and not
constitute an added burden. The Department of Education also expressed
concern about two of our categories of information- side effects and
comparative advantage. The Department noted that questions under these
categories can be diftkult for the program agency to address if they require
information about programs administered by another agency. We address these
concerns in turn.
Our Approach and GPRA While this study was in progress, the Congress passed
the Government Performance and Results Act of 1993 (P. L. 103- 62). By the
turn of the century, annual reporting under this act is expected to fiU some
of the information gaps we describe in this report. Among other things,
GPFLA requires every agency to establish indicators of performance, set
annual performance goals, and report on actual performance in comparison
with these goals each March beginning in the year 2000. Agencies are now
developing strategic plans (to be submitted September 30, 1997) articulating
the agency's mission, goals, and objectives preparatory to meeting these
reporting requirements.
We compared our approach with GPRA'S and concluded that the two are
compatible. Both approaches emphasize the importance of agency consultation
with the Congress as evaluation strategies are planned, goals and objectives
identified, and indicators selected Both note the importance of providing
information that indicates how well the program is doing with respect to its
intended objectives, and both call for annual reporting of information to
provide the basis for accountability and effective oversight.
We foresee little added burden- and some efficiency of effort- from using
our approach in conjunction with GPRA. The procedures for identifying and
adapting core questions presented earlier in this chapter are likely to be
Page 49 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Chapter 4 A Strategy for Requesting Evaluation Information
useful in preparing to meet GPRA requirements, and consultation with the
Committee should ensure that data collected to support GPRA reporting can
also be used to meet the Committee's special needs (for example, that
performance data can be disaggregated in ways that are important to the
Committee).
At the same time, our suggested approach provides a useful complement to
GPRA. Our approach to requesting interim- year information can be applied
immediately, to take advantage of currently available information while GPRA
reports are being planned. Moreover, while GPRA annual reporting focuses on
intended program outcomes, our approach covers additional categories of
information- description, side effects, and comparative advantage- that the
Committee finds useful, especially in connection with major program
reauthorizations or policy reviews.
Evaluating Side Effects and Comparative Advantage
We agree that to evaluate comparative advantage typically requires data
beyond what the administering agency can provide and that some side- effects
questions might also pose this problem. Our discussion in chapter 2 notes
that by virtue of their complex informational requirements, comparative
advantage studies typically are implemented through special arrangements.
Special arrangements would similarly be appropriate for conducting a
detailed evaluation of one program's side effects on other programs.
Many side- effects questions, however, fall within the scope of evaluation
for a particular program (as our case study examples illustrate). The
process of discussion that we have suggested should help the Committee and
the agency identify the side- effects questions that are pertinent to a
program and the level of detail or precision needed for each. Discussion
should also examine whether it is feasible for the administering agency to
gather this information or whether special arrangements will be needed.
Page 50 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Page 51 GAOIPEMD- 96- l Improving the Flow of Program Information to the ,,
( , * 8,
/ ,:::-: i ,pl ‘/ ' ;, : “‘, ,,;
.,/
Appendix I Comprehensive Child Development Program
Program Description Program Authorization and The Comprehensive Child
Development Program was authorized in 1988 Objectives under the
Comprehensive Child Development Centers Act of 1988 (P. L.
100297). CCDP was reauthorized through fiscal year 1994 under the Augustus
F. Hawkins Human Services Reauthorization Act of 1990. In May 1994, elements
of CCDP were incorporated into a new Head Start Program for Families With
Infants and Toddlers, authorized under titIe I of P. L. 103252, the Head
Start Act amendments1
CCDP was designed to enhance the physical, emotional, and inteIiectuaI
development of low- income children (infants to school age) and to
contribute to self- sufficiency by providing support to their parents and
other family members. This demonstration program's purpose is to test
whether it is feasible and effective to provide integrated and comprehensive
support services early in the child's Iife and within the family context.
Program Operation, Participation Levels, and Funding
The Department of Health and Human Services funded 24 CCDP grants in 1989
and 1990 and 10 more in 1992. Each grant provided funding for 5 years. There
are currently 34 projects in ruraI and urban areas providing core services
(health care, mental health care, child care, early educational
intervention, early childhood development, prenatal care, parenting
education, employment counseling, vocationaI training, adult education, and
nutritional assistance).
A CCDP grantee organization (which may be a health clinic, a family services
agency, a university, or even a school district) acts as a service
integrator by building supporting networks with community agencies and at
times facilitating and advocating for services. Each grantee organization
assesses local needs in its area and plans ways to meet these needs through
direct intervention of family- focused case management and a combination of
project- provided services and services arranged through third- party
providers. AR projects use the case management approach, however, the degree
of reliance on existing agencies in the community
‘This appendix describes CCDP until the May l& 1994, Head Start Act
amendments and reauthorization.
Page 62 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Appendix I Comprehensive Child Development Program
varies from project to project. Each project has an advisory board with
community, business, and client- family representation. 2
Fiscal year 1991 funding for this program totaled $24 million; for fiscal
year 1993, an increase of $23 million ($ 20 million received from the
Department of Education) brought the total to $47 million, which was also
the amount requested for fiscal year 1994.
Administrative Structure The Head Start Bureau of the Administration on
Children, Youth, and Families in the Department of Health and Human Services
is responsible for the administration of CCDP grants. ACYF'S functions
include selecting grantees, providing technical assistance, monitoring their
compliance with program regulations, and directing contractors in the
performance of process and impact evaluation studies.
Mandated Evaluation The authorizing legislation for CCDP mandated an
evaluation report on program impact and program feasibility that was due on
October 1,1993. HHS provided an interim evaluation report- on short- term
program impact and on the structure and mechanisms of service delivery- to
the Senate Committee on Labor and Human Resources in May 1994. That interim
report covered the results of approximately the first half of the &year
cycle of services. A final report covering the entire program is scheduled
for delivery in March 1996.
Major Sources of _ To meet the evaluation requirements, the agency has
arrangsd to collect Information diverse types and sources of information.
The grantee organizations are
required to collect data on program operations, user participation, and
compliance with program regulations. In addition, each project submits
progress reports on its activities (including vignettes on participants'
successes) to ACYF. A nonparticipant observes agency operations and records
information on side effects and local community dynamics in ethnographer
reports. ACYF conducts site visits that result in a letter assessing
regulatory compliance and implementation and discussing quality issues (site
visit reports). These program- monitoring documents also contribute to the
assessment of CCDP operations and feasibility.
2ACYF' refers to individual CCDP grantee sites as “centers.” We
use the term “project” instead to refer to the physical center
itself, community chsracteristics, and services availsble in the community.
Page 63 GAO/ PEMD- 96- 1 Improving the Flow of Program Information to the
Congress
Appendix I Comprehensive Child Development Program
In December 1991, the agency published a report that described CCDP and
start- up and enrollment issues (referred to as the CCDP First Annual
Report). The agency has recently published its interim evaluation report tc
the Congress entitled “Comprehensive Child Development Program- A
National Family Support Demonstration” (referred to as the CCDP
Interim Report). The CCDP Interim Report provided analyses of program
implementation and short- term program impacts on families and communities,
as well as a description of families and projects. In a fmal report to be
prepared after the enrolled families have received 5 years of services, the
agency expects to provide a greater understanding of how effects vary with
the intensity and duration of services, as well as with differences in the
effects of projects' characteristics. It will also provide a picture of the
effects of long- term participation on families and children.
Evaluation Questions and Information Needs
Interim- Year Questions CCDP projects have reported on start- up problems
and early progress. The Committee might also ask to be informed of new
feasibility issues and of program impacts or significant side effects
(discussed in more detail below) as they become evident at existing and new
projects under the reauthorized Head Start Program.
Reauthorization Questions After the Head Start reauthorization of 1994, CCDP
will not continue as a separate program, but its grantees will continue to
operate, together with new ones, under the newly authorized Head Start
Program for Families With Infants and Toddlers. Because this new program
appears generally similar to CCDP, we expect that the questions posed for
CCDP could also inform oversight of the new and continuing projects.
Table I. 1 suggests evaluation questions pertinent to a demonstration
program that are drawn from the list of core questions in chapter 2 and
restated in terms of CCDP. The table also identifies indicators or measures
needed to answer those questions and summarizes related information that is
currently available. It demonstrates the use of the question- selection
strategy proposed in chapter 4. The questions and indicators are suggestive
and not prescriptive for the program. Under our
Page 64 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Appendix I Comprehensive Child Development Program
strategy, the Committee and Department staffs would collaborate to work out
the specific questions and indicators needed.
Table 1.1: CCDP Evaluation Questions and Related Information Question
Indicator or measure Available information
Description What model or approach is Description of required Requirements
are stated in tested here (core services elements of the model authorizing
legislation and provided, service delivery program documents and method,
organization)? were summarized in the
CCDP Interim and First Annual Reports.
How do features of the model Narrative descriptions of Management
information differ across projects? major variants and data system, progress
reports,
showing the frequency with and ethnographer reports which each occurs
provide relevant
information; CCDP Interim Report notes significant variants on the model
(for example, type of grantee agency, staffing configurations, and
interagency agreements).
What are the conditions Range and amount of Management information (project
size and philosophy, variation in project system, progress reports, client
makeup, or community characteristics that affect and ethnographer reports
resources) under which implementation of program provide this information,
but services are provided, and model the CCDP Interim Report how do they
differ? does not compare critical
project- based characteristics of program operations; the final report is
expected to make such comparisons.
Implementation What start- up and operating Observational or participant
Site visit, ethnographer, and difficulties did projects face survey data
collected as progress reports described and how were they resolved? program
was implemented difficulties of putting the
new projects into operation and providing all core services; both the CCDP
First Annual and Interim Reports cover start- up and operational
difficulties and early solutions.
(continued) Page 56 GAO/ PEMD- 95- 1 Improving the Flow of Program
Information to the Congress
Appendix I Comprehensive Child Development Program
Question What modifications in the model were made by the grantees or
program (for example, in case management or in arrangements with other
service providers)?
Indicator or measure Observation or survey responses describing the
modifications and reasons for making them, descriptions and classification
of arrangements between organizations at difl erent times
Available information Progress, site visit, and ethnographer reports, as
well as refunding grant applications, contain pertinent information, some of
which is summarized in the CCDP First Annual Report: the basic program model
is also described there.
What has CCDP Data that illuminate The sources mentioned demonstrated about
the conditions for successful above, supplemented by feasibility of applying
the implementation, barriers grantees' financial records, program model? to
implementation, and include the needed
cost of implementation information; CCDP Interim Report provides information
on implementation problems and some solutions in the early years of the
program.
Targeting Question is not Not applicable Not applicable applicable- client
selection not discretionary
impact To what extent did services Measures of children's and improve child
development family members' skills and and family self- sufficiency?
behaviors upon enrollment
and after receiving services; measures of services received; comparable data
from comparable families that did not participate.
CCDP Interim Report provides information on a variety of outcomes for the
child, mother, and family as well as for members of the control group; these
data can be combined with descriptive data about users and services to
provide answers about imoact.
CCDP Interim Report provides programwide child and family outcomes;
subsequent reports are expected to describe how effects vary by services
provided and project- based characteristics. Under what conditions, and
Measures of child and
with what kinds of services or family outcomes by project delivery
mechanisms, has the and service, displayed to program been most or least
show range and variance. successful?
(continued: Page 56 GAO/ PEMD- 96- l Improving the Flow of Program
Information to the Congress
Appendix I Comprehensive Child Development Program
Question Indicator or measure Available information Side effects Have CCDP
activities Evidence of addition of new Ethnographer, advisory prompted
increased services or increased board, and project provision, quality, and
coordination among director's reports captured coordination of services to
existing services as an these observations, and the non- CCDP community?
outgrowth of project positive changes were
networking efforts noted in CCDP Interim Report.
Comparative advantage Would a reduced version of CCDP (which included only
the most viable elements) be more effective than other approaches?
Estimates of enhanced CCDP evaluation studies family self- sufficiency and
will provide some impact child development under data, which would be basis
modified- CCDP, impact for estimates of data from comparison modified- CCDP
effects; a program or other approach separate study is needed to
compare these estimated effects with the effects of other programs.
, Page 67
,, I' ‘II,
GAOL'EMD- 95- l Improving the Flow of Program Information to the Congress
Appendix II Community Health Centers -
Program Description Program Authorization and The Community Health Center
(CHC) program was authorized in 1975 Objectives under section 330 of the
Public Health Service Act. At the time this report
was written, the program was authorized through fiscal year 1994, and
reauthorization for fiscal' year 1995 was expected shortly. The program's
purpose is to provide access to comprehensive and case- managed primary
health care services to rural and urban populations living in medically
underserved areas1 The program also targets “at- risk”
populations (pregnant women, children, substance abusers, and elderly
persons), and centers seek to overcome barriers to health care access, such
as those related to culture and language differences.
Program Operation, The CHC program funded 579 grantee centers serving 1,575
sites on a Funding, and Participation budget of $558 million in fiscal year
1993. (The fiscal year 1994 Levels appropriation was about $603.5 million.)
All centers provide health care
services such as physician services, diagnostic laboratory and radiological
services, and pharmaceutical and emergency services, as well as preventive
care like immunizations, dentistry, family planning, and vision and hearing
screening. Other required services include translation services,
transportation, and referrals to other providers, and centers can opt to
provide supplemental services like health education and outreach. Services
are provided to about 6 million medically under- served or “at- risk'
people a year. The fiscal year 1994 appropriation of $604 miLlion represents
an increase of $46 million to fund new sites and serve more people.
CHC centers serve areas (urban and rural) with poor access to medical
services, and they also provide supporting services targeted to the local
community's needs (such as transportation, day care, or culturally sensitive
care to minority groups). Most centers operate on a fee- for- service basis,
while others provide managed care.
Administrative Structure The CHC program is administered by the Bureau of
Primary Health Care. BPHC is part of the Health Resources and Services
Administration within
‘An area is designated “medically underserved” by HHS, in
codunction with state authorities, based or a formula that includes four
variables: physician- to- population ratio, infant mortality r& e, poverty
population, and population 66 years of sge and older. An area may encompass
a neighborhood or an entire county,
Page 58 GAOA'EMD- 95- l Improving the Flow of Program Information to the
Congress
Appendlx II Community Health Centers
the Public Health Service of HHS. Regional JJHS offices assist headquarters
in monitoring grantees' compliance with program regulations. The CHC has a
subprogram, the Comprehensive Perinatal Care Program, that provides
additional funding for enhanced perinatal services and is treated here as a
separate program.
Mandated Evaluation The last reauthorization did not mandate an evaluation
or any type of report to the Congress on this program. However, HFEA has
initiated and conducted several evaluation studies, which are discussed in
the next section.
Major Sources of Information The legislation requires (1) that community he&
h centers have an
effective procedure for collecting information on their costs of operation,
(2) that information on the use of services (their availability and
accessibility) be collected, and (3) that grantees collect data on program
operations and user participation. HHS regional offices collect and enter
computerized data received from grantees into the Bureau's Common Reporting
Requirements (BCRR) data system. BPHC is planning changes in this system to
allow collection of additional data on clients, their health status and
needs, and types of services provided. A new provision is intended to result
in information about the age- appropriate preventive services provided at
different sites. Grant applications include information on each center, the
health needs of its community, and progress toward achieving its program
goals (such as retaining clients).
BPHC, with the aid of HHS regional offices and on- site assessments
conducted by federal staff and consultants, reviews center operations and
compliance with BPHc- mandated budgetary and performance standards (such as
following standard medical procedures and appropriate staff licensing
guidelines). This is intended to ensure that centers are properly managed
and operated. The bureau also has information on special populations' needs
and nationwide health needs derived from individual grant applications and a
variety of other sources such as the U. S. census, national health surveys,
and state and local organizations.
BPHC has conducted several evaluation studies and is planning others. The
completed studies have provided baseline information on the range of
preventive services offered, examined the effects of Federally Qualified He&
h Center provisions on individual centers and the capacity of each to expand
its service, and traced the impact of CHC use on some Medicaid
Page 69 GAWPEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Appendix II Community Health Centers
costs and hospitalizations. Ongoing studies will provide assessments of the
case management approach and the managed care system used in certain
centers. Several planned studies should provide in- depth descriptive
profiles of CHC users and of services provided, as well as an evaluation of
program effectiveness and whether CHCS do improve the health status of their
users. 2
Evaluation Questions and Information Needs
Interim- Year Questions The forthcoming reauthorization may introduce new
provisions that raise implementation questions for the next few interim
years. Beyond these, questions concerning the emergence of significant
feasibility or service quality problems and new evidence concerning program
targeting, impact, or side effects will be relevant.
Reauthorization Questions The CHC program is now scheduled for
reauthorization consideration in 1995. Table II. 1 suggests evaluation
questions pertinent to the CHC program that are drawn from the list of core
questions in chapter 2 and restated in terms of this program. The table also
identifies indicators or measures needed to answer those questions and
summarizes related information that is currently available. This table
demonstrates the use of the question- selection strategy proposed in chapter
4.
2These studies will also evaluate the Migrant Health Center progrsm and, in
one instance, will combine the two programs and their results. Siice the CHC
program funds some centers that also get Migrant Health program funding, the
programs are usually located and operated together. Roughly 100 jointly
funded centers provide the same basic services to a mix of migrant and
resident clients using the same staff, thereby making it difficult to
identify unique CHC program operations and effects. About 450 organizations
are CHC only, and 20 are Migrant Health only.
Page 60 GAO/ PEMD- 96- 1 Improving the Flow of Program information to the
Congress
Appendix II Community Health Centers
Table 11.1: CHC Evaluation Questions and Related Information Question
Indicator or measure Available information
What services are actually provided, whom do they reach, and in what kinds
of settings?
User profile of client User profile information is characteristics
(insurance accessible in BCRR; status, relative income, information on
required and ethnicity, age, sex); profile optional services is of services
provided provided; and information (required services and on center settings
is range of optional services) available in grant
apDlications and BCRR. How do centers differ in Profile of a sample of
Center location and terms of location (rural or centers, noting variation in
description are available in urban), client characteristics, each feature
across centers grant applications, and and area population? client
demographics are
accessible in BCRR. What other aspects of the program are significant for
understanding program management and use of federal money?
Information on relevant Information on these variables (such as payment
features is available from systems used, percentage BCRR and grant of
clients covered by applications. insurance, and delivery models)
Implementation How have recent increases in Number of new sites or
Information is accessible in funding affected program expansions; number and
BCRR and program files. implementation with regard to description of newly
served capacity to serve, services clients provided, and location?
Does evidence suggest Frequency and severity of Information on important
significant compliance reports on significant center grantee noncompliance
problems with centers' problems or noncompliance may be collected from
mandate to provide minimally with critical requirements diverse sources
during adequate care or with regard routine program monitoring. to any
statutorv requirement?
Are there serious problems Program management Information about the that
centers face in information suggesting problems centers face is implementing
the program? problems, noting frequency available in grant
and severity of situations applications and from the such as unmet staffing
annual center review; needs, facilities needing headquarters or regional
renovation, or high staff may also be aware of insurance costs other
relevant information.
Targeting To what extent are centers Portion of population in Medically
underserved covering their medically medically underserved population size
is available underserved populations or areas or counties that CHCs from
grant application and areas? serve number of users in BCRR,
so portion can- be calculated.
(continued) Page 61 GAO/ PEMD- 95- l Improving the Flow of Program
Information to the Congress
Appendll II Community Health Centers
Question Indicator or measure Available information Are centers distributed
Proportion of medically around the country to cover underserved areas in U.
S. the most seriously without a center; underserved populations or
identification of those areas areas? without centers that cover
the most severely underserved counties (among all counties)
Amount of medical need in a specific area is available from grant
applications.
Amount of need in unserved areas is available from various national and
local sources; agency has reported amount of need met by CHC program but not
its size relative to the amount of unmet national need or the severity of
need; NACHC has reported on the amount and severity of need.
To what extent do centers serve special population clients?
Proportion of clients who are special population members
Number of users, including some special population users (the elderly,
children, infants) is available from BCRR; information on each center's
assessment of services needed and services provided is available in grant
application and BCRR.
Are there major client needs Evidence that centers Information on each that
center services do not provide services that match center's assessment of
address? needs identified in the services needed and
grantee's needs services provided is assessment; evidence that available in
grant program services offered application and BCRR. match program services
needed
Impact Have CHC centers with Change in coverage of BCRR has relevant client
optional, culturally sensitive, clients and increased information over time;
and other outreach services utilization by clients; evaluation or public
health succeeded in increasing change in proportion of literature may report
on access to health care for the ethnic group and other- program outreach
effects. targeted populations? language members served
Has the medically Information on continuity of Health status information,
underserved population client care, reduced use of use of health care
facilities, served by a center achieved emergency room facilities CHC and
non- CHC client higher health status? for nonemergency care, information are
available;
and change in morbidity in agency has ongoing and area completed evaluation
studies regarding these issues.
(continued) Page 62 GAO/ PEMD- 96- l Improving the Flow of Program
Information to the Congress
Appendix II Community Health Centers
Question Indicator or measure Available information Side effects Have there
been effects on Changes in use of Grant application or ad hoc other health
care providers? emergency facilities, sources may report this
hospitals, or school- based information; evaluation care or nursing study on
change in
Medicaid usage is avaiiable. Comparative advantage Is providing clinics (as
Comparison of costs, types Agency has similar opposed to encouraging of
services provided, and evaluation studies planned. others to provide
service) clients served between more successful in CHCs and another
nonclinic increasing access to care for delivery approach the targeted
populations?
:: :‘, ‘I ‘Es ,,,, :;
> :: ‘1, ?:,‘,. , ,, .“-“:,
‘) “‘,:,,; *' ,. I,:
Page 63 ,,, i
,I-; “:, ,,
1, ,, ' )I $,,
:: ,I( ,\:, 5, (,,
GAO/ PEMD- 96- l Improving the Flow of Program Information to the Congress
._ I,, l,,
Appendix III apt& 1 Ele entary d Secondary Education Act
Program Description Program Authorization and The Chapter 1 program
(originally known as Title I) was established by the Objectives Elementary
and Secondary Education Act of 1965. This summary
describes the program as it existed under the 1988 Augustus F. Hawkins-
Robert T. Stafford Elementary and Secondary School Improvement Amendments
(P. L. 100- 297). (Substantial changes were introduced in 1994, as discussed
later in this appendix.) Chapter 1 provides financial assistance to local
education agencies to help educationally deprived children in low- income
areas (1) succeed in the regular program of the local education agency, (2)
attain grade- level proficiency, and (3) improve achievement in basic and
more advanced skills.
Program Operation, The program primarily provides compensatory instruction
in basic Funding, and Participation subjects (reading, mathematics, language
arts). Some supporting services Levels (social work, health and nutrition,
transportation) are also provided. In
most schools, only low- achieving students receive assistance, which may be
given in the classroom or via “pull out” to a separate setting.
Schools in which 75 percent or more of the students come from low- income
families may use Chapter 1 funds to support schoolwide improvement rather
than serve only the low achievers. Eligible students who attend private
schools receive services provided off- site through the public schools.
Parent participation is encouraged.
With an annual budget of $6.7 billion, Chapter 1 is the‘ largest
federal program in elementary and secondary education. Grants are awarded to
state education agencies, and through them to school districts in accordance
with the numbers of students from low- income families. All states, Puerto
Rico, the District of Columbia, and the outlying territories participate.
Funds flow to 90 percent of all school districts, over 70 percent of all
public elementary schools, and 30 percent of public secondary schools.
Nearly 5.5 million children are served.
Administrative Structure Chapter 1 is administered by the Office of
Compensatory Education Programs within the Department of Education's Office
of Elementary and Secondary Education. Federal program staff review the work
of state
Page 64 GAOREMD- 96- l Improving the Flow of Program Information to the
Congress
Appendix III Chapter 1 Elementary and Secondary Education Act
Chapter 1 program directors, who in turn review compliance and progress at
the local level.
Mandated Evaluations The Chapter 1 program was the subject of major mandated
evaluations in 1977, in 1987, and again in the 1992 National Assessment of
Chapter 1. The 1992 assessment drew on preliminary findings from the
“Prospects” study, a national longitudinal study mandated in
1988 to trace the effects of Chapter 1 participation to young adulthood. The
Congress has also requested GAO reports on various aspects of the program.
The 1988 legislation required states to evaluate their Chapter 1 programs at
least every 2 years. State review of student gains for each school in terms
of standardized test scores was deemed sufficient to meet this requirement
and was used to identify schools in need of improvement. Local education
agencies were to conduct program evaluations every 3 years. At the school
level, schoolwide projects were to be evaluated after 3 years of operation.
In addition, individual students' gains were to be examined and
individualized educational plans devised for those who were not making
progress.
Major Sources of Information
The 1992 National Assessment drew on numerous studies of special issues and
populations as welI as case studies of school programs. These are summarized
in its final report1 It also drew on preliminary results from two major
studies: (1) the “Prospects” study, which examines the school
experiences and achievement of nationally representative samples of
disadvantaged lst, 3rd, and 7th graders (some of them receiving Chapter 1
services, and others not) over many years, and (2) an observational study
entitled “Special Strategies for Educating Disadvantaged
Students.”
Annual state education agency reports to date have listed the number of
schools, staff, and students participating, as well as the number of schools
deemed in need of improvement. State agencies also reported the statewide
average pretest and posttest scores for Chapter 1 students.
As its basis for allocating funds to states, the program has relied
primarily on family income data from the decennial census. Local education
agencies have used data on numbers of poor children from the Department of
Agriculture's school lunch program to determine school
‘U. S. Department of Education, Reinventing Chapter 1: The Current
Chapter 1 Program and New Directions (Washington, D. C.: Feb. 1993).
Page 65 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Appendix III Chapter 1 Elementary and Secondary Education Act
eligibility. Although it cannot distinguish participants in Chapter 1 from
other students, the National Assessment of Educational Progress provides
general evidence of how disadvantaged students' achievement in basic
academic subjects compares with that of more advantaged students.
Changes Enacted in 1994 The Improving America's Schools Act of 1994 (P. L.
103- 382) renamed the Chapter 1 program as Title I and introduced
significant changes. The 1994 law specifies components or qualities that
funded activities should incorporate- including high standards, enriched
programs, upgraded instruction, and improved teacher professional
development- and alters the kinds of information that will be available.
Under this law, the requirement that student achievement be measured in
terms of performance on nationally normed standardized tests has been
dropped. Instead, consistent with the provisions of the Goals 2000: Educate
America Act (P. L. 103- 227), states are encouraged to evaluate student
achievement and school performance in terms of standards and assessments
specific to each state and applicable to all students, not just those
participating in Title I programs.
In place of the historic emphasis on providing compensatory services to
individual students, the Improving America's Schools Act emphasizes
strengthening the regular program in schools that serve low- income
students. It would permit many more schools to adopt schoolwide programs (in
which funding is used to upgrade the entire educational program and
“Title I students” are not distinct from other students).
Student achievement and school improvement would be evaluated only at
certain grades selected by each state (such as 3,8, and 11). Thus, student
achievement data will be cross- sectional and state- specific. Data will be
drati from a range of students who attend schoolwide programs and from low
achievers who have received special services in schools with
“targeted” programs. At the time of this writing, future
reporting requirements had not yet been worked out.
The act provides for data on individual students' year- to- year progress to
be collected through a national longitudinal study. The longitudinal study
will compare student performance against content standards of national
scope. A second mandated evaluation study will examine progress toward the
goal of having all children served by this program reach their state's
content and performance standards. This evaluation will also examine the
targeting of resources and the extent to which the high standards, enriched
programs, upgraded instruction, parental participation, improved
Page 66 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Appendix III Chapter 1 Elementary and Secondary Education Act
teacher professional development, and other program practices specified in
the act have been put in place. Projects to demonstrate effective practices
are also authorized.
Evaluation Questions In identifying the Committee's future information
needs, we took into and Information Needs
account the major changes adopted in 1994. Specifically, we assumed (1) that
school and student performance would be evaluated in terms of standards and
assessments specific to each state rather than in terms of nationally normed
tests, (2) that only cross- sectional data for certain grades would be
available from state records, and (3) that longitidinal data on a sample of
individual students would be available from the “Prospects”
study and from the newly authorized longitudinal study. Information may also
be available from projects funded to demonstrate effective practices.
Interim- Year Questions In light of these changes, in the years immediately
following the 1994 reauthorization, the Committee will likely wish to be
kept informed of
. progress in implementing the new requirement for state standards and
assessments and related changes in curriculum, instruction, and staff
development; . feasibility issues that have arisen as implementation has
moved forward; l early evidence that new assessments meet criteria of
technical soundness
and x. e aligned to state standards; . early evidence of effects of new
standards and assessment practices on
student achievement; and l early evidence that the changes have
unanticipated side effects on
instruction or on parents, teachers, or students (especially students with
disabilities or limited English proficiency).
Reauthorization Questions Table III. 1 lists evaluation questions that the
Committee may wish to ask at the next ESEA reauthorization (scheduled to
take place in approximately 5 years), indicators or measures needed to
answer them, and related information that is currently available. The
questions are drawn from the list of core questions in chapter 2 and
restated in terms of this program. As noted earlier, it is unclear what
information about student and school performance will be reported to the
administering agency under the new provisions. The question of whether and
how state data can be aggregated to form a national picture, given that each
state uses a different standard,
Page 67 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Appendix III Chapter 1 Elementary and Secondary Education Act
has yet to be answered. The entries under “Impact” in table III.
1 suggest one possible method of accomplishing this task. We consider our
approach illustrative of what might be done. Under our strategy, Committee
and Department staff would collaborate to work out an approach. 2
Table 111.1: Chapter 1 ESEA Evaluation Questions and Related Information
Question
Description indicator or measure Available information
How different are High, low, and modal Socioeconomic data participating
schools and value( s) for each feature; available from free or districts
from one another in typical (frequently subsidized lunch program; terms of
socioeconomic occurring) combinations of achievement data from variables,
student features “Prospects” study; resource performance, and
school data from special studies resources?
How do schools differ in Narrative description of Capsule descriptions of
instructional strategy and major variants and the successful schools, case
method of delivery and in the school conditions under studies of school
practices, added services provided to which they occur; and observations
drawn low- achieving students? frequency count or estimate from site
reviews;
of relative frequency of “Prospects” study may each variant
contain data
How many states have Number of states that have Standards to be set forth in
established content and standards; examples or each state's plan; additional
performance standards, and illustrations of standards details may be needed
for how widely do these that differ in scope and level adequate description
standards vary?
To what extent are state assessments aligned to state content and
performance standards, of acceptable technical quality, and appropriately
adapted for limited English proficiency students and students with
disabilities?
Findings from reviews of Federal and state quality instrument content,
criteria and review administration, and scoring procedures not yet methods;
evidence of established reliability and validity of results
(continued) 2Consistent with the 1988 as well as the 1994 law, this
program's central goal is to bring participating students up to a certain
level of educational performance. Thus, its impact is properly evaluated in
terms of how many students, and which kinds of participating students,
achieve this goal. The program also is intended to lead states, school
districts, and schools to adopt improved practices. We treat these practices
as standards for program operation, an aspect of implementation.
Page 68 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
Congress
Appendix III Chapter 1 Elementary and Secondary Education Act
Question Indicator or measure Available information What proportion of
Frequency data from school Future evaluation studies participating schools
have surveys and from authorized to collect programs that exhibit the
observations of actual relevant data, but such characteristics of quality
school practice studies not yet designed; education embodied in the state or
other site reviews authorizing statute? What also a potential source
proportion lack the essential characteristics of this model?
Targeting To what extent are services targeted to schools with high
concentrations of low- income students?
To what extent are services reaching low- achieving students in these
schools, as well as students with special educational needs (such as limited
Enghlish proficiency, disability, or high mobility)?
Proportion of participating schools in which there are high concentrations
of students from low- income families; proportions of participating students
in low, moderate, and high poverty schools
Proportion of students served/ benefited who are low achieving, have special
needs, or both
Data available at the school level but may not be reported;
“Prospects” study includes relevant data
Data on achievement, handicap, and English language proficiency for each
student served available at the school level, but not yet clear whether
these data to be reported; “Prospects” study contains relevant
data
How many severely needy Number of unserved Local education agencies schools
and students remain schools, by percentage of gather school poverty data
unserved by this or other low- income and for school selection but
comparable program? Have low- achieving or need not report it; National
numbers and percentages multiple- needs students; Center for Education
decreased since 1994? number of unserved needy Statistics Schools and
students in all eligible Staffing Survey contains schools relevant
information
Impact In the aggregate, is the Measure of student National Assessment of
performance of economically achievement and economic Educational Progress
disadvantaged students status for national sample, provides this kind of
approaching that of their such that distributions can information advantaged
peers? be compared
(continued) Page 69 GAOLPEMD- 96- l Improving the Flow of Program
Information to the Congress
Appendix III Chapter 1 Elementary and Secondary Education Act
Question . Indicator or measure Are students served by this Percentage of
students program being brought to the whose performance (1) is level of
performance seriously deficient and expected of all students? shows little
improvement,
(2) has improved but is still below expected level, (3) has reached the
expected level but student needs assistance to maintain gains, or (4) is
sufficiently strong to “graduate” from Chapter 1
Percentage of students who have received Chapter 1 services for 3 years or
more and have not reached the expected level
Available information Schools classify students‘ progress and
achievement in order to determine who needs extra assistance: however, these
judgments are not reported, and it may be some years before they can be
linked to state standards; new longitudinal study may gather relevant data
“Prospects” study to have length- of- participation data for a
sample of students
Under what conditions, and Student performance data “Prospects”
study to with what kinds of students, linked to school, student, provide
information; has the program been most and service delivery data additional
data will be or least successful? Are needed schoolwide programs more
effective for low achievers than programs that target specific students?
Has the school improvement Number of schools needing Criteria for school
procedure led to improved improvement and improvement to be set by student
outcomes? percentage in which states but are not yet
student performance has established improved substantially; comparative data
on performance of low- achieving students in schools that serve similar
populations but differ in type of program
Side effects Has the adoption of Observational data on
“Prospects” study may schoolwide programs extent of services to
provide some data but may decreased the extent or low- achieving students
prior include too few cases effectiveness of assistance to and after
adoption of provided to low- achieving schoolwide programs;. if not students
in those schools? available, cross- sectional
comparative data Has the nature, amount, and Percent of districts that New
issue; what information cost of student testing drop annual testing, cease
will be available not yet changed substantially? to use a nationally normed
known
test, or use either performance- based tests or multiple assessments; per-
student cost of testing
(continued) Page 70 GAO/ PEMD- 95- l Improving the Flow of Program
Information to the Congrees
Appendix III Chapter 1 Elementary and Secondary Education Act
Question Indicator or measure Available information Where the nature of
Proportion of classrooms in New issue; what information assessment has
changed, to which such changes have will be available not yet what extent
have parallel been reported or observed known changes occurred in
instruction?
Has use of standards- Changes in proportion of “Prospects” study
may based testing adversely such students who are provide some data but
affected students who face included in assessment; additional information
will special barriers (language, patterns of attendance, likely be needed
handicap) to meeting such effort, and persistence in standards? school on
the part of such
students Comparative advantage Is there evidence that tying Change in rate
of National and state results Chapter 1 criteria to state aggregate
achievement of from the National standards has been more disadvantaged
students Assessment of Educational advantageous in relation to nationally
and in each state: Progress may be useful; its cost than the prior changes
in costs; case plans for other data approach? (Generally, or only study data
from varied collection not yet known in some states?) states
:' .,.‘,,
;1 ,:., ),
:: ::
Page 71 -;. ,,‘ y
: 1,:-. : i ,.
GAO/ PEMD- 96- l Improving the Flow of Program Information to the Congress
“'
Appendix IV Comments From the Department of Education
UNITEDSTATES DEPARTMENT OF EDUCATION OFFXE OF ELEMENTARY AND SECONDARY
EDUCATION
THE ASSISlXNT SECRETARY October 19, 1994 Ms. Terry E. Hedrick Assistant
Comptroller General Program Evaluation and Methodology Division United
States General Accounting Office Washington, D. C. 20548
Dear Ms. Hedrick: The Secretary has asked that I respond to your request for
comments on the GAO draft report, @Tongressional Oversight: Obtaining
Information for Program Evaluation" (GAO Sob Code 973766), which was
transmitted to the Department of Education by your letter of September 2,
1994.
We believe it is very important for Congress to have timely accurate
information regarding the operation of our programs and the report prepared
by GAO will be helpful in furthering this aim.
We concur with your recommendation that a useful strategy for the Department
and for Congressional staff is to increase dialogue with each other. Good
communication, shared planning, and mutually agreed upon approaches are
essential. Improved dialogue would not only strengthen the work that we do
but ensure that we provide information Congress believes is important. We
also concur with the recommended strategy of having some core questions that
will permit Congress and the Department to have information on programs
during interim years. Our primary concern, however, is that the program
goals, indicators, and
outcome data requested for reauthorization be as consistent as possible with
the requirements of the Government Perfornance and Results Act (GPHA) which
is driving our strategic planning and budget processes.
We do have a concern with GAO's call for assessments "side- effects" and
"comparative advantage". Questions such as these are often beyond the scope
of the evaluation of particular programs, and difficult to address if the
programs used for comparison are administered by another agency.
Page72 GAOL'EMD- 96- 1 ImprovingtheFlowofProgramInformationtotheCongress
” ‘:, Appendix IV Comments From the Department of Education
I' Ms. Terry E. Hedrick Page 2
Thank you for the opportunity to comment. My staff and I are prepared to
respond if you or your representatives have any questions.
<;, pw Thomas W. Payzant TWP: jt cc: Alan Ginsburg
Page 73 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Appendix V Comments rom the Department of Health and Human Services
DEPARTMENT OF HEALTH 8 HUMAN SERVICES Olllca al Inspectcw General
Washington, D. C. 20201
OCT I8 I994 Ms. Terry E. Hedrick Assistant Comptroller General United States
General
Accounting Office Washington, D. C. 20548
Dear Ms. Hedrick: Enclosed are the Department, '6 comments on your draft
report, "Congressional Oversight: Obtaining Information for Program
Evaluation." The comments represent the tentative position of the Department
and are subject to reevaluation when the final version of this report is
received.
The Department appreciates the opportunity to convnent on this draft report
before its publication.
Sincerely yours, Inspector General Enclosure
Page 74 GAO/ PEMD- 96- 1 Improving the Flow of Program Information to the
Congress
Appendix V Comments From the Department of Health and Human Services
COMMENTS OF THE DEPARTMENT OF HEALTH AND HUMAN SERVICES ON THE GENERAL
ACCOUNTING OFFICE [GAO) DRAFT REPORT
"CONGRESSIONAL OVERSIGHT: OBTAINING INFORMATION FOR PROGRAM EVALUATION."
SEPTEMBER 1994
The Department has reviewed the General Accounting Office (GAO) draft report
and has the following comments.
MATTERS FOR CONGRESSIONAL CONSIDERATION This report proposes a strategy to
obtain information for program oversight and reauthorization that the Senate
Committee on Labor and Human Resources may wish to adopt. The three
components of this strategy are
-- selecting and adapting, from a core list, the descriptive and evaluative
questions to be asked about a program in interim years and at
reauthorization;
-- arranging explicitly to obtain timely oversight information in interim
years as well as to receive results of evaluation studies at
reauthorization; and
-- providing for increased communication with agency staff to help ensure
that information needs are understood and that requests and reports are
suitably framed and adapted as needs evolve.
This strategy can be adapted to take institutional realities into account.
For example, in view of the many demands on its attention, the Committee
might select future reauthorization questions for some programs and invite
agencies to propose questions for others.
DEPARTMENT COMMENT We generally support the GAO draft report's
recommendation that congressional committees more clearly express their
evaluative information needs to agencies and consult with agencies on those
needs. However, we have concerns about GAO recommending a process that would
burden agencies with additional evaluation reporting requirements on top of
the extensive annual reporting of performance required under the recently
enacted Government Performance and Results Act of 1993 (GPRA).
From our analysis, much of the information GAO cites as useful to
congressional committees for both annual oversight and reauthorization
activities could be part of the annual performance report required of each
program activity under .GPRA. The GAO draft report, however, does not give
adequate recognition to this fact or to the point that Congress
Page 75 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Appendix V Comments From the Department of Health and Human Services
recently addressed the need for regular evaluative information by enacting
GPRA.
Page 76 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Appendix VI Major Contributors to This. Report
Divisior man, Assistant Director
I Venkareddy Chekareddy, Referencer
Page 77 GAOIPEMD- 95- 1 Improving the Flow of Program Information to the
Congress
' Bibliography References on Program Evaluation and Congressional Decision-
Making
Abramson, Mark. The Funding of Social Knowledge Production and Application:
A Survey of Federal Agencies. Washington, D. C.: National Academy of
Sciences, 1978.
BarkdoIl, Gerald L., and James B. BeII (eds.). Evaluation and the Federal
Decision Maker. New Directions in Program Evaluation, No. 41. San Francisco:
Jossey- Bass, 1989.
Bimber, Bruce. Congressional Support Agency Products and Services for
Science and Technology Issues. Report prepared for the Carnegie Commission
on Science, Technology, and Government, Sept. 17,199O.
Boyer, John F., and Laura Langbein. “Factors Influencing the Use of
Health Evaluation Research in Congress.” Evaluation Review, 15: 5
(Oct. 1991), 507- 32.
Boruch, Robert, David Cordray, Georgine Pion, and Laura Leviton. “A
Mandated Appraisal of Evaluation Practices: Digest of Recommendations to the
Congress and the Department of Education.” Educational Researcher, 10~
4 (Apr. 1981), 1031.
Bozeman, Barry, and Julia Melkers (eds.). Evaluating R& D Impacts: Methods
and Practice. Boston: Kluwer Academic, 1993.
Caro, Francis G. “Leverage and Evaluation Effectiveness.”
Evaluation and Program Planning, 3 (1980), 83- 89.
Chehmsky, Eleanor (ed.). “Program EvaIuation: Patterns and
Directions.” American Society for Public Administration, 1985.
Citro, Constance F., and Eric A. Hanushek (eds.). Improving Information for
Social Policy Decisions: The Uses of Microsimulation Modeling, Volume I,
Review and Recommendations. Washington, D. C.: National Academy Press, 1991.
Cook, Thomas D., and William R. Shadish, Jr. “Metaevaluation: An
Assessment of the Congressional Mandated Evaluation System for Community
Mental He& h Centers.” In Gerald L. StahIer and WiIIiam Tash (eds.),
Innovative Approaches to Mental Health Evaluation, pp. 221- 53. New York
Academic Press, 1982.
Page 78 GAO/ F'EMD- 96- l Improving the Flow of Program Information to the
Congress
Bibliography Cronbach, Lee J., and Associates. Toward Reform of Program
EvaIuation. San Francisco: Jossey- Bass, 1981.
Levitan, Sar A, and Gregory Wurzburg. Evaluating Federal Social Programs.
Philadelphia: W. E. Upjohn Institute for Employment Research, Sept. 1979.
Lowi, Theodore J. “American Business, Public Policy, Case Studies, and
Political Theory.” World Politics, 164 (1964), 677- 715.
Martin, Margaret E., and Miron L. St- raf (eds.). Principles and Practices
for a Federal Statistical Agency. Washington, D. C.: National Academy Press,
1992.
Mohr, Lawrence B. Impact Analysis for Program Evaluation. Newbury Park,
Calif.: Sage Publications, 1992.
Rossi, Peter H., and Howard E. Freeman. Evaluation: A Systematic Approach,
4th Ed. Newbury Park, Calif: Sage Publications, 1989.
SaIamon, Lester M. (ed.). Beyond Privatization: The Tools of Government
Action. Washington, D. C.: The Urban Institute Press, 1989.
Striven, Michael. “Hard- Won Lessons in Program Evaluation.” New
Directions for Program Evaluation, 58 (Summer 1993).
Thompson, Mark. Decision Analysis for Program Evaluation. Cambridge, Mass.:
BaUinger Publication, 1982.
Weiss, Carol H. Using Social Research in Public Policy Making. Lexington,
Mass.: Lexington Books, 1977.
Wholey, Joseph S. Federal Evaluation Policy: Analyzing the Effects of Public
Programs. Washington, D. C.: Urban Institute, 1970.
WindIe, Charles, and J. Richard Woy. “From Programs to Systems:
Implications for Program Evaluation Ihustrated by the Community Mental
Health Centers Program Experience.” Evaluation and Program Planning, 6
(1983), 53- 68.
Woods, Patricia. The Dynamics of Congress: A Guide to the People and Process
in the Congress. Washington, D. C: Woods Institute, 1987.
Page 79 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
Bibliography .2s for Case 3grarns
Advisory Committee on Head Start Quality and Expansion. “Creating a
21st Century Head Start.” FinaI Report. Washington, D. C.: Dec. 1993.
Arnold, Judith, Ann Zuvekas, and Nina Teicholz. “A Manual for Using
Health Outcome Measures to Evaluate the Primary Care System.”
Discussion Draft, Feb. 1989. Washington, D. C.: Lewin/ ICF, Task Order #340-
86- 0504.
Freeman, Howard E., KIJ. Kiecolt, and AlIan Harris, II. “Community
Health Centers: An Initiative of Enduring Utility.” Milbank Memorial
Fund Quarterly: Health and Society, 60: 2 (1982), 24567.
Hawkins, Daniel, Jr., and Sara Rosenbaum. “Lives in the Balance: The
Health Status of America's Medically Underserved PopuIations.” Special
report. Washington, D. C.: NACHC, Mar. 1993.
Puma Michael J., et al. Prospects: The Congressionahy Mandated Study of -
Educational Growth and Opportunity- The- Interim Report. Washington, D. C.:
U. S. Department of Education, 1993. Stringfield, Sam, et al. Urban and
Suburban/ Rural Special Strategies for Educating Disadvantaged Children:
First Year Report. Washington, D. C.: U. S. Department of Education, 1994.
U. S. Department of Education. Reinventing Chapter 1: The Current Chapter 1
Program and New Directions. Washington, D. C.: 1993. i
‘This report drew on many other reports that were included in our
review but are too numerous to be listed here. A full list of those reports
is available from the Department of Education's Planning and Evaluation
Service.
Page 80 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
‘1 1, “I “' !i
,, ' 1 ‘i ” I 11
1. : Page 81 GAO/ PEMD- 96- l Improving the Flow of Program Information to
the Congress
1% : ,'
‘I Page 82 GAO/ PEMD- 95- 1 Improving the Flow of Program Information
to the Congresle
:I' ,; / .‘,,
,‘.,, S, ,. T, 8: : ‘-: /, 2,
I, ,I,:
,” ,: Page 83 GAO/ PEMD- 95- 1 Improving the Flow of Program
Information to the Congress
Related GAO Products (978766)
Elementary School ChiIdren: Many Change Schools Frequently, Harming Their
Education (GAO/ HE& 94- 45, Feb. 4, 1994).
Public Health Service: Evaluation Set- Aside Has Not Realized Its Potential
to Inform the Congress (GAO/ PEMD- 93- 13, Apr. 8, 1993).
Exiting Program Improvement (GAOLBRD- 93- 2R, Mar. 30, 1993). Chapter 1
Accountability: Greater Focus on Program Goals Needed (GAO/ HRD- 93- 69,
Mar. 29, 1993).
Improving Government: Measuring Performance and Acting on Proposals for
Change (GAO/ T- GGD- 9% 14, Mar. 23 1993).
Compensatory Education: Difficulties in Measuring Comparability of Resources
Within School Districts (GAO/ nun- 93- 37, Mar. 11, 1993).
Improving Government: Need to Reexamine Organization and Performance (GAO/
T- GGD- 93- 9, Mar. 11, 1993).
Compensatory Education: Additional Funds Help More Private School Students
Receive Chapter 1 Services (GAo/ nnn-%- 65, Feb. 26,1993).
Government Management Issues (GAO/ OCG- 93- 3TR, Dec. 1992). Children's
Programs: A Comparative Evaluation Framework and Five R. histrations (GAO/
PEMDS~~ BR, Aug. 31,1988).
Page 84 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
Congress
*** End of document ***