    Program Evaluation: Improving the Flow of Information to the
    Congress (30-JAN-95, PEMD-95-1).

    Pursuant to a congressional request, GAO provided information on:
    (1) the kinds of information that would be useful to the Senate
    Committee on Labor and Human Resources for program oversight and
    reauthorization reviews; (2) the extent to which agencies
    currently collect and report such information; and (3) a strategy
    the Committee could use to improve its access to agency
    information.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   PEMD-95-1
        ACCNO:   153296
      TITLE:     Program Evaluation: Improving the Flow of Information to
                 the Congress
         DATE:   01/30/1995
      SUBJECT:   Agency reports
    	     Authorizing committees
    	     Communication
    	     Congressional oversight
    	     Congressional/executive relations
    	     Evaluation methods
    	     Program evaluation
    	     Reporting requirements
    	     Statistical data
    	     Dept. of Education Chapter 1 Program for
    	     Educationally Disadvantaged Children
    	     Dept. of Education National Assessment
    	     of Educational Progress
    	     Head Start Program
    	     HHS Community and Migrant Health Centers
    	     Program
    	     HHS Comprehensive Child Development
    	     Program

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Testimony.                                               **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    PEMD-95-1

    Program Evaluation and Methodology Division

    B- 257897 January 30,1995 The Honorable Edward M. Kennedy Ranking Minority
    Member Committee on Labor and Human Resources United States Senate

    Dear Senator Kennedy: This report responds to your request that we assist
    the Committee in identifying and obtaining the information needed to oversee
    and evaluate the programs under its jurisdiction. We review the questions
    that the Committee is likely to find useful for evaluating programs of
    different types. We then explore the extent to which three programs collect
    information pertinent to these questions and report it to the Committee.
    Finally, we suggest a strategy that the Committee might use to identify
    evaluation questions of interest for a given program and to request
    information in advance from agency officials for later use in
    reauthorization or oversight hearings.

    We will send copies of this report to the Secretaries of Education and
    Health and Human Services and to interested congressional committees. Copies
    will also be made available to others upon request.

    If you have any questions or would like additional information, please call
    me at (202) 512- 2900 or Robert L. York, Director of Program Evaluation in
    Human Services Areas, at (202) 512- 5885. Major contributors are listed in
    appendix VI.

    Sincerely yours, Terry E. Hedrick Assistant Comptroller General

    Executive Summary Purpose Committees of the Congress need evaluative
    information to help them

    make decisions about the programs they oversee- information that tells them
    whether, and in what important respects, a program is working well or
    poorly. Concerned that the information it receives from administrative
    agencies is often insufficient, the Senate Committee on Labor and Human
    Resources asked GAO to (1) identify the kinds of information that would be
    useful for oversight and reauthorization review of the types of programs
    under its jurisdiction; (2) examine the extent to which agencies collect and
    report such information; and (3) propose a strategy the Committee could use
    to improve its access to agency information.

    Background The Senate Committee on Labor and Human Resources oversees most
    of the programs administered by the Departments of Education, Labor, and

    Health and Human Services (HHS). These departments fund a wide variety of
    programs, from basic medical research to education services for
    disadvantaged children. The Committee can request information in various
    ways, but it does not automatically receive evaluative information about
    each program as it comes up for legislative action.

    GAO drew from the literature of program evaluation and consulted experts and
    Committee staff to identify categories of information and core questions
    within each category that would be useful for oversight and reauthorization
    review of various programs. Statistics, research, demonstration, service,
    and regulatory programs were included in the review. The information
    categories covered included descriptive information about the programs and
    evaluative information about program implementation, targeting, impact, side
    effects, and comparative advantage. GAO then examined documents and
    interviewed officials of three programs to ascertain which core questions
    were pertinent to each program, what information relevant to those questions
    was available, what information had been requested by and reported to the
    Committee, and how requests and responses had been communicated. The three
    programs, selected with the Committee's agreement, were

    l the Comprehensive Child Development Program (CCDP), which aims to
    demonstrate that providing very young children and their families with
    educational, health, and social services contributes to child development
    and family self- sufficiency; l the Community Health Centers (CHC) program,
    which provides clinical

    services to medically underserved areas or populations; and Page 2 GAO/
    PEMD- 95- 1 Improving the Flow of Program Information to the Congress

    Executive Summary GAO's Analysis Committee staff accorded good,
    appropriately disaggregated descriptive

    information high priority and indicated that it would answer many of their
    questions about a program. Information about implementation is critical for
    oversight of new programs and provisions and for evaluation of programs
    whose activities must be implemented in accordance with professional
    standards if they are to succeed. The relevance of questions about program
    targeting, impact, side effects, and comparative advantage reflects features
    specific to each program and is best determined on a case- by- case basis.

    The programs included in this study offer contrasting examples of
    information requested, collected, and received. The CCDP was designed with
    evaluation in mind and collected a wealth of descriptive information on
    clients and services, as well as data that could answer evaluative questions
    about program implementation and effects. Although the mandated evaluation
    report on the program was still in preparation, some emerging findings about
    CCDP had reached the Committee through various means in 1993 and early 1994.
    The CHC program collected information for program management purposes that
    could be organized to answer questions about service quality, program
    targeting, and other relevant issues. Agency staff used this information to
    answer specific questions from the Committee, but available information had
    not been synthesized and much of it remained unreported. The Chapter 1 ESEA
    program collected little management information from grantees but drew on
    data from a wide range of studies that were summarized in a mandated
    evaluation report in 1993.

    In each of these cases, the Committee obtained the program information it
    requested. It asked for and received information on particular aspects of
    the CHC program. The Congress mandated studies on each of the other two
    programs, and the Committee received reports accordingly. However, the
    information the Committee had initially asked for- once received- did not
    always meet its current needs. For example, information on discrete aspects
    of the CHC program did not convey an overall picture of the program. Both
    the CCDP and the Chapter 1 ESEA evaluation mandates as expressed in the
    statute presented feasibility problems or design constraints that limited
    the usefulness of results. (For example, information on program impact was
    to be presented before new provisions could have been fully implemented.) In
    both cases, the policy issues had also changed somewhat by the time the
    report was due.

    Page 4 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Executive Summary . the Chapter 1 Elementary and Secondary Education Act
    (ESEA) program of

    grants to states, which in turn fund school districts to provide
    compensatory education services in schools serving low- income students.

    Drawing on the broad review of information needs and the case studies, GAO
    framed a strategy the Committee could use to request information from an
    agency in order to evaluate whether a program is adequately serving its
    intended purpose. Of course, the purpose itself may need to be reevaluated.
    However, such reevaluations draw on information from many . sources, not
    primarily from the administering agency, and thus fall outside

    the scope of this report. Results in Brief GAO concluded that a brief list
    of descriptive and evaluative questions

    could capture the agency information most useful for program evaluation.
    Questions can be selected and adapted from this list to fit the program and
    the Committee's purpose in seeking information. Descriptive information is
    essential to the Committee and should be disaggregated to show how
    activities, settings, and clients vary within a program. For oversight
    purposes in the years between reauthorizations, it is important that the
    Committee receive information on progress in implementing new provisions and
    notification of new developments that may require future attention.
    Reauthorization decisions may call for systematic answers to evaluative
    questions about program implementation, targeting, impact, and side effects.
    When associated with policy review, such decisions may also call for
    comparison of the current program to various alternatives.

    Each of the programs GAO studied collected a great deal of useful
    information. However, much of this information was not requested and did not
    reach the Committee. Information that was specifically requested did reach
    the Committee, but much of it was difficult to digest, too highly
    aggregated, or received too late to be useful. Communication between
    Committee and agency staff on information issues was limited and afforded
    little opportunity to build a shared understanding of the Committee's needs
    and how to meet them.

    GAO concluded that obtaining useful information involves selecting pertinent
    descriptive and evaluative questions, explicitly requesting a response, and
    communicating with agency staff to ensure mutual understanding of what is
    needed.

    Page 3 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Executive Summary GAO observed that contacts between Committee and agency
    staff about these programs consisted of a series of one- way communications
    (from the Committee to the agency or the reverse), with little opportunity
    for direct discussion. There appeared to be no current mechanism for .
    adjusting requests in the interest of technical soundness or new priorities,
    or of adjusting reporting plans and formats to increase the timeliness and
    accessibility of the information reported. Communications between the
    executive and legislative branches on evaluative questions are policy-
    sensitive and follow formalized procedures. However, increased communication
    about information needs could be accommodated within these procedures.

    Matters for Congressional Consideration

    This report proposes a strategy for obtaining information to assist with
    program oversight and reauthorization review that the Committee may wish to
    adopt. The three components of this strategy are (1) selecting and adapting,
    from a core list, the descriptive and evaluative questions to be asked about
    a program in interim years and at reauthorization; (2) arranging explicitly
    to obtain timely oversight information in interim years as well as to
    receive results of evaluation studies at reauthorization; and (3) providing
    for increased communication with agency program and evaluation staff to help
    ensure that information needs are understood and that requests and reports
    are suitably framed and are adapted as needs evolve. This strategy can be
    adapted to take institutional realities into account. For example, in view
    of the many demands on its attention, the Committee might select future
    reauthorization questions for some programs and invite agencies to propose
    questions for others.

    Agency Comments Responsible officials from the Departments of Education and
    HHS provided written comments on a draft of this report. (See appendixes IV
    and V.)

    Officials from both agencies concurred with GAO'S suggestion that the
    dialogue between the Committee and the agencies be strengthened at critical
    points in the evaluation process. While generally supporting GAO'S approach,
    agency officials were concerned that it be consistent with the requirements
    of the Government Performance and Results Act (GPRA) and ‘ not
    constitute an added burden. GAO considers its approach to be compatible with
    GPRA and foresees little, if any, added burden on the annual reporting of
    program performance data When the Committee requests information about other
    aspects of a program, such as side effects or comparative advantage,
    consultation should ensure that burden is taken into account as evaluations
    are planned.

    Page 6 GAOfPEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    ' Contents Executive Summary 2 Chapter 1 Introduction

    8 Background Objectives, Scope, and Methodology Types of Programs the
    Committee Oversees Case Study Programs Reviewed Strengths and Limitations of
    Our Study Organization of This Report

    8 9 10

    12 13 14

    Chapter 2 What Information Would Be Most

    Categories of Information and Related Questions Different Forms of Program
    Information and Why They Are

    Needed 15

    16 23

    Useful? Summary Observations 24 Chapter 3 27 What Information Is
    Introduction 27 Available? Three Case Demonstration Program: CCDP 27

    Service Programs: CHC and Chapter 1 ESEA 31 Studies Communicating About
    Information Needs 37

    Concluding Observations 39 Chapter 4 41 A Strategy for Identifying and
    Adapting Core Questions 41

    Arranging for Responses 45 Requesting Evaluation Communicating About
    Information Needs 47 Information Feasibility of Our Approach 48

    Agency Comments and Our Response 49 Appendixes Appendix I: Comprehensive
    Child Development Program

    Appendix II: Community Health Centers Appendix III: Chapter 1 Elementary and
    Secondary Education

    Act 52

    58 64

    Appendix IV Comments From the Department of Education Appendix V Comments
    From the Department of Health and

    Human Services 72

    74 Appendix VI: Major Contributors to This Report 77

    Page 6 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Contents Bibliography 78 Related GAO Products 84 Tables Table 2.1: Core
    Questions by Category of Information

    Table 3.1: Core Questions Adapted to CHC and Chapter 1 ESEA Programs

    26 32

    Table 4.1: Guide to Identifying Needed Information Table I. 1: CCDP
    Evaluation Questions and Related Information Table II. 1: CHC Evaluation
    Questions and Related Information Table III. 1: Chapter 1 ESEA Evaluation
    Questions and Related

    Information 44

    55 61

    68 Abbreviations ACYF Administration on Children, Youth, and Families AFDC
    Aid to Families With Dependent Children BCRR Bureau's Common Reporting
    Requirements BPHC Bureau of Primary Health Care CCDP Comprehensive Child
    Development Program CHC Community Health Centers ESEA Elementary and
    Seconds@ Education Act GPFU Government Performance and Results Act HHS
    Department of Health and Human Services HMO Health maintenance organization
    HRSA Health Resources and Services Administration NACHC NationaI Association
    of Community Health Centers PHS Public Health Service

    Page 7 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    I Chapter 1

    Introduction Congressional committees need information to help them make
    decisions about the programs they oversee- information that tells them
    whether, and in what important respects, a program is working well or
    poorly. Executive agencies provide eva+ ations, studies of the
    implementation of programs, and other reports that can contain information
    helpful to congressional committees. But authorizing committees may not
    automatically receive such information about an existing program as it comes
    up for reauthorization or oversight review and thus may receive insufficient
    program information.

    The then- Chairman of the Senate Committee on Labor and Human Resources
    expressed concern that the Committee was not getting the evaluative
    information it needs to carry out its oversight responsibilities and asked
    us to ascertain what information would be most useful and how this
    information could be obtained from the responsible program agencies.

    Background This authorizing committee oversees programs administered by the
    Departments of Education, Labor, and Health and Human Services. These

    programs perform functions as diverse as funding education for disadvantaged
    children, providing medical care, conducting basic medical research,
    collecting data on American jobs, and enforcing occupational safety
    standards.

    The Committee has used a variety of strategies to request or obtain
    information on these varied programs. For some programs, it has proposed
    (and the Congress has adopted) legislation that identifies specific
    questions to be answered in an evaluation study or annual report. For
    others, legislation has mandated an evaluation study or report of
    activities, but in very general terms. Another legislative strategy has been
    to direct an agency to set aside some portion of its appropriation for
    evaluation, yet permit the department to decide which programs to evaluate
    as well as which questions to ask. The Committee also obtains program
    information by asking questions as they arise or in connection with
    oversight and appropriations processes.

    The Committee can also draw information from reports that all agencies must
    provide, such as appropriation request documents, the Chief Financial
    Officers Act report, and the Federal Managers' Financial Integrity Act
    report. In the future, the Congress will also receive reports on program
    performance and outcome measures under the Government Performance and
    Results Act. Yet despite these different ways of obtaining

    Page 8 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 1 Introduction

    information, the Committee finds that it does not necessarily get the
    information it needs, when it needs it, in a usable form.

    Objectives, Scope, and Methodology

    Objectives and Scope The then- Chairman of the Senate Committee on Labor and
    Human Resources asked us to help the Committee learn more about how to
    evaluate the programs under its jurisdiction. F'irst, the Committee
    requested that we examine the effectiveness of a specific evaluation
    provision, one permitting a set- aside of program funds for evaluation of
    public health programs. Our 1993 report found that these funds were not
    fully utilized and that evaluation results were not synthesized or regularly
    communicated to the Congress, suggesting that other approaches for getting
    information to the Congress were needed. l Second, the Committee asked us to
    identify information needed to evaluate the types of programs it oversees
    and to suggest how it might request such information from the agencies.

    This report addresses the second task. Based on the letter of request and
    discussions with Committee staff, we identified three report objectives: (1)
    to identify the kinds of information that would be useful to the Committee
    for oversight and reauthorization review of the various types of programs
    under its jurisdiction, (2) to examine the extent to which the agencies
    currently collect and report such information, and (3) to propose a strategy
    the Committee could use to improve its access to useful information from the
    agencies.

    Our objective was to help the Committee obtain agency information about the
    performance of existing programs, to assist them with decisions such as how
    to refocus or improve an existing program that meets an evident need. Of
    course, decisions about a program's future may involve fundamental policy
    questions- such as the question of whether program continuation is warranted
    in light of social and demographic changes, current budget conditions, and
    policy priorities. Information to resolve such questions comes to the
    Committee from a wide range of sources, not primarily from the administering
    agency. Since our concern was with how

    %ee Public Health Service: Evaluation Set- Aside Has Not Realized Its
    Potential to Inform the Congress (GAO/ PEMD- 93- 13; Apr. 8, 1993).

    Page 9 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 1 Introduction

    to obtain information from administrative agencies, we touch on the
    fundamental policy questions only briefly. However, readers should bear in
    mind that broad- based policy evaluation often takes place prior to or in
    tandem with the evaluation of program performance.

    Methodology To identify the kinds of information that would be most useful,
    we (1) consulted the literature and experts familiar with the evaluation of
    public programs of various types, (2) asked Committee staff members what
    information they and Committee members would most like to have for oversight
    and reauthorization, (3) reviewed statutory and other requests for
    information about our case- study programs, and (4) compiled a list of core
    questions drawing from all of the foregoing. In this way, we identified
    categories of information that are useful for evaluation purposes as well as
    for ongoing oversight activities.

    To learn how much of this information agencies collect and report, we
    narrowed our scope to conduct in- depth case studies of three programs
    selected after consultation with the Committee. (These programs and the
    selection criteria are described in more detail later in the report.) For
    each program, we reviewed documents and conducted interviews with agency
    staff to inventory what information was available to answer core questions
    pertinent to that program. We also ascertained from Committee aud agency
    staff what information the Congress had requested about each program, what
    Committee and agency communication had occurred around these requests, and
    what the Committee had received. We did not assess the quality of the
    agencies' data.

    To frame a strategy the Committee could use to request information, we drew
    upon our observations concerning major gaps in the information the Committee
    receives and shortcomings in communications between the Committee and
    program agencies. Responsible officials from the Departments of Education
    and HHS commented on our findings at exit conferences and provided written
    comments on a draft of this report. (See appendixes IV and V.) We conducted
    our work from July 1993 to October 1994 in accordance with generaLly
    accepted government auditing standards.

    Types of Programs the Our first objective required that we classify the
    programs under the Committee Oversees Committee's jurisdiction by program
    type, using a typology that had

    relevance for evaluation. Our review of public policy and program Page 10
    GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the Congress

    Chapter 1 Introduction

    evaluation literature found that a program's purpose is central to any
    consideration of how to evaluate its effectiveness or worth. 2 Five major
    program purposes- statistics, research, demonstration, service, and
    regulation- appeared relevant to the work of the Committee. We describe the
    five here as ideal types.

    Statistics programs compile and analyze data and disseminate information,
    typically on a recurring basis. They develop indicators from data collected
    for administrative, enforcement, or statistical analysis purposes relevant
    to public policy issues such as health care, employment, or education.
    Programs of this type under the Committee's jurisdiction include the Bureau
    of Labor Statistics and the National Center for Education statistics.

    Research programs primarily aim to develop new knowledge or increase
    understanding of the subject studied. Research may be conducted in- house by
    agency personnel or by individuals, research institutions, or other
    organizations supported by grants or contracts. Examples of research
    programs are the National Institute on Drug Abuse and the Research and
    Demonstration Center program of the Office of Educational Research and
    Improvement.

    Demonstration programs are defined here as those that aim to provide
    evidence of the feasibility or effectiveness of a new approach or practice.
    They may be undertaken to learn about the suitability of a particular
    approach under real- world conditions, to try out a variety of approaches
    and examine their effects, or to develop a new approach. 3 Examples are the
    Jobstart demonstration program and the Comprehensive Child Development
    Program.

    Service programs provide services, directly or indirectly, to users or
    clients. Not only are there many different kinds of service (human,
    organizational, informational, fmancial), but they are delivered and funded
    through a variety of mechanisms, making service the most varied of the
    program types. Some programs deliver a specific service (such as student
    loans for postsecondary education) while others (such as the Community

    2Some public policy literature classifies programs sccording to the
    “tool” through which they are carried out (for example, through
    a government agency, project grant, formula grant, loan, loan guamntee, or
    regulation) or by their maturity (whether the program is new or
    establiihed). Program purpose is the basis for our typology, but the other
    dimensions are discussed where relevant

    3The term “demonstration” may also refer to activities whose
    purpose is simply to provide an example of an approach In our view, such
    purely illustrative demonstrations are most appropriately classified (for
    these purposes) as service pro- since they mainly do not result in
    evaluative conclusions about sn approach

    Page 11 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 1 Introduction

    Services Block Grant program) support a broad range of services that may
    vary across providers or locations. Government agencies may make grants or
    loans directly to recipient individuals or organizations, to service
    providers, or to states (which in turn make subgrants to providers).

    Regulatory programs are intended to protect people from harm or from a
    violation of their rights by influencing the behavior of individuals or
    organizations. Some regulations prescribe actions (such as requiring
    employers to follow certain workplace safety practices) and others prohibit
    actions (such as employer discrimination against employees on the basis of
    age). Federal agencies (directly or through the states) adopt rules or
    standards of behavior and compel or coax public compliance with the
    regulation. Examples are the Occupational Safety and Health Administration
    and the Department of Education's Office of Civil Rights programs.

    We reviewed programs overseen by the Committee and found that we could
    classify them according to these types.* To do so, we examined the programs'
    actual functions, not simply the title or label given to a program by
    enabling legislation. (For example, if a “demonstration” program
    focused on providing services rather than on gathering evidence about what
    was demonstrated, we treated it as a service program.) In addition, we
    observed that some programs have a secondary purpose that complements their
    primary purpose, such as a statistics program that is intended not only to
    produce data, but also to perform the service of disseminating data to
    clients. Such programs would need to be evaluated in terms of each relevant
    purpose.

    Case Study Programs To address our second objective and assess the extent to
    which potentially Reviewed useful descriptive and evaluative information was
    collected and reported

    by federal agencies, we conducted case studies of three programs the
    Committee oversees. The following three programs were selected, in
    consultation with the Committee staff, from a list of programs due to be
    reauthorized:

    . the Comprehensive Child Development Program, which aims to demonstrate the
    effectiveness of various approaches to providing very young children and
    their families with educational, health, and social services;

    40ur typology is comprehensive for a. ll programs under this Committee's
    jurisdiction. It does not cover other possible program types, such as cash
    assistance programs, entitlement programs (such as Social Security), or tax-
    expenditure programs.

    Page 12 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congresr!

    Chapter 1 Introduction

    l the Community Health Centers program, which funds centers that provide
    clinical services to medically underserved areas or populations; and l the
    Chapter 1 Elementary and Secondary Education Act program of grants

    to states, which in turn fund school districts to provide compensatory
    education services in schools serving low- income students.

    The Chapter 1 ESEA program is administered through the Department of
    Education, and the other two programs are administered through HHS. (For
    program details, see appendixes I- III.)

    This portfolio of one demonstration and two service programs covers only two
    of our five program types. However, our three cases offer the following
    advantages:

    . All three areas- health, education, and early childhood development- are
    high on the current policy agenda . Information about a demonstration
    program and its evaluation is

    particularly important, since a demonstration (such as CCDP) provides the
    Congress with an opportunity to learn the effects of a proposed new program
    on a small scale before committing to major implementation. l Service
    programs constitute much of the Committee's workload and

    funding authority. Our cases represent two major variants- one provides
    federal grants to the states, which in turn fund local providers of services
    (Chapter 1 ESEA), and the other provides grants from the federal government
    directly to service providers (CHC).

    Strengths and Limitations of Our Study

    The major strength of our study is that, in identifying this Committee's
    evaluation information needs, we considered the structure of the legislative
    process, the nature of the programs, and the agencies' program
    administration practices. Both the Congress and federal agencies could find
    this perspective useful in developing both better information and a better
    match between evaluative needs and the information provided.

    Another strength of the study is our examination of the communication
    process between this Committee and the three agencies it oversees, including
    how that process affected what information was reported and when it was
    received. Our report also provides detailed information on the three
    programs, information for which there is both an immediate and a longer term
    need.

    Page 13 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Organization of This Report

    Chapter 1 Introduction

    A limitation of this report is that the three programs, while illustrative,
    are not representative of all programs. The case studies exemplify the
    importance of, and barriers to, gathering (or not getting) program
    information but do not, of course, reflect all possible types of problems.
    In addition, observations based on the study of a particular Senate
    authorizing committee at a particular point in time may not reflect the
    information needs and procedures of other Senate or House authorizing
    committees or of appropriations committees. However, we are confident that
    the general strategy we propose could be readily adapted to cover other
    policy mechanisms (such as tax incentives) and evaluation questions (such as
    whether there is still a demonstrable need for the program) that we did not
    specifically address.

    Chapter 2 describes the kinds of information we found to be of most
    potential use to the Committee, as well as how the core questions to be
    asked vary by program type. Chapter 3 summarizes the information available
    for each of our three case programs and examines congressional requests for
    information and how agencies responded to them. Chapter 4 proposes a
    strategy the Committee staff could use to obtain useful evaluative
    information in the future.

    Appendixes I- III contain detailed information on the three programs
    reviewed in chapter 3. Each appendix includes a brief description of the
    program and a table listing evaluation questions, specific information
    needed to answer those questions, and an assessment of whether the needed
    information was available. Appendixes IV and V reproduce the comments from
    the Departments of Education and HFIS on a draft of this report.

    Page 14 GAOREMD- 95- 1 Improving the Flow of Program Information to the
    Congresa

    Chapter 2 What Information Would Be Most Useful?

    Our lit- t task was to determine what kinds of information would be most
    useful to the Committee as it evaluates various types of programs for
    possible reauthorization and oversees their progress during interim years.
    Our approach to this task included reviewing the literature on public
    program evaluation and congressional oversight and interviewing experts
    familiar with the evaluation questions associated with different types of
    programs as well as congressional requests for information about such
    programs1 We discussed the results of this inquiry and their own views with
    senior members of the Committee staff with responsibilities in each of
    several program areas and obtained their assessments of the kinds of
    information they and Committee members would most like to receive.

    From this approach, we identified priority categories of evaluative
    information, compiled a list of core questions connected with each category,
    and noted the kinds of data or other material that would be needed to
    respond to these questions. We examined the relevance of the various core
    questions to programs of different types. Finally, we identified different
    forms of information that are useful for different purposes, such as
    oversight, reauthorization, and policy change.

    Our inquiry focused primarily on agency- provided information that could
    help the Committee evaluate how well an existing program is working. Such
    information is helpful in deciding whether a program is adequately serving
    its intended purpose and if so, what level of funding is appropriate. Of
    course, the purpose itself may need to be reevaluated: changing conditions
    may have rendered it irrelevant or decreased the need for the program.
    Because such reevaluations draw on information from outside the program
    agency, they fall outside the scope of this report and we do not discuss
    them specifically. However, guidelines to identifying the information needed
    for such studies can be found in the general literature of evaluation and in
    other GAO reports2 The general strategy we outline in chapter 4 should
    assist the Committee in obtaining information concerning the need for a
    program, as well as the other categories of information outlined in the
    remainder of this chapter.

    ‘We interviewed program specialists from the four congressional
    support agencies Congressional Research Service, Office of Technology
    Assessment, Congressional Budget Office, and GAO.

    2For a discussion of criteria for evaluating the need for a program, see
    Children's Programs::‘ A Comparative Evaluation Framework and Five
    Illustrations (GAO/ PEMlN? Z- 28BR; Aug. 341988).

    Page 15 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information Would Be Most Useful?

    Categories of Information and Related Questions

    The main congressional evaluation questions we derived can be broadly
    categorized as seeking descriptive information about the program,
    information that can be used to evaluate its implementation, and information
    about various effects. These three broad categories of information are
    generally applicable to all five of the program types we identified.
    However, the relative priority of different categories and questions varied
    somewhat from type to type as did the particular manner in which our core
    questions were phrased. We describe our information categories and core
    questions next.

    Program Description Descriptive information tells what the program is and
    what it does. Descriptive questions include the following: What activities
    does the program support? Toward what end or purpose? Who performs these
    activities? How extensive and costly are the activities, and whom do they
    reach? Are conditions, activities, purposes, and clients fairly similar
    throughout the program, or is there substantial variation across program
    components, providers, or subgroups of clients?

    The descriptive information that such questions elicit is not itself
    evaluative and receives relatively little emphasis in the program evaluation
    literature. It is important for evaluation, nonetheless, and was accorded
    high priority by the Committee staff we interviewed. Descriptive narration
    and statistics provide the foundation for identifying key evaluation
    questions and interpreting the evaluative information concerning the
    program. Better descriptive information would answer many of the Committee's
    questions about the program and thus would enable the Committee to focus its
    oversight and evaluation efforts more effectively.

    Committee staff indicated that they typically do not get sufficient
    descriptive information to understand the variety of conditions under which
    a program actually operates and how federal funds are actually, being used.
    (General summaries of activities and programwide totals of actions performed
    or clients served, such as accompany appropriation requests, do not meet
    this need.) Nor do they typically receive site- specific observational data
    to convey the “flavor” of a program as participants see it (for
    example, to help them understand how a student's experiences in federally
    supported bilingual education classes relate to the rest of his or her
    school day). Such information, they told us, is essential for understanding
    how well a program is working and thus is relevant to the Committee even
    when site- level conditions are a state or local responsibility.

    Page 16 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information WouM Be Most Useful?

    Our review and interviews suggested that descriptive information is useful
    for every type of program, be it concerned with statistics, research,
    demonstration, service, or regulation. The nature and amount of information
    needed vary with such program features as internal diversity and contact
    with the public. For example, the typical summary description may provide a
    sufficient basis for understanding a statistics program or a demonstration
    that performs a uniform set of activities in a uniform way in . one or a few
    sites. Much more information, appropriately disaggregated, is

    needed to understand a demonstration, service, or regulatory program that
    operates through diverse providers who perform different mixes of activities
    under diverse conditions (for example, in schools, clinics, or workplaces
    ranging from the very large and urban to the very small and IWd).

    Implementation Our second category concerns information about program
    implementation- specifically, about how and to what extent activities have
    been implemented as intended and whether they are targeted to appropriate
    populations or problems.

    Extent and Nature of Implementation Questions about the extent and nature of
    program implementation are

    concerned with procedural issues- that is, with how program activities are
    carried out. Relevant questions include the following: (1) Are mandated or
    authorized activities actually being carried out? (2) Are the activities in
    accordance with the purpose of the law and implementing regulations? (3) Do
    they conform to the intended program model or to professional standards of
    practice, if applicable? (4) Are program resources efficiently managed and
    expended? 4

    These implementation questions typically arise in the years immediately
    after a new program or provision is authorized, as well as at
    reauthorization. With respect to new provisions, our respondents were
    interested in learning whether acceptable progress had been made in putting
    new activities into practice, whether significant feasibility or management
    problems had arisen, and whether program modifications were needed as a
    result. Information obtained through the administering

    3Thii and the preceding question refer to what is sometimes called
    “program fidelity” in the evaluation literature.

    4F'rogram evaluations and reauthorization discussions typically do not focus
    on management issues unless there is evidence of problems with the program
    in question. Our review accordingly did not focus on such issues or the
    information needed to address them However, we consider it useful for the
    Committee to ask whether there is evidence of such problems and, if so, to
    request follow- up information.

    Page 17 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information Would Be Most Useful?

    agency's standard reporting and monitoring procedures may be sufficient to
    answer these interim- year questions and to identify feasibility or
    management problems that arise in connection with ongoing activities. For
    reauthorization, interest focused on questions concerning conformance to
    program purposes, model specifications, or professional standards.

    Descriptive data obtained through routine reporting, if designed with this
    purpose in mind and known to be sufficiently reliable, may provide a
    sufficient basis for comparing program activities against program
    requirements5 However, special data collection- obtained through program
    monitoring or through external studies- may be necessary to ensure the
    objectivity and detail required to explore issues of program quality.

    Our review indicated that most of the implementation questions we have
    discussed are pertinent to all of the types of programs we considered.
    However, there are some differences with respect to the question of
    conformance to professional standards. This question not only applies but
    also is critical to the evaluation of two of the types of programs we
    reviewed: statistics and research. Well- established professional standards
    apply to these types of programs, and procedures and products that do not
    meet such standards will not be considered credible. 6 Applying these
    standards, an evaluation of the National Center for Education Statistics
    should ask whether the education indicators generated were free from bias
    and technically sound and whether data collection methods, analysis methods,
    and limitations of the data were fully disclosed. Expert judgments commonly
    are used to provide the needed evaluative information.

    Obtaining information on the extent to which prescribed procedures were
    followed is also especially important for demonstration projects. If a
    prescribed approach was to be demonstrated, the Committee will want to know
    whether the approach was implemented as planned and, if not, how

    5Client counts and other monitoring data, dependent as they are on the
    limited resources that program offices or grantees typically can devote to
    data collection, may well be imprecise. The cost of achieving precision is
    considerable, and often outweighs the benefits. In such instances, the
    agency should make a systematic estimate of the extent of uncertainty in
    their numbers and then present this estimate along with the numbers
    themselves.

    ‘% 3iteria to be met by federal statistical agencies have been set
    forth by the National Research Council. See Margaret E. Martin and Miron L
    Straf (eds.), Principles and Practices for a Federal Statistical Agency
    (Washington, D. C: National Academy Press, 1992). The Council's report
    emphasizes that statistical agencies' work will have the necessary
    credibility only if it ls impartial, tech& ally sound, and fully described.
    Similar professional criteria apply to research.

    Page 18 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Targeting Chapter 2

    What Information Would Be Most Useful? it was modified and why modifications
    were deemed necessary. Similarly, determining how activities and
    arrangements differ across projects is necessary in order to assess those
    demonstration programs designed to encourage and compare alternate
    approaches.

    For regulatory and service programs, the applicability of models or
    standards and the importance of evaluating conformance to them must be
    examined on a case- by- case basis. Some regulatory and service programs
    emphasize the importance of following specific procedures (such as medical
    procedures or procedures for gathering evidence of compliance with
    regulatory requirements), while other programs involve activities for which
    standards are not available or have not been specified. 7 (See our case
    studies of service programs in chapter 3.)

    Targeting has to do with the objects (problems or populations) at which a
    program's activities are directed and the coverage that these activities
    achieve. Many programs under the Committee's jurisdiction are targeted to
    particular populations or problems (such as migrant students or unsound
    management of pension plans). Others are directed to a broad area (such as
    research on infectious diseases), from which the agency is expected to
    select high- priority problems or clients (such as a particular disease that
    endangers children or the elderly). In either case, it is useful for the .
    Committee to know what kind of coverage a targeted program has achieved.
    Relevant questions include the following: (1) Have program efforts focused
    on appropriate problems? (2) Has the program reached the appropriate people
    or organizations? (3) Do current resources and targeting practices leave
    significant needs unmet (that is, priority problems unaddressed or priority
    clients unserved)?

    Assessing a program's targeting success typically requires not only good
    measures of program activities and data on the clients served, but also
    information about the size and distribution of the eligible population (for
    target groups) or problem area as a whole. Thus, this assessment may require
    a special data collection activity or access to sources of data outside the
    program (such as the decennial census).

    The question of whether appropriate problems are targeted is critical to
    statistics programs (which should address policy- relevant issues) and
    research programs (which should address important gaps in knowledge). For
    other types of programs, the importance of targeting and the particular

    ‘Our focus here is on standards for conducting the service or
    regulatory activities central to the purpose of the program. Federally
    funded programs of every type are expected to meet financial and certain
    other management standards.

    Page 19 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information Would Be Most Useful?

    questions to be asked must be determined on a case- by- case basis.
    Questions for programs of different types are illustrated below.

    . Statistics: Are Bureau of Labor Statistics indicators relevant to current
    issues of public policy? To what extent have they reached key federal,
    state, and private sector policymakers? Are there important issues or
    populations on which labor statistics are not being collected? * Service:
    What proportion of the students served by the migrant education

    program are currently migrant (that is, have moved within the past 5 years)?
    What proportion of recent migrants are not served? 0 Regulatory: Are
    Occupational Safety and Health Administration

    enforcement efforts targeted to the safety problems that pose the greatest
    dangers or that endanger the greatest number of people?

    Program Effects We found it useful to distinguish three different aspects of
    program effects: (1) whether the program is achieving its intended purposes
    or outcomes (impact), (2) whether it has other important effects that relate
    to congressional concerns (side effects), and (3) how it compares with
    alternative strategies for achieving the same ends (comparative advantage).
    We address each of these aspects below.

    Impact Impact questions center on whether program activities actually
    resulted in the improvements the program was designed to produce, as well as
    on what progress was made toward achieving the program's goals. Where
    programs or demonstrations are designed to produce changes as a result of
    program activities, assessing a program's impact is often central to
    reauthorization deliberations. For example, a job training program is
    expected to show that participation in the program led to significantly
    higher income or more stable employment. Where impact was expected, our
    respondents were interested in learning, for reauthorization purposes, about
    (1) the aggregate impact of the program; (2) how impact or outcomes varied
    across participants and approaches; and (3) how impact varied across
    providers- specifically, whether the program was supporting providers whose
    performance was consistently weak.

    Good evaluative information about program effects is difficult to obtain.
    Each of the tasks involved- measuring outcomes, ensuring the consistency and
    quality of data collected at various sites, establishing the causal
    connection between outcomes and program activities, and separating out the
    influence of extraneous factors- raises formidable technical or logistical
    problems. Thus, evaluating program impact

    Page 20 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information Would Be Most Useful?

    generally requires a planned study and, frequently, considerable time and
    expense. Program features affect the relative difficulty of getting good
    impact information. The more varied the program activities, and the less
    direct the connection between the provider and the federal agency, the
    greater the likely difficulty of getting comparable, reliable data
    concerning clients and services. For example, a federal agency whose own
    employees deliver a specified service is likely to be able to obtain impact
    data more easily than one that administers grants that states then pass on
    to a variety of local entities to be used in any of several ways. Also, due
    to the absence of a contrasting comparison group, it is practically
    impossible to estimate the impact of a long& anding program that covers all
    who are eligible.

    It is critical to obtain impact information from demonstration programs
    intended to test whether an approach (or any of several approaches) can
    obtain results, as well as from regulatory or service programs that are
    intended to produce specified kinds of outcomes or changes. However, impact
    questions are not commonly directed at statistics programs (which are
    designed to produce a product rather than a change). Nor are they directed
    at programs that regulate or offer a service when it is considered certain
    that the desired outcomes will occur as a result of program activities (for
    example, that a vaccine, properly administered, will produce immunization or
    that safety regulations, properly enforced, will reduce workplace injuries).
    Information showing that activities have been soundly implemented and have
    reached a high proportion of the relevant client population may be
    sufficient to justify a decision to continue such programs.

    The purpose of research programs is to produce new knowledge, either basic
    or applied. The impact of basic research programs depends on whether the
    projects, in the aggregate, have added to the knowledge in their fields by
    eliminating or confirming hypotheses (for example, hypotheses concerning
    brain chemistry or human perception and cognition). Applied research, on the
    other hand, is expected to result in the development of new or improved
    products (such as better medications or new approaches to the teaching of
    reading). If it achieves its intended immediate impact, either type of
    research program may ultimately have a much broader social impact (such as a
    reduction in the incidence of alcoholism or a rise in educational
    achievement levels). However, it would be inappropriate to evaluate either
    type of program primarily in terms of such ultimate impacts. The influence
    of basic research on either product development or practice is hard to trace
    and may not be evident for many

    Page 21 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information Would Be Most Useful?

    years, and products or practices may fail to gain widespread use for reasons
    that are beyond their developers' control.*

    Side Effects Public programs of any type may have important (and sometimes
    unforeseen) effects beyond those they were intended to produce. These side
    effects, which were of interest to our Committee respondents, should be
    noted in any comprehensive evaluation. For example, a program might have
    unforeseen effects- either positive or negative- on the problem or clients
    it was designed to address. Or, a new program might have an effect on other
    programs aimed at a similar problem or population.

    Information to identify unforeseeable side effects is hard to plan for
    except by encouraging reflective observation and maintaining open channels
    of communication between program administrators and clients. However, some
    kinds of side effects can be foreseen- indeed, possible side effects may
    well come up in debate when a program is authorized- and attention to them
    built into the program's data collection plans. For example, a program might
    arrange from the outset to collect data that would enable it to determine
    whether its procedures were unintentionally resulting in unequal treatment
    of similarly situated individuals. Or, a service program that refers clients
    to another program might arrange to monitor the level of demand placed on
    the other program in order to ascertain whether meeting this demand was
    impeding that program's ability to meet its own goals.

    Comparative Advantage Finally, evaluation may focus on the comparative
    advantage of continuing the current program. Typical comparative advantage
    questions include the following: (1) are the effects gained through the
    program worth its financial and other costs? and (2) taking both costs and
    effects into account, is the current program superior to alternative
    strategies for achieving the same goals? Options against which the current
    program might be compared could include:

    l discontinuing the program; 9 consolidating it with other federal programs
    that serve a similar purpose; 0 utilizing a different type of program (such
    as tax incentives rather than

    regulation) to address that purpose; . transferring responsibility for
    program decisions to the state or local level

    of government (for example, by replacing a federal categorical service
    program with block grants to support locally determined services); or

    *For a comprehensive recent treatment of the subject, see Barry Bozeman and
    Julia Melkers (eds.), Evaluating R& D Impacts: Methods and Practice (Boston:
    Kluwer Academic, 1993).

    Page 22 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information Would Be Most Useful?

    l transferring the function to the private sector. Comparative advantage
    questions typically arise when programs are up for reauthorization. Our
    Committee respondents indicated that such questions were of special interest
    in the current climate of budget pressure (which may force choices) and of
    major policy change (which may affect all programs in a part&& r policy
    area).

    Comparison of the current program to alternatives raises special challenges
    for information collection. It requires not only good, comprehensive data on
    the current program, but also truly comparable data concerning alternatives
    or policy options. Simulation data or other forms of data based on
    projection may be needed when considering options for which there is no
    directly relevant experience. Because of their complexity, comparative
    studies are likely to require considerable lead time, careful planning, and
    special arrangements for implementation.

    Different Forms ,of We observed from our interviews that program information
    serves several Program Information different functions for the Committee-
    functions that call for the receipt

    of this information at different times and in different forms. The primary
    and Why They Are functions of program information are to (1) signal the need
    for attention to Needed a program in the interim years between
    reauthorizations, (2) guide

    reauthorization decisions, and (3) assist the Committeein deciding whether a
    major policy change should be recommended. We refer to the corresponding
    forms of information as (1) notification, (2) answers, and (3) evidence of
    comparative advantage.

    Notification That Attention Our Committee respondents emphasized that it was
    very important that Is Needed they be kept informed of significant
    developments on an ongoing basis and

    particularly that they receive early notification when a problem or the need
    for change in a program becomes evident (for example, notification that a
    new provision is having unwelcome side effects or that shifting needs have
    rendered program targeting obsolete). Such notification helps the Committee
    focus its efforts on the programs that need attention, avoid unpleasant
    surprises, and recognize successful program sites or practices.

    Our interviews with Committee and agency staff revealed that developments
    that may merit notification come to their attention in a number of ways,
    often as a by- product of program management. Initial clues to such
    developments can emerge from observations made during

    Page 23 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 2 What Information Would Be Most Useful?

    site visits, from the review of applications or of program performance
    measures, from concerns that site officials raise in meetings, from
    conversations with clients, or from media coverage of a local program. Since
    a single clue may reflect erroneous information or represent an isolated
    instance, other evidence is needed to verify the existence and importance of
    the development suggested by the clue. The program managers we interviewed
    could draw on a variety of resources- networks and data sources, as well as
    their own expert knowledge of the program- to make such a verification and
    to judge whether there was sufficient evidence to justify notice to the
    Committee.

    The varied evidence that supports managers' judgment that something worthy
    of notice has occurred is not generally thought of as a form of evaluation
    information. This may be because the evidence on which notice is based is
    rarely sufficient to support evaluative conclusions about a program.
    However, this form of evidence is sufficient to identify evaluation
    questions to which the Committee will want answers in the future. We
    therefore inquired about information and notification concerning current
    developments in our case studies, and we report our findings in chapter 3.

    Answers to Guide At times when major reauthorization decisions are being
    made, the Reauthorization Decisions Committee needs answers concerning
    evaluation questions such as those

    just discussed. Answers arise out of systematic evaluation studies. These
    studies must be planned and budgeted for in advance, information relevant to
    the questions the Committee will face must be gathered, and findings must be
    reported in time to inform deliberation.

    Comparative Advantage Information for Policy Decisions

    At times, the program under review concerns a policy area that is being
    reexamined. In this event, the Committee may need information that evaluates
    the current program against alternative strategies (including dropping the
    program if it cannot be justified in terms of cost- effectiveness or need).
    As previously noted, such comparative studies require data beyond that which
    the administering agency can provide and raise complex analytic (as well as
    policy) issues. Thus, these studies should be planned and arranged for well
    in advance.

    s= w Observations Our interviews and reading of the literature led us to
    three conclusions

    concerning what information would be useful, Page 24 GAO/ PEMD- 95- 1
    Improving the Flow of Program Information to the Congress

    Chapter 2 What Information Would Be Most Usefbl?

    1. Descriptive information is more useful to the Committee than has
    generally been realized. Such information should be sufficiently specific,
    detailed, and disaggregated to convey an in- depth understanding of the
    program.

    2. In the interim years between reauthorizations, it is useful to the
    Committee to obtain information on progress and problems in implementing new
    provisions, as well as to be notified of developments that raise new
    evaluation questions.

    3. Reauthorization decisions (including those associated with major changes
    of policy) call for information that provides answers to evaluation
    questions concerning program implementation and effects and the comparative
    advantage of continuing the program or pursuing other options. Advance
    planning is generally needed to identify questions relevant to each program
    and collect the data needed to answer them.

    We further observed that the questions that it might be useful to ask,
    expressed in general form, comprise a relatively short list. We have listed
    these questions- grouped according to whether they concern program
    description, implementation, targeting, impact, side effects, or comparative
    advantage-- in table 2.1. For any given program, some of these questions
    (but probably not all) will be relevant. Knowledge of program type is of
    some help in narrowing the list, but features specific to the program (such
    as expectations concerning targeting) must also be taken into account.

    Page 25 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 2 what Information Would Be Most Useful?

    Table 2.1: Core Questions by Category of Information Category of

    information Core auestion Description Overall, what activities are
    conducted? By whom? How extensive

    and costly are the activities, and whom do they reach? If conditions,
    activities, and purposes are not uniform throughout the program, in what
    significant respects do they vary across croaram conaoonents, oroviders, or
    subaroucs of clients?

    implementation What progress has been made in implementing new provisions? a
    Have feasibility or management problems become evident? a

    Targeting If activities and products are expected to conform to professional

    standards or to program specifications, have they done so? Have program
    activities or products focused on appropriate issues or problems?

    To what extent have they reached the appropriate people or organizations?

    Do current targeting practices leave significant needs unmet (problems not
    addressed, clients not reached)?

    Impact Overall, has the program led to improvements consistent with its
    purpose?

    If impact has not been uniform, how has it varied across program components,
    approaches, providers, or client subgroups?

    Are there components or providers that consistently have failed to show an
    imoact?

    Side effects Comparative advantage

    Have program activities had important positive or negative side effects,
    either for program participants or outside the program? a

    Is this program's strategy more effective in relation to its costs than
    others that serve the same rourroose?

    alnformation important for interim years. We used these core questions to
    categorize and summarize the information that was available from the three
    programs that we used as case studies, as reported in chapter 3 and in the
    case study appendixes. In chapter 4, we suggest how the Committee might draw
    from this list to request interim- year information and to frame evaluation
    questions for reauthorization review of programs of different types and
    features.

    Page 26 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chanter 3 What Information Is Available? Three Case Studies

    Introduction The second objective of this study was to determine to what
    extent information is actually available to answer oversight and
    reauthorization

    questions pertinent to various programs. To address this objective, we used
    one demonstration program and two contrasting service programs as case
    studies.

    We identified core questions (described in chapter 2) pertinent to each
    program based on our interviews and our review of enabling legislation and
    congressional requests. Next, we reviewed all the information available on
    the programs (from site reports and management information systems to formal
    studies) to determine what information was available on core evaluation
    questions (as adapted to fit each program), how much of this information
    made its way to the Committee staff, and when and in what form it had
    reached the staff. We also interviewed Committee and agency staff to learn
    how adequately the Committee members and staff felt that they had been
    informed, and what additional communication occurred as the agency sought to
    interpret and respond to the Committee's requests.

    Our three cases offer richly contrasting examples of information requested,
    available and reported. The Comprehensive Child Development Program is a new
    demonstration program whose design included explicit provision for the
    collection of descriptive and evaluative information, Chapter 1 of the
    Elementary and Secondary Education Act is a long- standing education service
    program that collects minimal data from grantees but has been studied
    extensively through several mandated evaluations, the most recent of them
    reported to the Congress in 1993. The Community Health Centers program
    maintains extensive data on program operations but has not been asked for,
    nor has it provided the Committee with, comprehensive evaluation or program
    reports for years. Taken together, the three cases offer considerable
    insight into what information is collected and how it reaches (or fails to
    reach) the Committee.

    Demonstration Program: CCDP

    Core Questions About CCDP

    The Comprehensive Child Development Program, which originated in 1988, was
    designed to demonstrate whether providing comprehensive and continuous
    services to very young low- income children, their parents, and

    Page 27 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available7 Three Case StUdieS

    family members enhances the child's well- being as well as his or her
    physical, emotional, and intellectual development. ' The various sites
    funded through the program offer certain services in common- parent
    education and training, case management, early childhood development
    intervention, health care, child care, and family and child needs
    assessment. However, sites vary considerably in the way they deliver these
    services and in the specific objectives emphasized. (For example, some sites
    focus on helping clients with day- today basic survival, while others also
    provide preventive counseling.)

    The descriptive, implementation, and effects questions for oversight and
    reauthorization of the CCDP program are as follows: What is the basic model
    or approach tested here? How are the centers different or similar in terms
    of the local population, geographic location, and service provision? What
    are some of the start- up and implementation difficulties the centers faced?
    What are the principal program impacts? Have there been beneficial or
    detrimental side effects to this program or any aspect of it? 2

    Available Information Relevant to These Questions

    On the whole, we found that CCDP was designed (through legislation and
    administration) to collect the categories of information critical to the
    evaluation of this demonstration program, and it has done so. The
    Administration for Children, Youth, and Families (ACYF), which administers
    the program, has descriptive and evaluative information to answer the core.
    questions we identified.

    Many sources provide the agency with useful information about the CCDP.
    Between the program's management information system and observational
    reports, the agency has the raw material from which to construct both
    quantitative and qualitative descriptions of the CCDP approach as it is
    demonstrated across the centers, as well as to measure CCDP'S impact and
    document other effects. (See table I. 1 for more details on the data
    available.)

    Description ACYF has rich descriptive information on program operations,
    clients, and settings for the approximately 34 centers. The agency's
    information system includes standardized information from each center on
    client needs as well as on what services are offered and how often they are
    provided. Information is also collected on client and community
    characteristics-

    ‘Although the enabling legislation did not refer to CCDP as a
    demonstration program, the agency has collected implementation and impact
    data to meet the information needs of a demonstration program

    %ee appendix I for more information on the program and what specific
    information is needed. Page 28 GAO/ PEMD- 96- 1 Improving the Flow of
    Program Information to the Congress

    Chapter 3 What Information Is Available? Three Case StlldieS

    whether the center is located in a rural or urban area and what the center's
    philosophy and staffing patterns are- through center- written progress
    reports, program- staff site visits, and ethnographers' reports. ACYF thus
    has information that can communicate a “feel” for the challenges
    providers face in delivering a comprehensive set of services and how clients
    see the program as affecting their lives.

    Implementation We found much useful information on CCDP implementation
    processes and feasibility issues, as well as evaluative information on
    whether the program is being implemented as intended. ACYF has qualitative
    and quantitative information that is useful for both compliance assessment
    and service improvement. The ethnographers' reports from the first year of
    the program included specific information on stat+ up problems. They also
    described how site officials completed such necessary steps as setting up
    interagency agreements for service delivery and how grantees' approaches to
    service delivery were adapted as programs took shape. The grant application,
    as well as the ethnographer, site visit, and progress reports, provided
    additional insights into planned implementation.

    Impact The agency has data on participating children's developmental
    progress based on the results of standardized tests, as well as on whether
    parents gained parenting skills. Measures of short- term changes in parental
    income and employment status, as well as of general parental well- being and
    level of education, are also available. The agency has gathered similar
    information on a nonparticipating control group of families to compare with
    clients to determine whether any observed changes in the program families
    were a consequence of the demonstration program. Anecdotal information from
    participating family members (in progress or ethnographer reports) about how
    the program has improved their lives has also been collected. These data
    should be sufficient to support conclusions concerning the program's impact
    on low- income families and children.

    Side Effects and Comparative Advantage The agency has diverse sources of
    information that would capture CCDP'S

    side effects (intended or not) on the local community and its social
    services agencies. CCDP evahmtion studies have focused on this program alone
    and were not designed to compare CCDP with similar or different programs. A
    study comparing programs or models would need CCDP program data as well as
    information on other programs, and it would need to be arranged for
    separately. In addition, cross- site comparisons of effectiveness may be
    difficult to make since CCDP centers differ in many ways. If the Committee
    wants a comparison of the CCDP approach with

    Page 29 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case StUdieS

    I another, or an assessment of which CCDP features are most needed, it
    should request such a study.

    What the Congress Requested and What It Received

    Enabling legislation for the CCDP mandated a study that was to report on
    results of the program by October 1,1993. The legislation set out evaluation
    objectives for a study (describe service delivery mechanisms and assess
    project impacts), specified that project impacts be measured with control
    group comparisons, and provided for an evaluation of program successes, but
    allowed the agency to select its evaluation design.

    In response to this mandate, the agency provided the Committee with a report
    on program impacts on families and communities, an analysis of program
    implementation, and a description of families and projects3 It reported
    short- term effects on children, families, and mothers in terms of their
    education, income, health, and parenting shills. Because it was due before
    the demonstration was complete, the report provided only interim findings.

    This mandated report had both strengths and limitations. It provided needed
    outcome conclusions, useful feasibility information, and a separate,
    readable executive summary. Programwide impact results (and answers to
    feasibility and process evaluation questions) were reported succinctly and
    clearly, but disaggregated statistics would have shown the Committee the
    range of performance and reflected possibly important different features of
    the CCDP model. The feasibility information would have been useful during
    the deliberations concerning reauthorization of the Head Start Program and
    its inclusion of ccrAike services, but the report came out in May 1994 just
    as final action on the legislation was occurring.

    In the interim, the Committee obtained program information through three
    other means (apart from normal ad hoc information requests). In December
    1991, the program agency sent the Committee copies of a report it had
    prepared to inform the public and other agencies about CCDP. Although
    intended for another audience, the report contained information on program
    operations and differences among sites and also evaluative information on
    program feasibility of use to the Committee in its CCDP oversight and Head
    Start reauthorization activities.

    sACYF', “Comprehensive Child Development Program-- A National Family
    Support Demonstration,” interim report to the Congress, May 6,1994.

    Page 30 GAO/ PEMD- 951 Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case Stlldi~

    In addition, the agency provided information on CCDP to a special commission
    (consisting of congressional committee staff- including representatives from
    this Committee- as well as agency officials and outside experts) formed to
    make recommendations concerning the Head Start Program. The CCDP program
    officer provided information to assist the commission in understanding the
    possible effects of extending the Head Start Program to include ccm4ike
    services to very young children. Finally, a member of the Committee staff
    requested and obtained current data on program impact- data more recent than
    the figures to be summarized in the CCDP interim report- from child
    development center directors.

    ACYF plans to provide a final report to the Congress in 1996. It will be
    based on the entire 5 years of experience with the program and is expected
    to provide an assessment of the overall impact of CCDP.

    Service Programs: CHC and Chapter 1 ESEA

    Core Questions About The first of our service programs, the CHC program,
    funds medical centers These Programs to provide outpatient care (and
    supporting services such as patient

    transportation) to populations that otherwise would lack access to such
    health care. Some community health centers serve remote rural areas that
    lack medical facilities, and others serve urban populations that do not have
    access to care through other providers. (For a description of the CHC
    program, see appendix II.) The Chapter 1 ESEA program provides financial
    assistance (through grants to states) to local education agencies to meet
    the special needs of educationally deprived children who live in areas with
    high concentrations of low- income families. Student demographics,
    educational achievement levels, and the services delivered vary considerably
    from school to school. For example, some schools assist

    I low- performing students within the regular classroom, while others pull
    these students out to a special setting for assistance. (For a description
    of

    ! I the Chapter 1 ESEA program, see appendix III.) Table 3.1 summarizes the
    core questions adapted to fit the type and nature of each program. The two
    programs have certain questions in common. Both operate in diverse settings,
    so it is important to ask how services and conditions vary as welI as to
    secure a good summary description. Both are

    Page 31 GAO/ PEMD- 951 Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case StUdieS

    targeted to needy populations, so it is relevant to ask what proportion of
    eligible clients were served and whether significant numbers of eligible
    clients remained unserved. (For Chapter 1 ESEX, attention has focused
    particularly on eligible students in the neediest schools- that is, schools
    in which a high percentage of students come from low- income families.)

    Table 3.1: Core Questions Adapted to CHC and Chapter 1 ESEA Programs
    Category of

    information Questions for CHC Questions for Chapter 1 ESEA Description

    implementation Targeting

    Overall, what services are Overall, what services are provided and to whom?
    How do provided and to whom? How do services, center locations, and services
    and school and student mix of clients vary from site to characteristics vary
    from site to site? site?

    Do clinical practices meet professional quality standards?

    What fraction of those who What fraction of the neediest would otherwise
    lack access to schools and students care does the CHC program participate?
    How does this serve? How does this fraction fraction vary across sites? How
    vary across grantees? What many sites lack services? significant areas,
    health problems, or populations are not covered?

    Impact Has the long- term health status In the aggregate, has of CHC
    patients improved as a disadvantaged students' result of program services?
    achievement improved? Has

    individual participants' achievement'? How does impact differ across
    approaches and across schools? Do some schools or approaches consistently
    fail to produce gains?

    Side effects Comparative advantage

    What are the effects on other Does the program fragment health care
    providers (if any) in responsibility for ensuring that the CHC communities?
    students' progress improves?

    What are the comparative advantages of shifting to schoolwide, rather than
    individually targeted, aooroaches?

    Beyond the common ground delineated in table 3.1, the two programs are
    dissimilar. For the CHC program, the question of whether the clinicaI
    services provided meet current standards of good medical practice is
    fundamental. In terms of program effect, the simple provision of quality
    clinical services is an important program outcome. The broader medical

    Page 32 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case Studies

    research question of whether these services influence a patient's health
    status in the long run is pertinent to any health care service system (not
    just this program) and thus needs to be answered in a broader context. The
    Chapter 1 ESEA program, by contrast, focuses on questions of impact, such as
    whether the provided services produced increases in student achievement.
    Defining and evaluating the quality of implementation of Chapter 1 ESEA
    services is a state and local responsibility. However, the Congress has an
    interest in learning whether some of the general . approaches used are more
    advantageous than others.

    Available Information Relevant to These Questions

    The agencies that administer the service programs we reviewed collect a
    considerable amount of information potentially relevant to the foregoing
    questions. HHS' Health Resources and Services Administration (HRSA) receives
    narrative information in center applications; collects data on CHC program
    services, costs, and clients; and maintains records of quality reviews. When
    drawn together, this information could address many of the questions we have
    identified. The numerous evaluation studies concerning the Chapter 1 ESEA
    program similarly include information that covers many of our questions.
    Moreover, HRSA and the Department of Education's Office of Compensatory
    Education receive additional useful information- including clues to emerging
    concerns- through various program management activities as well as through
    active networking with service providers. In the sections below, we
    summarize what is available with respect to each of the information
    categories and questions listed in table 3.1.

    Description Both of these programs have gathered information from grantees
    and have used this information to provide aggregate descriptions and
    statistics, rather than to describe the range of variation to be found
    within the program. For Chapter 1 ESEA, data on clients, services, and
    student achievement are initially collected at the school or center level
    but are subsequently aggregated and reported at the state and national
    levels. CHC reports data only for the program as a whole. HR, SA could piece
    together information from site visit reports and from its routine project
    management activities to describe for the Committee how community health
    centers and patients' experiences within them differ. Descriptive
    information of this kind is available from school case studies conducted in
    connection with the recent Chapter 1 ESEA evaluation. 4 The summary report
    of the evaluation included vignettes drawn from these studies and

    %am Stringfield et al., Urban and SuburbanRural Special Strategies for
    Educating Disadvantaged Students- Fht Year Report (Washington, D. C.: U. S.
    Department of Education, Office of the Under secretary, 1994).

    Page 33 GAOREMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case Studies

    also presented data that was disaggregated to describe differences between
    high- poverty and low- poverty schools. 6

    Implementation Information about how the CHC clinics implement recommended
    medical procedures is based on site visits, HRSA'S quality review, and
    federal and regional officials' routine program monitoring. As mentioned
    previously, Chapter 1 ESEA has considered the quality of instructional
    practices to be a state and district responsibility, Federal program
    monitoring thus has not included the gathering of data on instructional
    quality. However, there are plans to make observation of instructional
    practices a larger part of federal monitoring in the future.

    Targeting The CHC program generates data about the numbers of clients served
    in its centers, and HRSA could develop population estimates to calculate the
    proportion of the targeted population that is served by each center and by
    the program as a whole in areas already designated as medically underserved.
    HRSA has relied on applicants to make known unmet needs in areas not
    currently included. (Community groups or local health organizations often
    apply for a designation of need.) In the past, it has not initiated its own
    assessment of medical need for areas from which there has been no
    application. However, based on Bureau of Primary Health Care (BPHC)
    estimates of areas with high priority for federal intervention, HRSA could
    determine the areas or people with the greatest general health care need and
    estimate the amount of unmet need in areas not now covered by CHCS. 6

    The 1993 Chapter 1 ESEA evaluation included a special study that examined
    the degree to which program funds were reaching the schools with the
    greatest concentrations of disadvantaged students, as well as the number of
    such schools that remain unserved. The question of whether funds are going
    to the most educationally needy students cannot be answered from grantee
    reports, but a study currently being conducted (the “Prospects”
    study) will shed light on this issue. (The final report from that study is
    due in January 1997.)

    Ykhools in which 75 percent or more of the students came from low- income
    families were designated high- poverty schools; low- poverty schools were
    those in which 0 to 19 percent of the students came from such families.

    %RSA is developing a proactive method of estimating the amount of need for
    primary care that does not restrict the focus to currently designated ~IW.
    S. Some recent data on the amount of unmet medically underserved needs are
    available from the National Association of Community Health Centers.

    Page 34 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case Studies

    Impact Side Effects and Comparative Advantage

    As noted previously, the primary purpose of the CHC program is to provide
    medically underserved people with access to medical services. Program data
    have focused on numbers served and service quality. 7 ‘I'he express
    purpose of Chapter 1 ESEA, however, is to improve student achievement.
    Traditionally, two sources of information have been used to evaluate program
    impact. Results from the National Assessment of Educational Progress-- which
    periodically tests a national sample of 4th, 8th, and 12th grade students in
    reading, writing, and mathematics- have been used to gauge whether
    disadvantaged students' achievement is improving overall. The program has
    also required states to report year- to- year gains in participating
    students' scores (aggregated to the state level) on nationally normed
    standardized tests. .

    These gain- score data give some idea of whether students in the aggregate
    are making progress, but they do not show the extent to which individual
    participants are benefiting from the program to the extent that they no
    longer need compensatory services. Nor do these aggregated data illuminate
    variations in impact across schools or subgroups of students. Attempts to
    use gain scores to identify schools that consistently fail to get results
    have been problematic, in part because of inconsistency in reporting and
    problems of data reliability.

    In the future, the aforementioned “Prospects” study will fill in
    some of the gaps left by traditional information sources. That study will
    provide more differentiated information on services received and on
    achievement gains of Chapter 1 ESEA participants in comparison with other
    students. As of July 1,1995, states will no longer be required to report
    nationally normed test score data, Rather, each state will assess student
    progress and adequacy of school performance in terms of its own standards
    for content coverage and student performance, using achievement tests and
    other instruments aligned to those standards. lt is not yet clear what
    information will be reported to the federal program office.

    An issue for Chapter 1 ESEA has been whether providing compensatory
    education services through separate teachers or in separate settings has had
    undesirable side effects, such as impeding coordination between regular and
    compensatory instruction or encouraging classroom teachers to disclaim
    responsibility for Chapter 1 students' progress. While the program has data
    on how many schools use “pull- out” and other approaches and how
    much time students spend in each, these data do not

    %FBA collects some health outcome data and has recently contracted for an
    impact study of the CHC program. It has also supported efforts to develop
    health outcome measures pertinent to any primary care program.

    Page 35 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case StUdieS

    illustrate how commonly used versions of each approach actually operate in
    relation to the regular classroom, how quality varies within each, and how
    variations in quality and approach are linked to student outcomes, The
    program has only begun to gather information at the level of detail needed
    to shed light on these questions and to compare the advantages of various
    models of delivering compensatory educational services. For the CHC program,
    information is available on program side effects. The comparative advantage
    of the CHC program over other programs has not been an evaluation issue,
    although it may be in the future.

    What the Congress Requested and What It Received

    The Congress did not mandate an evaluation study or reports on the CHC
    prbgram, It did encourage evaluation of programs under the Public Health
    Service, which would include CHC, through legislation that set aside funds
    for that purpose. However, the set- aside provision did not require the
    agency to synthesize information and report it to the Congress. Reports have
    been provided when specificzilly requested, but this approach has left much
    useful information unreported, as the CHC case illustrates. 8

    The Committee informally requested studies and reports about particular
    aspects of the CHC program (such as the costs and effects of the
    Comprehensive Perinatal Care Program, quality of care, and capital
    improvement needs), and the agency provided them. In the absence of a
    request for a summary evaluation of the program, however, the bulk of the
    agency's information resources remained untapped. The only overall report
    the Committee received on the CHC program was the annual appropriations
    request- which stated the program's purpose, noted some of the services
    offered and the number of users, and reported previous program funding
    levels. However, this annual request, in keeping with the standard format
    for such requests (which are focused on appropriations issues), described
    the program in very broad terms. Committee staff commented that such
    general, highly aggregated information was not particularly useful for
    overseeing or evaluating the CHC program.

    The Chapter 1 ESEA program offers a contrasting case in which the Committee
    has requested and received a great deal of information, both in interim
    years and for reauthorization. Soon after maor changes were made to the
    program in 1988, congressional and agency officials became

    sFor wonnation on the use of the PHS set- aside, see our earlier report
    entitled Public Health Service: Evaluation Set- Aside Has Not Realiied Its
    Potential to Inform the Congress, GAO/ PEMD- 93- 13 (Washington, D. C.: Apr.
    8,1993). The Preventive He& h Amendments of 1993 (P. L. 103- 183) amended
    the Public Health Service Act to establish a requirement for annual
    reporting of findings of evaluations conducted under the set- aside.

    Page 36 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case StUdie. 9

    aware of implementation problems with the “maintenance of
    effort” provision for schoolwide programs? The two sides communicated,
    and the problems were remedied. (We were told that this kind of interim-
    year problem identification and correction are common.)

    The Congress mandated two major evaluation efforts for Chapter 1 ESEA: (1) a
    longitudinal study of a national sample of students (the
    “Prospects” study) and (2) an evaluation of the implementation
    and the effects of changes made in 1988. It also established a special panel
    to examine the use of standardized tests to assess student achievement.

    In the course of its reauthorization deliberations in 1993 and 1994, the
    Committee received reports from the longitudinal study, the assessment
    study, and the numerous special studies that comprised the mandated
    evaluation. There was also a summary report that drew together findings and
    derived recommendations from all of these efforts. The plethora of study
    reports provided more information than the Committee could comfortably
    process. However, the summary report- which was organized around policy
    questions and provided easy- to- find summaries of main points,
    disaggregated data, and illustrative vignettes on case study school- was
    viewed as useful.

    The Congress also requested, and the Committee has received, a summary of
    Chapter 1 ESEA program participation and achievement data for each year-
    unfortunately, not until a year or more after the end of the school year for
    which data are collected. lO General descriptive material and summary data
    are included in appropriation requests and in the Department of Education's
    annual summary of evaluations.

    Communicating About In each of the cases we studied, we observed very
    limited communication Information Needs between the parties concerning the
    Committee's questions and how the

    agency might best respond to them. Ad hoc inquiries aside, congressional
    requests for evaluative information took the form of statutory mandates.
    These mandates were quite specific, setting forth design details and report
    dates as well as objectives for the evaluation. The specifics were worked
    out in Committee deliberations with relatively little input from agency
    staff. In one of our cases, agency officials reviewed and suggested changes

    sThis provision concerns expenditures per pupil under a schoolwide program
    compared to expenditures in the same schools in the previous fmcal year.

    ‘Drhe agency has established new procedures and, ss a result, expects
    to make data available more promptly in the future.

    Page 37 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case Studies

    in an early draft that was under House committee review, but they did not
    discuss their concerns with committee representatives.

    Once the evaluation mandates became law, agency officials in the Department
    of Education and HHS took them as a given. In preparing to implement the
    mandates, they went to considerable effort to interpret the interests and
    concerns expressed in the written record, but they did not confirm their
    interpretation with the Committee. Education Department officials did seek
    to obtain a sense of the relative priority of the questions listed and to
    discuss what they judged to be unworkable deadlines or design
    specifications, but Committee staff informed us that such behavior was the
    exception.” (In another case, the program manager did not see
    discussion of concerns with the Committee as a realistic option.) We found
    that, for the most part, agency officials simply did their best to do what
    was mandated.

    Interchange was similarly limited as evaluations were conducted and
    reported. For example, Committee staff attended some of the meetings of the
    advisory group to the Chapter 1 ESEA evaluation, but as observers rather
    than as spokespersons for the Committee's information needs. Department of
    Education officials briefed the Committee on the forthcoming report, but
    Committee staff perceived that these briefings were designed to inform the
    Committee of what the Department had done rather than as an occasion for
    dialogue about priorities or about the best way to present the information.

    Finally, we observed that the Committee rarely provided feedback to the
    agency concerning the reports it received. Committee staff did recall
    telling Department of Education officials that the clear organization and
    policy focus of the Chapter 1 report had been helpful. But they indicated
    that the Committee was unlikely to provide feedback on reports that were
    unsatisfactory- that is, that either failed to provide relevant information
    or buried it in an undigested mass of facts and observations.

    The literature on public policy and the legislative process suggests that
    the communication patterns we observed are to be expected. The executive and
    legislative branches of government have different constituencies,
    institutional perspectives, and interests. Evaluative information often has
    policy implications that could have important consequences for the agency
    and department.

    IThe National Assessment of Chapter 1 was mandated in May 1990, with a final
    report due at the end of 1992. Meeting this schedule posed major challenges
    for the Department.

    Page 38 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case Studies

    In view, of their importance, policy- sensitive communications with the
    Congress are channeled through central congressional liaison offices, and
    contacts with committees are subject to high- level policy review and
    clearance. Discussions with our Committee respondents and agency officials,
    together with. our review of agency policy, revealed that the liaison
    officer's chief role is to facilitate the exchange of information as
    legislation is drafted and enacted. Liaison officers also pass along ad hoc
    committee requests for information and arrange the briefings that precede
    the issuance of an evaluation study report. But they typically do not bring
    agency and committee staff together for evaluation planning.

    Concluding Observations

    Each of the three programs we studied collected a great deal of information
    relevant to questions of interest to the Committee. True, there were gaps in
    the information, and some useful questions could not be answered from
    available sources. However, we observed that lack of information does not
    appear to be the main problem. Rather, the problem seems to be that
    available information is not organized and communicated effectively. Much of
    the available information did not reach the Committee, or reached it in a
    form that was too highly aggregated to be useful or that was difficult to
    digest.

    Second, we observed that the Committee was given no more than it asked for.
    It asked for no summary evaluation report on the CHC program, and it
    received none- either in interim years or at reauthorization. It mandated
    studies on a host of implementation and evaluation questions concerning the
    Chapter 1 ESEA program, and it received a report on every one. It mandated a
    report evaluating the CCDP, and it received such a report- although the
    report results could only be based on information from a few years of
    experience.

    However, receiving what it initially asked for did not always meet the
    Congress' current needs, since these evolved over time (whereas statutory
    mandates remained fixed). For example, the Chapter 1 ESEA mandate called for
    a study of the implementation of services to children in private schools,
    but implementation was no longer a major issue by the time the report was
    due. The reauthorization and consideration of expanding Head Start services
    provided another example: it created a need for an earlier- than- scheduled
    report on emerging findings (on feasibility and implementation issues, in
    particular) from the CCDP program 0ffice. l'

    %ome of the emerging findings on implementation drawn from the report could
    have been fruitfully shared in a letter or briefing to the Committee while
    report editing and clearance proceeded as usual.

    Page 39 GAO/ PEMD- 95- 1 improving the Flow of Program Information to the
    Congress

    Chapter 3 What Information Is Available? Three Case StUdieS

    F'inally, we observed that communication between the Committee and agency
    staff knowledgeable about program information was limited and comprised a
    series of one- way communications (from the Committee to the agency or the
    reverse) rather than joint discussion. This pattern of communication, which
    was reinforced by departmental arrangements for congressional liaison,
    affords little opportunity to build a shared understanding about the
    Committee's needs and how to meet them.

    Taking into consideration what we found and reported in chapter 2, our case
    studies led us to conclude that obtaining timely and useful information for
    oversight and reauthorization requires not only knowing what questions to
    ask, but also ensuring that the information is in fact requested when needed
    and communicating with the agency to establish a mutual understanding of
    information needs and how they can be met. In chapter 4, we propose a
    strategy that incorporates each of these features.

    Page 40 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    Our final task for this report was to propose a strategy that the Committee
    might use to request information- one that would take into account the
    various factors that have contributed to unsatisfactory results in the past.
    The Committee's request letter recognized one such factor: the difficulty of
    knowing what questions to ask about programs of various types. Our strategy
    addresses that factor as well as two others that came to light through our
    case studies: agencies' disposition to provide information only when they
    have been asked to do so, and insufficient communication between the
    Committee and the agency with respect to information needs. We thus propose
    a strategy that attends not only to the content of information requests, but
    also to communication practices that can help ensure satisfactory responses
    to those requests.

    The strategy we suggest for the Committee's consideration includes the
    following three components:

    l selecting and adapting, from a core list, the descriptive and evaluative
    questions to be asked about a program in interim years and at
    reauthorization; . arranging explicitly to obtain timely oversight
    information in interim years

    as well as to receive results of evaluation studies at reauthorization; and
    l providing for increased communication with agency program and

    evaluation staff to help ensure that information needs are understood and
    that requests and reports are suitably framed and are adapted as needs
    evolve.

    Identifying and We begin by proposing guidance to assist the Committee in
    identifying the Adapting Core questions to ask concerning a given program.
    In chapter 2, we outlined a

    set of core questions that might be asked about programs of various types.
    Questions In the section that follows, we suggest how the Committee, drawing
    on its

    sense of the policy issues and of the history of a particular program, can
    select the core questions most important to that program, restate them in
    program- specific terms, and outline the forms of information needed to
    answer them. The Committee may elect to select and state the questions
    initially (as in the process of framing the questions for a mandated
    evaluation) and then request comments from the agency. Or, it may ask the
    agency to perform this initial step and to conlirm the questions with the
    Committee.

    I In either case, we suggest that the Committee begin by reviewing the

    particular program's purpose and history, current policy issues regarding
    Page 41 GAOREMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    this program, and the policy context. This initial review will help to
    determine which of the core questions listed in table 2.1 apply, what
    additional questions are of interest, and whether a full evaluation is
    needed. (It would probably be useful to ask the agencies to review the list
    also, as a first step in building a common framework for thinking about
    evaluation information.) For example, if policy review determines that the
    program's function could now be performed at the state level or that its
    dwindling population could be covered under a similar federal program, there
    might be little reason to seek additional agency information about program
    implementation or impact.

    As noted in chapter 2, knowledge of a program's type or purpose is helpful
    in identifying the categories of information that are likely to be most
    critical to its evaluation. For example, the evaluation of demonstration
    programs (other than those that merely seek to provide an example) generally
    involves questions concerning implementation and impact, Statistics and
    research programs virtually always involve questions of implementation in
    accordance with professional standards and targeting to relevant problems.

    Beyond this, programs must be reviewed on a case- by- case basis to
    determine which core questions are most pertinent. For example, the
    Committee or agency might ask itself the following questions:

    0 Do conditions within this program vary so substantially that the Committee
    will need disaggregated descriptive information? 0 Are there new program
    aspects or provisions that are so significant that

    the Committee will want information on their implementation in interim
    years? 0 Is targeting a critical issue for this program? If so, which of the
    targeting

    questions apply? 0 Have possible side effects been identified that warrant a
    specific question?

    In most cases, the Committee or the agency should restate each relevant core
    question so that it fits the program- for example, to identify the
    particular target population as “recently migrant students”
    rather than as “the appropriate people.” The case studies
    reported in chapter 3 and in the appendixes provide examples of such
    restatements.

    The Committee should also consider what form of information it needs in
    response to each question, and When and for what purpose (oversight or
    reauthorization) the information is needed. This is not to say that the

    Page 42 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    Committee should spell out in detail the specific data to be submitted.
    Rather, it should let the agency know its intent with regard to such
    questions as the following:

    Is the information needed for the next reauthorization or at an interim date
    before that time? Is an empirically precise answer- and hence time and
    resources to support a planned study- needed?

    Examining its information needs in this way will help the Committee focus on
    the central issues, express its intent clearly, and avoid elaborate and
    expensive data collection when an estimate based on existing evidence would
    do as well. (For example, rather than developing an exact count of the
    number of sites that have implemented a new provision, program managers
    might estimate based on recent monitoring visits that about two thirds of
    the sites had done so.) This outline will form the basis for framing
    information requests that make the Committee's objectives clear, while
    avoiding detailed specifications that may prove to be unworkable. (Under our
    strategy, the agency is responsible for working out the details but must
    communicate with the Committee as it does so to ensure that the resulting
    plan meets the Committee's needs.)

    To assist the Committee in creating such an outline, we have drawn on our
    observations from this study to create a guidance table (table 4.1). This
    table contains a row for each of our categories of information: description,
    implementation, targeting, impact, side effects, and comparative advantage.
    In addition, column one of the table lists our core questions in abbreviated
    form, column two notes the kinds of information that may be needed for each
    question, and column three notes the types of programs to which the question
    is likely to apply.

    Page 43 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    Table 4.1: Guide to Identifying Needed information Information needed to

    Question respond Applicability Description Overall, what activities are
    Narratives and summary All programs conducted? By whom? How data that
    describe the extensive and costly are the scope and extent of the
    activities, and whom do they program reach?

    If conditions, activities, and Narratives and Programs that feature purposes
    are not uniform disaggregated data that varied activities, clients,
    throughout the program, in show the extent of variation and settings what
    significant respects do within the program they vary across program
    components, providers, or subgroups of clients?

    Implementation What progress has been made in implementing new provisions? a

    Notification of early All demonstration programs; difficulties; descriptive
    data other types of programs on the extent of when new or when
    implementation at specified important new features are intervals; answers
    introduced concernina feasibilitv issues

    Have feasibility or Same as above Any program management problems become
    evident? a

    If activities and products are Answers concerning All statistics and
    research expected to conform to whether and to what extent programs; other
    types of professional standards or to criteria have been met; programs when
    criteria are program specifications, have notification of evidence that
    specified they done so? standards or criteria may be

    inappropriate or outdated Targeting Have program activities or Description
    of the issues or Statistics, research, service, products focused on problems
    targeted and and regulatory programs appropriate issues or evidence that
    supports their problems? selection; if different

    components of the program target different problems, report them separately

    To what extent have they reached the appropriate people or organizations?

    Answers showing how Demonstration, service, and many of each relevant
    regulatory programs; category of person or statistics and research
    organization were reached; programs whose objectives if targeting differs by
    site or include increasing the by program component, participation of or
    report separately for each disseminating results to

    certain populations (continued)

    Page 44 GAO/ PEMD- 95- 1 Improving the Flow of Pronram Infcwmatinn t. n the
    Cnnano~

    A Sirategy for Requesting Evaluation Information Chauter 4

    Question Information needed to resoond Aoolicabilitv . . . Do current
    targeting Answers that show the Service or regulatory practices leave
    significant nature and extent of critical programs intended to cover needs
    unmet (problems not unmet needs a specified population fully; addressed,
    clients not research and statistics reached)? programs intended to cover

    a problem area fully . Overall, has the program led Answers concerning All
    demonstrations; to improvements consistent changes in outcomes and research,
    regulatory, and with its purpose? how they are related to service programs
    that are

    program activities intended to produce specified results

    If impact has not been Answers concerning Demonstration, service, and
    uniform, how has it varied variations in conditions and regulatory programs
    that across program components, program impact under each deal with varied
    clients or approaches, providers, or condition varied conditions of client
    subgroups? implementation

    Are there components or Answers concerning Same as above providers that
    consistently variation in impact across have failed to show an providers or
    components impact? and extent of turnover in the

    low- performance category Side effects Have program activities had
    Notification of possible Any program important positive or negative effects;
    answers if evidence side effects, either for shows need for further program
    participants or investigation outside the program?=

    Is this program's strategy Comparative analysis of Any program when more
    effective in relation to effects of this program and alternatives and
    evidence its costs than others that of alternative approaches in concerning
    them are serve the same purpose?, relation to costs

    %formation important for interim years. available -

    Arranging for Responses

    Once questions have been identified, the Committee will need to arrange
    explicitly to obtain responses to them. Based on our observations, we
    suggest that the Committee set up procedures for reporting in interim years
    as well as arranging for program or comparative evaluations to be conducted
    in connection with reauthorization and policy review decisions.

    Page 46 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    Soliciting Interim- Year Information

    Currently, there is no standard mechanism for prompting agencies to report
    information the Committee would find useful for oversight in interim years-
    that is, time- sensitive information that would keep the Committee informed
    of progress in implementing new provisions and of significant new evidence
    concerning program activities and effects. (As noted previously, neither
    budget submissions nor annual reports- as currently configured- are designed
    to serve this purpose.)

    We propose that the Committee initiate a mechanism for prompting agencies
    (or a subset of agencies the Committee selects) to respond to program
    oversight questions each year. Agencies already prepare responses to
    questions about their budget requests each year, and they could provide
    responses to the authorizing committee on a similar schedule.

    In the interest of evoking responses that are brief and up- todate, we
    suggest that the Committee express its request for interim- year information
    through a letter (transmitted to the program agency through established
    departmental channels) rather than more formally through legislation. 1 The
    annual letter could include questions tailored to a particular program (such
    as progress in remedying a problem that was identified in a previous
    evaluation or oversight review). However, we believe that many of our core
    questions pertinent to interim years (for example, “what progress has
    been made in implementing signiiicant new provisions” or “have
    feasibility problems become evident?“) could be asked in general form
    in cases where the Committee's specific concerns have been communicated in
    earlier discussions with the agency. 2

    Agency officials indicated that they can respond to telephone or letter
    requests for time- sensitive factual information relatively quickly.
    However, responses to statutory reporting mandates or to Committee requests
    for information that raise budget or policy issues or that involve changes
    in legislation take time to develop. Such responses often involve policy or
    management concerns-- concerns that go beyond the immediate program and that
    program staff are not authorized to address. Department officials
    appropriately require that these responses be centrally reviewed so that

    ‘We expect that currently established departmental procedures would be
    followed in transmitting Committee requests to program agency staff and in
    approving and transmitting the latter's response.

    ‘The questions to be included in the letter, and the timing of the
    requests, should be reviewed when the system of annual performance reports,
    established through the Government Performance and Results Act of 1993 (P.
    L. 103- G!), is implemented in the year 2000. The fit between our approach
    and the act's ls discussed in more detail later in this chapter.

    Page 46 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    these broader concerns can be addressed and input from different units
    coordinated.

    It may be helpful if, along with stating the Committee's questions, the
    request letter or other communication included such language as:
    “Please contact (Committee staff person) if you have questions; we
    welcome the opportunity to discuss this request.” This would encourage
    contact from the agency in cases where clarification was needed and would
    also give the department an opportunity to indicate which questions could be
    answered quickly and directly and which raised complex issues that would
    take more time to resolve.

    Preparing for Reauthorization or Policy Review

    As discussed in chapter 2, answering the evaluation questions that arise
    during reauthorization often requires data or narrative information beyond
    what the program routinely collects. We urge the Committee to notify
    agencies well in advance when it has such questions in mind so that they can
    plan (and budget) to obtain and analyze the necessary material in time. Such
    advance notice would help the agency set priorities for using the resources
    available to support evaluations. Comparative studies also require
    considerable advance preparation.

    Communicating About The foregoing leads us to the final element of our
    strategy: increased Information Needs committee- agency communication
    concerning information needs. As

    stated in chapter 3, present practice affords little opportunity for joint
    discussion and gives agency program and evaluation staff little basis for
    understanding what the Committee wants to know and what method of
    “packaging” the information would be most useful to it. We agree
    with other authors that increasing the opportunities for discussion would
    likely lead to more satisfactory results.

    Increased communication is critical at two points and would be useful at
    several others. The critical points occur

    l when the Committee frames a request for information, to ensure that the
    agency understands what is wanted and thus can alert the Committee to issues
    of content or feasibility that need resolution, and l as report drafting
    begins, to assist the agency in understanding the issues

    that will be before the Committee and what kind of presentation format is
    thus likely to be most useful.

    Page 47 GAO/ PEMD- 95- l Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    These points are critical because unverilied agency assumptions or
    misinterpretations at either point can cause even the best- intentioned
    agency response to be off the mark.

    The additional points at which we suggest communication occur l when the
    agency develops the design of a major study, to verify that it is

    on target from the Committee's perspective; . midway through the period
    between reauthorizations, to consider whether

    issues and priorities have changed and to involve any new Committee staff;
    and 0 after a report is submitted, to convey feedback concerning its
    usefulness

    to the Committee. Communication at these points might not be necessary in
    cases where the task before the agency is clear- cut, the issues are
    unlikely to change, and the report is well received. However, in less
    favorable circumstances, these communications points would likely be useful.

    Additional communication can be facilitated by using organizational
    structures and procedures that are already in place. It is important for the
    Committee to initiate requests and to give agencies a clear signal that it
    welcomes discussion with the agency, as previously outlined. The
    congressional liaison officer for the agency can then assist by providing
    agency program and evaluation staff with background information about the
    Committee's intent. He or she can also bring agency and Committee
    representatives together so that issues can be resolved and mutual
    understanding achieved.

    Feasibility of Our Approach

    During the course of our study, we discussed our proposed strategy with
    Committee staff and with officials in the agencies that served as case
    studies. Both groups found our core questions useful and acknowledged the
    potential benefits and even a common interest in moving in the direction we
    propose.

    Both groups also noted factors (ranging from demands on Committee time to
    potential conflicts over where to draw the line between legislative
    oversight and executive responsibility) that create obstacles to the kinds
    of collaboration the strategy envisions. Given the different constituencies,
    institutional perspectives, and interests of the Congress and the executive

    Page 48 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    departments, these obstacles are to be expected. However, we were impressed
    by the commonality of interest that we observed between the two groups. We
    believe that it constitutes a solid basis for moving forward to improve the
    usefulness of evaluation information.

    Agency Comments and Our Response

    In written comments on a draft of this report, the Departments of Education
    and HHS agreed to the benefits of increased consultation between the
    Committee and the agency regarding evaluative information needs. However,
    they were concerned that our proposed strategy be consistent with the
    requirements of the Government Performance and Results Act and not
    constitute an added burden. The Department of Education also expressed
    concern about two of our categories of information- side effects and
    comparative advantage. The Department noted that questions under these
    categories can be diftkult for the program agency to address if they require
    information about programs administered by another agency. We address these
    concerns in turn.

    Our Approach and GPRA While this study was in progress, the Congress passed
    the Government Performance and Results Act of 1993 (P. L. 103- 62). By the
    turn of the century, annual reporting under this act is expected to fiU some
    of the information gaps we describe in this report. Among other things,
    GPFLA requires every agency to establish indicators of performance, set
    annual performance goals, and report on actual performance in comparison
    with these goals each March beginning in the year 2000. Agencies are now
    developing strategic plans (to be submitted September 30, 1997) articulating
    the agency's mission, goals, and objectives preparatory to meeting these
    reporting requirements.

    We compared our approach with GPRA'S and concluded that the two are
    compatible. Both approaches emphasize the importance of agency consultation
    with the Congress as evaluation strategies are planned, goals and objectives
    identified, and indicators selected Both note the importance of providing
    information that indicates how well the program is doing with respect to its
    intended objectives, and both call for annual reporting of information to
    provide the basis for accountability and effective oversight.

    We foresee little added burden- and some efficiency of effort- from using
    our approach in conjunction with GPRA. The procedures for identifying and
    adapting core questions presented earlier in this chapter are likely to be

    Page 49 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Chapter 4 A Strategy for Requesting Evaluation Information

    useful in preparing to meet GPRA requirements, and consultation with the
    Committee should ensure that data collected to support GPRA reporting can
    also be used to meet the Committee's special needs (for example, that
    performance data can be disaggregated in ways that are important to the
    Committee).

    At the same time, our suggested approach provides a useful complement to
    GPRA. Our approach to requesting interim- year information can be applied
    immediately, to take advantage of currently available information while GPRA
    reports are being planned. Moreover, while GPRA annual reporting focuses on
    intended program outcomes, our approach covers additional categories of
    information- description, side effects, and comparative advantage- that the
    Committee finds useful, especially in connection with major program
    reauthorizations or policy reviews.

    Evaluating Side Effects and Comparative Advantage

    We agree that to evaluate comparative advantage typically requires data
    beyond what the administering agency can provide and that some side- effects
    questions might also pose this problem. Our discussion in chapter 2 notes
    that by virtue of their complex informational requirements, comparative
    advantage studies typically are implemented through special arrangements.
    Special arrangements would similarly be appropriate for conducting a
    detailed evaluation of one program's side effects on other programs.

    Many side- effects questions, however, fall within the scope of evaluation
    for a particular program (as our case study examples illustrate). The
    process of discussion that we have suggested should help the Committee and
    the agency identify the side- effects questions that are pertinent to a
    program and the level of detail or precision needed for each. Discussion
    should also examine whether it is feasible for the administering agency to
    gather this information or whether special arrangements will be needed.

    Page 50 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Page 51 GAOIPEMD- 96- l Improving the Flow of Program Information to the ,,
    ( , * 8,

    / ,:::-: i ,pl ‘/ ' ;, : “‘, ,,;

    .,/

    Appendix I Comprehensive Child Development Program

    Program Description Program Authorization and The Comprehensive Child
    Development Program was authorized in 1988 Objectives under the
    Comprehensive Child Development Centers Act of 1988 (P. L.

    100297). CCDP was reauthorized through fiscal year 1994 under the Augustus
    F. Hawkins Human Services Reauthorization Act of 1990. In May 1994, elements
    of CCDP were incorporated into a new Head Start Program for Families With
    Infants and Toddlers, authorized under titIe I of P. L. 103252, the Head
    Start Act amendments1

    CCDP was designed to enhance the physical, emotional, and inteIiectuaI
    development of low- income children (infants to school age) and to
    contribute to self- sufficiency by providing support to their parents and
    other family members. This demonstration program's purpose is to test
    whether it is feasible and effective to provide integrated and comprehensive
    support services early in the child's Iife and within the family context.

    Program Operation, Participation Levels, and Funding

    The Department of Health and Human Services funded 24 CCDP grants in 1989
    and 1990 and 10 more in 1992. Each grant provided funding for 5 years. There
    are currently 34 projects in ruraI and urban areas providing core services
    (health care, mental health care, child care, early educational
    intervention, early childhood development, prenatal care, parenting
    education, employment counseling, vocationaI training, adult education, and
    nutritional assistance).

    A CCDP grantee organization (which may be a health clinic, a family services
    agency, a university, or even a school district) acts as a service
    integrator by building supporting networks with community agencies and at
    times facilitating and advocating for services. Each grantee organization
    assesses local needs in its area and plans ways to meet these needs through
    direct intervention of family- focused case management and a combination of
    project- provided services and services arranged through third- party
    providers. AR projects use the case management approach, however, the degree
    of reliance on existing agencies in the community

    ‘This appendix describes CCDP until the May l& 1994, Head Start Act
    amendments and reauthorization.

    Page 62 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Appendix I Comprehensive Child Development Program

    varies from project to project. Each project has an advisory board with
    community, business, and client- family representation. 2

    Fiscal year 1991 funding for this program totaled $24 million; for fiscal
    year 1993, an increase of $23 million ($ 20 million received from the
    Department of Education) brought the total to $47 million, which was also
    the amount requested for fiscal year 1994.

    Administrative Structure The Head Start Bureau of the Administration on
    Children, Youth, and Families in the Department of Health and Human Services
    is responsible for the administration of CCDP grants. ACYF'S functions
    include selecting grantees, providing technical assistance, monitoring their
    compliance with program regulations, and directing contractors in the
    performance of process and impact evaluation studies.

    Mandated Evaluation The authorizing legislation for CCDP mandated an
    evaluation report on program impact and program feasibility that was due on
    October 1,1993. HHS provided an interim evaluation report- on short- term
    program impact and on the structure and mechanisms of service delivery- to
    the Senate Committee on Labor and Human Resources in May 1994. That interim
    report covered the results of approximately the first half of the &year
    cycle of services. A final report covering the entire program is scheduled
    for delivery in March 1996.

    Major Sources of _ To meet the evaluation requirements, the agency has
    arrangsd to collect Information diverse types and sources of information.
    The grantee organizations are

    required to collect data on program operations, user participation, and
    compliance with program regulations. In addition, each project submits
    progress reports on its activities (including vignettes on participants'
    successes) to ACYF. A nonparticipant observes agency operations and records
    information on side effects and local community dynamics in ethnographer
    reports. ACYF conducts site visits that result in a letter assessing
    regulatory compliance and implementation and discussing quality issues (site
    visit reports). These program- monitoring documents also contribute to the
    assessment of CCDP operations and feasibility.

    2ACYF' refers to individual CCDP grantee sites as “centers.” We
    use the term “project” instead to refer to the physical center
    itself, community chsracteristics, and services availsble in the community.

    Page 63 GAO/ PEMD- 96- 1 Improving the Flow of Program Information to the
    Congress

    Appendix I Comprehensive Child Development Program

    In December 1991, the agency published a report that described CCDP and
    start- up and enrollment issues (referred to as the CCDP First Annual
    Report). The agency has recently published its interim evaluation report tc
    the Congress entitled “Comprehensive Child Development Program- A
    National Family Support Demonstration” (referred to as the CCDP
    Interim Report). The CCDP Interim Report provided analyses of program
    implementation and short- term program impacts on families and communities,
    as well as a description of families and projects. In a fmal report to be
    prepared after the enrolled families have received 5 years of services, the
    agency expects to provide a greater understanding of how effects vary with
    the intensity and duration of services, as well as with differences in the
    effects of projects' characteristics. It will also provide a picture of the
    effects of long- term participation on families and children.

    Evaluation Questions and Information Needs

    Interim- Year Questions CCDP projects have reported on start- up problems
    and early progress. The Committee might also ask to be informed of new
    feasibility issues and of program impacts or significant side effects
    (discussed in more detail below) as they become evident at existing and new
    projects under the reauthorized Head Start Program.

    Reauthorization Questions After the Head Start reauthorization of 1994, CCDP
    will not continue as a separate program, but its grantees will continue to
    operate, together with new ones, under the newly authorized Head Start
    Program for Families With Infants and Toddlers. Because this new program
    appears generally similar to CCDP, we expect that the questions posed for
    CCDP could also inform oversight of the new and continuing projects.

    Table I. 1 suggests evaluation questions pertinent to a demonstration
    program that are drawn from the list of core questions in chapter 2 and
    restated in terms of CCDP. The table also identifies indicators or measures
    needed to answer those questions and summarizes related information that is
    currently available. It demonstrates the use of the question- selection
    strategy proposed in chapter 4. The questions and indicators are suggestive
    and not prescriptive for the program. Under our

    Page 64 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Appendix I Comprehensive Child Development Program

    strategy, the Committee and Department staffs would collaborate to work out
    the specific questions and indicators needed.

    Table 1.1: CCDP Evaluation Questions and Related Information Question
    Indicator or measure Available information

    Description What model or approach is Description of required Requirements
    are stated in tested here (core services elements of the model authorizing
    legislation and provided, service delivery program documents and method,
    organization)? were summarized in the

    CCDP Interim and First Annual Reports.

    How do features of the model Narrative descriptions of Management
    information differ across projects? major variants and data system, progress
    reports,

    showing the frequency with and ethnographer reports which each occurs
    provide relevant

    information; CCDP Interim Report notes significant variants on the model
    (for example, type of grantee agency, staffing configurations, and
    interagency agreements).

    What are the conditions Range and amount of Management information (project
    size and philosophy, variation in project system, progress reports, client
    makeup, or community characteristics that affect and ethnographer reports
    resources) under which implementation of program provide this information,
    but services are provided, and model the CCDP Interim Report how do they
    differ? does not compare critical

    project- based characteristics of program operations; the final report is
    expected to make such comparisons.

    Implementation What start- up and operating Observational or participant
    Site visit, ethnographer, and difficulties did projects face survey data
    collected as progress reports described and how were they resolved? program
    was implemented difficulties of putting the

    new projects into operation and providing all core services; both the CCDP
    First Annual and Interim Reports cover start- up and operational
    difficulties and early solutions.

    (continued) Page 56 GAO/ PEMD- 95- 1 Improving the Flow of Program
    Information to the Congress

    Appendix I Comprehensive Child Development Program

    Question What modifications in the model were made by the grantees or
    program (for example, in case management or in arrangements with other
    service providers)?

    Indicator or measure Observation or survey responses describing the
    modifications and reasons for making them, descriptions and classification
    of arrangements between organizations at difl erent times

    Available information Progress, site visit, and ethnographer reports, as
    well as refunding grant applications, contain pertinent information, some of
    which is summarized in the CCDP First Annual Report: the basic program model
    is also described there.

    What has CCDP Data that illuminate The sources mentioned demonstrated about
    the conditions for successful above, supplemented by feasibility of applying
    the implementation, barriers grantees' financial records, program model? to
    implementation, and include the needed

    cost of implementation information; CCDP Interim Report provides information
    on implementation problems and some solutions in the early years of the
    program.

    Targeting Question is not Not applicable Not applicable applicable- client
    selection not discretionary

    impact To what extent did services Measures of children's and improve child
    development family members' skills and and family self- sufficiency?
    behaviors upon enrollment

    and after receiving services; measures of services received; comparable data
    from comparable families that did not participate.

    CCDP Interim Report provides information on a variety of outcomes for the
    child, mother, and family as well as for members of the control group; these
    data can be combined with descriptive data about users and services to
    provide answers about imoact.

    CCDP Interim Report provides programwide child and family outcomes;
    subsequent reports are expected to describe how effects vary by services
    provided and project- based characteristics. Under what conditions, and
    Measures of child and

    with what kinds of services or family outcomes by project delivery
    mechanisms, has the and service, displayed to program been most or least
    show range and variance. successful?

    (continued: Page 56 GAO/ PEMD- 96- l Improving the Flow of Program
    Information to the Congress

    Appendix I Comprehensive Child Development Program

    Question Indicator or measure Available information Side effects Have CCDP
    activities Evidence of addition of new Ethnographer, advisory prompted
    increased services or increased board, and project provision, quality, and
    coordination among director's reports captured coordination of services to
    existing services as an these observations, and the non- CCDP community?
    outgrowth of project positive changes were

    networking efforts noted in CCDP Interim Report.

    Comparative advantage Would a reduced version of CCDP (which included only
    the most viable elements) be more effective than other approaches?

    Estimates of enhanced CCDP evaluation studies family self- sufficiency and
    will provide some impact child development under data, which would be basis
    modified- CCDP, impact for estimates of data from comparison modified- CCDP
    effects; a program or other approach separate study is needed to

    compare these estimated effects with the effects of other programs.

    , Page 67

    ,, I' ‘II,

    GAOL'EMD- 95- l Improving the Flow of Program Information to the Congress

    Appendix II Community Health Centers -

    Program Description Program Authorization and The Community Health Center
    (CHC) program was authorized in 1975 Objectives under section 330 of the
    Public Health Service Act. At the time this report

    was written, the program was authorized through fiscal year 1994, and
    reauthorization for fiscal' year 1995 was expected shortly. The program's
    purpose is to provide access to comprehensive and case- managed primary
    health care services to rural and urban populations living in medically
    underserved areas1 The program also targets “at- risk”
    populations (pregnant women, children, substance abusers, and elderly
    persons), and centers seek to overcome barriers to health care access, such
    as those related to culture and language differences.

    Program Operation, The CHC program funded 579 grantee centers serving 1,575
    sites on a Funding, and Participation budget of $558 million in fiscal year
    1993. (The fiscal year 1994 Levels appropriation was about $603.5 million.)
    All centers provide health care

    services such as physician services, diagnostic laboratory and radiological
    services, and pharmaceutical and emergency services, as well as preventive
    care like immunizations, dentistry, family planning, and vision and hearing
    screening. Other required services include translation services,
    transportation, and referrals to other providers, and centers can opt to
    provide supplemental services like health education and outreach. Services
    are provided to about 6 million medically under- served or “at- risk'
    people a year. The fiscal year 1994 appropriation of $604 miLlion represents
    an increase of $46 million to fund new sites and serve more people.

    CHC centers serve areas (urban and rural) with poor access to medical
    services, and they also provide supporting services targeted to the local
    community's needs (such as transportation, day care, or culturally sensitive
    care to minority groups). Most centers operate on a fee- for- service basis,
    while others provide managed care.

    Administrative Structure The CHC program is administered by the Bureau of
    Primary Health Care. BPHC is part of the Health Resources and Services
    Administration within

    ‘An area is designated “medically underserved” by HHS, in
    codunction with state authorities, based or a formula that includes four
    variables: physician- to- population ratio, infant mortality r& e, poverty
    population, and population 66 years of sge and older. An area may encompass
    a neighborhood or an entire county,

    Page 58 GAOA'EMD- 95- l Improving the Flow of Program Information to the
    Congress

    Appendlx II Community Health Centers

    the Public Health Service of HHS. Regional JJHS offices assist headquarters
    in monitoring grantees' compliance with program regulations. The CHC has a
    subprogram, the Comprehensive Perinatal Care Program, that provides
    additional funding for enhanced perinatal services and is treated here as a
    separate program.

    Mandated Evaluation The last reauthorization did not mandate an evaluation
    or any type of report to the Congress on this program. However, HFEA has
    initiated and conducted several evaluation studies, which are discussed in
    the next section.

    Major Sources of Information The legislation requires (1) that community he&
    h centers have an

    effective procedure for collecting information on their costs of operation,
    (2) that information on the use of services (their availability and
    accessibility) be collected, and (3) that grantees collect data on program
    operations and user participation. HHS regional offices collect and enter
    computerized data received from grantees into the Bureau's Common Reporting
    Requirements (BCRR) data system. BPHC is planning changes in this system to
    allow collection of additional data on clients, their health status and
    needs, and types of services provided. A new provision is intended to result
    in information about the age- appropriate preventive services provided at
    different sites. Grant applications include information on each center, the
    health needs of its community, and progress toward achieving its program
    goals (such as retaining clients).

    BPHC, with the aid of HHS regional offices and on- site assessments
    conducted by federal staff and consultants, reviews center operations and
    compliance with BPHc- mandated budgetary and performance standards (such as
    following standard medical procedures and appropriate staff licensing
    guidelines). This is intended to ensure that centers are properly managed
    and operated. The bureau also has information on special populations' needs
    and nationwide health needs derived from individual grant applications and a
    variety of other sources such as the U. S. census, national health surveys,
    and state and local organizations.

    BPHC has conducted several evaluation studies and is planning others. The
    completed studies have provided baseline information on the range of
    preventive services offered, examined the effects of Federally Qualified He&
    h Center provisions on individual centers and the capacity of each to expand
    its service, and traced the impact of CHC use on some Medicaid

    Page 69 GAWPEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Appendix II Community Health Centers

    costs and hospitalizations. Ongoing studies will provide assessments of the
    case management approach and the managed care system used in certain
    centers. Several planned studies should provide in- depth descriptive
    profiles of CHC users and of services provided, as well as an evaluation of
    program effectiveness and whether CHCS do improve the health status of their
    users. 2

    Evaluation Questions and Information Needs

    Interim- Year Questions The forthcoming reauthorization may introduce new
    provisions that raise implementation questions for the next few interim
    years. Beyond these, questions concerning the emergence of significant
    feasibility or service quality problems and new evidence concerning program
    targeting, impact, or side effects will be relevant.

    Reauthorization Questions The CHC program is now scheduled for
    reauthorization consideration in 1995. Table II. 1 suggests evaluation
    questions pertinent to the CHC program that are drawn from the list of core
    questions in chapter 2 and restated in terms of this program. The table also
    identifies indicators or measures needed to answer those questions and
    summarizes related information that is currently available. This table
    demonstrates the use of the question- selection strategy proposed in chapter
    4.

    2These studies will also evaluate the Migrant Health Center progrsm and, in
    one instance, will combine the two programs and their results. Siice the CHC
    program funds some centers that also get Migrant Health program funding, the
    programs are usually located and operated together. Roughly 100 jointly
    funded centers provide the same basic services to a mix of migrant and
    resident clients using the same staff, thereby making it difficult to
    identify unique CHC program operations and effects. About 450 organizations
    are CHC only, and 20 are Migrant Health only.

    Page 60 GAO/ PEMD- 96- 1 Improving the Flow of Program information to the
    Congress

    Appendix II Community Health Centers

    Table 11.1: CHC Evaluation Questions and Related Information Question
    Indicator or measure Available information

    What services are actually provided, whom do they reach, and in what kinds
    of settings?

    User profile of client User profile information is characteristics
    (insurance accessible in BCRR; status, relative income, information on
    required and ethnicity, age, sex); profile optional services is of services
    provided provided; and information (required services and on center settings
    is range of optional services) available in grant

    apDlications and BCRR. How do centers differ in Profile of a sample of
    Center location and terms of location (rural or centers, noting variation in
    description are available in urban), client characteristics, each feature
    across centers grant applications, and and area population? client
    demographics are

    accessible in BCRR. What other aspects of the program are significant for
    understanding program management and use of federal money?

    Information on relevant Information on these variables (such as payment
    features is available from systems used, percentage BCRR and grant of
    clients covered by applications. insurance, and delivery models)

    Implementation How have recent increases in Number of new sites or
    Information is accessible in funding affected program expansions; number and
    BCRR and program files. implementation with regard to description of newly
    served capacity to serve, services clients provided, and location?

    Does evidence suggest Frequency and severity of Information on important
    significant compliance reports on significant center grantee noncompliance
    problems with centers' problems or noncompliance may be collected from
    mandate to provide minimally with critical requirements diverse sources
    during adequate care or with regard routine program monitoring. to any
    statutorv requirement?

    Are there serious problems Program management Information about the that
    centers face in information suggesting problems centers face is implementing
    the program? problems, noting frequency available in grant

    and severity of situations applications and from the such as unmet staffing
    annual center review; needs, facilities needing headquarters or regional
    renovation, or high staff may also be aware of insurance costs other
    relevant information.

    Targeting To what extent are centers Portion of population in Medically
    underserved covering their medically medically underserved population size
    is available underserved populations or areas or counties that CHCs from
    grant application and areas? serve number of users in BCRR,

    so portion can- be calculated.

    (continued) Page 61 GAO/ PEMD- 95- l Improving the Flow of Program
    Information to the Congress

    Appendll II Community Health Centers

    Question Indicator or measure Available information Are centers distributed
    Proportion of medically around the country to cover underserved areas in U.
    S. the most seriously without a center; underserved populations or
    identification of those areas areas? without centers that cover

    the most severely underserved counties (among all counties)

    Amount of medical need in a specific area is available from grant
    applications.

    Amount of need in unserved areas is available from various national and
    local sources; agency has reported amount of need met by CHC program but not
    its size relative to the amount of unmet national need or the severity of
    need; NACHC has reported on the amount and severity of need.

    To what extent do centers serve special population clients?

    Proportion of clients who are special population members

    Number of users, including some special population users (the elderly,
    children, infants) is available from BCRR; information on each center's
    assessment of services needed and services provided is available in grant
    application and BCRR.

    Are there major client needs Evidence that centers Information on each that
    center services do not provide services that match center's assessment of
    address? needs identified in the services needed and

    grantee's needs services provided is assessment; evidence that available in
    grant program services offered application and BCRR. match program services
    needed

    Impact Have CHC centers with Change in coverage of BCRR has relevant client
    optional, culturally sensitive, clients and increased information over time;
    and other outreach services utilization by clients; evaluation or public
    health succeeded in increasing change in proportion of literature may report
    on access to health care for the ethnic group and other- program outreach
    effects. targeted populations? language members served

    Has the medically Information on continuity of Health status information,
    underserved population client care, reduced use of use of health care
    facilities, served by a center achieved emergency room facilities CHC and
    non- CHC client higher health status? for nonemergency care, information are
    available;

    and change in morbidity in agency has ongoing and area completed evaluation

    studies regarding these issues.

    (continued) Page 62 GAO/ PEMD- 96- l Improving the Flow of Program
    Information to the Congress

    Appendix II Community Health Centers

    Question Indicator or measure Available information Side effects Have there
    been effects on Changes in use of Grant application or ad hoc other health
    care providers? emergency facilities, sources may report this

    hospitals, or school- based information; evaluation care or nursing study on
    change in

    Medicaid usage is avaiiable. Comparative advantage Is providing clinics (as
    Comparison of costs, types Agency has similar opposed to encouraging of
    services provided, and evaluation studies planned. others to provide
    service) clients served between more successful in CHCs and another
    nonclinic increasing access to care for delivery approach the targeted
    populations?

    :: :‘, ‘I ‘Es ,,,, :;

    > :: ‘1, ?:,‘,. , ,, .“-“:,

    ‘) “‘,:,,; *' ,. I,:

    Page 63 ,,, i

    ,I-; “:, ,,

    1, ,, ' )I $,,

    :: ,I( ,\:, 5, (,,

    GAO/ PEMD- 96- l Improving the Flow of Program Information to the Congress
    ._ I,, l,,

    Appendix III apt& 1 Ele entary d Secondary Education Act

    Program Description Program Authorization and The Chapter 1 program
    (originally known as Title I) was established by the Objectives Elementary
    and Secondary Education Act of 1965. This summary

    describes the program as it existed under the 1988 Augustus F. Hawkins-
    Robert T. Stafford Elementary and Secondary School Improvement Amendments
    (P. L. 100- 297). (Substantial changes were introduced in 1994, as discussed
    later in this appendix.) Chapter 1 provides financial assistance to local
    education agencies to help educationally deprived children in low- income
    areas (1) succeed in the regular program of the local education agency, (2)
    attain grade- level proficiency, and (3) improve achievement in basic and
    more advanced skills.

    Program Operation, The program primarily provides compensatory instruction
    in basic Funding, and Participation subjects (reading, mathematics, language
    arts). Some supporting services Levels (social work, health and nutrition,
    transportation) are also provided. In

    most schools, only low- achieving students receive assistance, which may be
    given in the classroom or via “pull out” to a separate setting.
    Schools in which 75 percent or more of the students come from low- income
    families may use Chapter 1 funds to support schoolwide improvement rather
    than serve only the low achievers. Eligible students who attend private
    schools receive services provided off- site through the public schools.
    Parent participation is encouraged.

    With an annual budget of $6.7 billion, Chapter 1 is the‘ largest
    federal program in elementary and secondary education. Grants are awarded to
    state education agencies, and through them to school districts in accordance
    with the numbers of students from low- income families. All states, Puerto
    Rico, the District of Columbia, and the outlying territories participate.
    Funds flow to 90 percent of all school districts, over 70 percent of all
    public elementary schools, and 30 percent of public secondary schools.
    Nearly 5.5 million children are served.

    Administrative Structure Chapter 1 is administered by the Office of
    Compensatory Education Programs within the Department of Education's Office
    of Elementary and Secondary Education. Federal program staff review the work
    of state

    Page 64 GAOREMD- 96- l Improving the Flow of Program Information to the
    Congress

    Appendix III Chapter 1 Elementary and Secondary Education Act

    Chapter 1 program directors, who in turn review compliance and progress at
    the local level.

    Mandated Evaluations The Chapter 1 program was the subject of major mandated
    evaluations in 1977, in 1987, and again in the 1992 National Assessment of
    Chapter 1. The 1992 assessment drew on preliminary findings from the
    “Prospects” study, a national longitudinal study mandated in
    1988 to trace the effects of Chapter 1 participation to young adulthood. The
    Congress has also requested GAO reports on various aspects of the program.

    The 1988 legislation required states to evaluate their Chapter 1 programs at
    least every 2 years. State review of student gains for each school in terms
    of standardized test scores was deemed sufficient to meet this requirement
    and was used to identify schools in need of improvement. Local education
    agencies were to conduct program evaluations every 3 years. At the school
    level, schoolwide projects were to be evaluated after 3 years of operation.
    In addition, individual students' gains were to be examined and
    individualized educational plans devised for those who were not making
    progress.

    Major Sources of Information

    The 1992 National Assessment drew on numerous studies of special issues and
    populations as welI as case studies of school programs. These are summarized
    in its final report1 It also drew on preliminary results from two major
    studies: (1) the “Prospects” study, which examines the school
    experiences and achievement of nationally representative samples of
    disadvantaged lst, 3rd, and 7th graders (some of them receiving Chapter 1
    services, and others not) over many years, and (2) an observational study
    entitled “Special Strategies for Educating Disadvantaged
    Students.”

    Annual state education agency reports to date have listed the number of
    schools, staff, and students participating, as well as the number of schools
    deemed in need of improvement. State agencies also reported the statewide
    average pretest and posttest scores for Chapter 1 students.

    As its basis for allocating funds to states, the program has relied
    primarily on family income data from the decennial census. Local education
    agencies have used data on numbers of poor children from the Department of
    Agriculture's school lunch program to determine school

    ‘U. S. Department of Education, Reinventing Chapter 1: The Current
    Chapter 1 Program and New Directions (Washington, D. C.: Feb. 1993).

    Page 65 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Appendix III Chapter 1 Elementary and Secondary Education Act

    eligibility. Although it cannot distinguish participants in Chapter 1 from
    other students, the National Assessment of Educational Progress provides
    general evidence of how disadvantaged students' achievement in basic
    academic subjects compares with that of more advantaged students.

    Changes Enacted in 1994 The Improving America's Schools Act of 1994 (P. L.
    103- 382) renamed the Chapter 1 program as Title I and introduced
    significant changes. The 1994 law specifies components or qualities that
    funded activities should incorporate- including high standards, enriched
    programs, upgraded instruction, and improved teacher professional
    development- and alters the kinds of information that will be available.
    Under this law, the requirement that student achievement be measured in
    terms of performance on nationally normed standardized tests has been
    dropped. Instead, consistent with the provisions of the Goals 2000: Educate
    America Act (P. L. 103- 227), states are encouraged to evaluate student
    achievement and school performance in terms of standards and assessments
    specific to each state and applicable to all students, not just those
    participating in Title I programs.

    In place of the historic emphasis on providing compensatory services to
    individual students, the Improving America's Schools Act emphasizes
    strengthening the regular program in schools that serve low- income
    students. It would permit many more schools to adopt schoolwide programs (in
    which funding is used to upgrade the entire educational program and
    “Title I students” are not distinct from other students).
    Student achievement and school improvement would be evaluated only at
    certain grades selected by each state (such as 3,8, and 11). Thus, student
    achievement data will be cross- sectional and state- specific. Data will be
    drati from a range of students who attend schoolwide programs and from low
    achievers who have received special services in schools with
    “targeted” programs. At the time of this writing, future
    reporting requirements had not yet been worked out.

    The act provides for data on individual students' year- to- year progress to
    be collected through a national longitudinal study. The longitudinal study
    will compare student performance against content standards of national
    scope. A second mandated evaluation study will examine progress toward the
    goal of having all children served by this program reach their state's
    content and performance standards. This evaluation will also examine the
    targeting of resources and the extent to which the high standards, enriched
    programs, upgraded instruction, parental participation, improved

    Page 66 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Appendix III Chapter 1 Elementary and Secondary Education Act

    teacher professional development, and other program practices specified in
    the act have been put in place. Projects to demonstrate effective practices
    are also authorized.

    Evaluation Questions In identifying the Committee's future information
    needs, we took into and Information Needs

    account the major changes adopted in 1994. Specifically, we assumed (1) that
    school and student performance would be evaluated in terms of standards and
    assessments specific to each state rather than in terms of nationally normed
    tests, (2) that only cross- sectional data for certain grades would be
    available from state records, and (3) that longitidinal data on a sample of
    individual students would be available from the “Prospects”
    study and from the newly authorized longitudinal study. Information may also
    be available from projects funded to demonstrate effective practices.

    Interim- Year Questions In light of these changes, in the years immediately
    following the 1994 reauthorization, the Committee will likely wish to be
    kept informed of

    . progress in implementing the new requirement for state standards and
    assessments and related changes in curriculum, instruction, and staff
    development; . feasibility issues that have arisen as implementation has
    moved forward; l early evidence that new assessments meet criteria of
    technical soundness

    and x. e aligned to state standards; . early evidence of effects of new
    standards and assessment practices on

    student achievement; and l early evidence that the changes have
    unanticipated side effects on

    instruction or on parents, teachers, or students (especially students with
    disabilities or limited English proficiency).

    Reauthorization Questions Table III. 1 lists evaluation questions that the
    Committee may wish to ask at the next ESEA reauthorization (scheduled to
    take place in approximately 5 years), indicators or measures needed to
    answer them, and related information that is currently available. The
    questions are drawn from the list of core questions in chapter 2 and
    restated in terms of this program. As noted earlier, it is unclear what
    information about student and school performance will be reported to the
    administering agency under the new provisions. The question of whether and
    how state data can be aggregated to form a national picture, given that each
    state uses a different standard,

    Page 67 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Appendix III Chapter 1 Elementary and Secondary Education Act

    has yet to be answered. The entries under “Impact” in table III.
    1 suggest one possible method of accomplishing this task. We consider our
    approach illustrative of what might be done. Under our strategy, Committee
    and Department staff would collaborate to work out an approach. 2

    Table 111.1: Chapter 1 ESEA Evaluation Questions and Related Information
    Question

    Description indicator or measure Available information

    How different are High, low, and modal Socioeconomic data participating
    schools and value( s) for each feature; available from free or districts
    from one another in typical (frequently subsidized lunch program; terms of
    socioeconomic occurring) combinations of achievement data from variables,
    student features “Prospects” study; resource performance, and
    school data from special studies resources?

    How do schools differ in Narrative description of Capsule descriptions of
    instructional strategy and major variants and the successful schools, case
    method of delivery and in the school conditions under studies of school
    practices, added services provided to which they occur; and observations
    drawn low- achieving students? frequency count or estimate from site
    reviews;

    of relative frequency of “Prospects” study may each variant
    contain data

    How many states have Number of states that have Standards to be set forth in
    established content and standards; examples or each state's plan; additional
    performance standards, and illustrations of standards details may be needed
    for how widely do these that differ in scope and level adequate description
    standards vary?

    To what extent are state assessments aligned to state content and
    performance standards, of acceptable technical quality, and appropriately
    adapted for limited English proficiency students and students with
    disabilities?

    Findings from reviews of Federal and state quality instrument content,
    criteria and review administration, and scoring procedures not yet methods;
    evidence of established reliability and validity of results

    (continued) 2Consistent with the 1988 as well as the 1994 law, this
    program's central goal is to bring participating students up to a certain
    level of educational performance. Thus, its impact is properly evaluated in
    terms of how many students, and which kinds of participating students,
    achieve this goal. The program also is intended to lead states, school
    districts, and schools to adopt improved practices. We treat these practices
    as standards for program operation, an aspect of implementation.

    Page 68 GAO/ PEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    Appendix III Chapter 1 Elementary and Secondary Education Act

    Question Indicator or measure Available information What proportion of
    Frequency data from school Future evaluation studies participating schools
    have surveys and from authorized to collect programs that exhibit the
    observations of actual relevant data, but such characteristics of quality
    school practice studies not yet designed; education embodied in the state or
    other site reviews authorizing statute? What also a potential source
    proportion lack the essential characteristics of this model?

    Targeting To what extent are services targeted to schools with high
    concentrations of low- income students?

    To what extent are services reaching low- achieving students in these
    schools, as well as students with special educational needs (such as limited
    Enghlish proficiency, disability, or high mobility)?

    Proportion of participating schools in which there are high concentrations
    of students from low- income families; proportions of participating students
    in low, moderate, and high poverty schools

    Proportion of students served/ benefited who are low achieving, have special
    needs, or both

    Data available at the school level but may not be reported;
    “Prospects” study includes relevant data

    Data on achievement, handicap, and English language proficiency for each
    student served available at the school level, but not yet clear whether
    these data to be reported; “Prospects” study contains relevant
    data

    How many severely needy Number of unserved Local education agencies schools
    and students remain schools, by percentage of gather school poverty data
    unserved by this or other low- income and for school selection but
    comparable program? Have low- achieving or need not report it; National
    numbers and percentages multiple- needs students; Center for Education
    decreased since 1994? number of unserved needy Statistics Schools and

    students in all eligible Staffing Survey contains schools relevant
    information

    Impact In the aggregate, is the Measure of student National Assessment of
    performance of economically achievement and economic Educational Progress
    disadvantaged students status for national sample, provides this kind of
    approaching that of their such that distributions can information advantaged
    peers? be compared

    (continued) Page 69 GAOLPEMD- 96- l Improving the Flow of Program
    Information to the Congress

    Appendix III Chapter 1 Elementary and Secondary Education Act

    Question . Indicator or measure Are students served by this Percentage of
    students program being brought to the whose performance (1) is level of
    performance seriously deficient and expected of all students? shows little
    improvement,

    (2) has improved but is still below expected level, (3) has reached the
    expected level but student needs assistance to maintain gains, or (4) is
    sufficiently strong to “graduate” from Chapter 1

    Percentage of students who have received Chapter 1 services for 3 years or
    more and have not reached the expected level

    Available information Schools classify students‘ progress and
    achievement in order to determine who needs extra assistance: however, these
    judgments are not reported, and it may be some years before they can be
    linked to state standards; new longitudinal study may gather relevant data

    “Prospects” study to have length- of- participation data for a
    sample of students

    Under what conditions, and Student performance data “Prospects”
    study to with what kinds of students, linked to school, student, provide
    information; has the program been most and service delivery data additional
    data will be or least successful? Are needed schoolwide programs more
    effective for low achievers than programs that target specific students?

    Has the school improvement Number of schools needing Criteria for school
    procedure led to improved improvement and improvement to be set by student
    outcomes? percentage in which states but are not yet

    student performance has established improved substantially; comparative data
    on performance of low- achieving students in schools that serve similar
    populations but differ in type of program

    Side effects Has the adoption of Observational data on
    “Prospects” study may schoolwide programs extent of services to
    provide some data but may decreased the extent or low- achieving students
    prior include too few cases effectiveness of assistance to and after
    adoption of provided to low- achieving schoolwide programs;. if not students
    in those schools? available, cross- sectional

    comparative data Has the nature, amount, and Percent of districts that New
    issue; what information cost of student testing drop annual testing, cease
    will be available not yet changed substantially? to use a nationally normed
    known

    test, or use either performance- based tests or multiple assessments; per-
    student cost of testing

    (continued) Page 70 GAO/ PEMD- 95- l Improving the Flow of Program
    Information to the Congrees

    Appendix III Chapter 1 Elementary and Secondary Education Act

    Question Indicator or measure Available information Where the nature of
    Proportion of classrooms in New issue; what information assessment has
    changed, to which such changes have will be available not yet what extent
    have parallel been reported or observed known changes occurred in
    instruction?

    Has use of standards- Changes in proportion of “Prospects” study
    may based testing adversely such students who are provide some data but
    affected students who face included in assessment; additional information
    will special barriers (language, patterns of attendance, likely be needed
    handicap) to meeting such effort, and persistence in standards? school on
    the part of such

    students Comparative advantage Is there evidence that tying Change in rate
    of National and state results Chapter 1 criteria to state aggregate
    achievement of from the National standards has been more disadvantaged
    students Assessment of Educational advantageous in relation to nationally
    and in each state: Progress may be useful; its cost than the prior changes
    in costs; case plans for other data approach? (Generally, or only study data
    from varied collection not yet known in some states?) states

    :' .,.‘,,

    ;1 ,:., ),

    :: ::

    Page 71 -;. ,,‘ y

    : 1,:-. : i ,.

    GAO/ PEMD- 96- l Improving the Flow of Program Information to the Congress
    “'

    Appendix IV Comments From the Department of Education

    UNITEDSTATES DEPARTMENT OF EDUCATION OFFXE OF ELEMENTARY AND SECONDARY
    EDUCATION

    THE ASSISlXNT SECRETARY October 19, 1994 Ms. Terry E. Hedrick Assistant
    Comptroller General Program Evaluation and Methodology Division United
    States General Accounting Office Washington, D. C. 20548

    Dear Ms. Hedrick: The Secretary has asked that I respond to your request for
    comments on the GAO draft report, @Tongressional Oversight: Obtaining
    Information for Program Evaluation" (GAO Sob Code 973766), which was
    transmitted to the Department of Education by your letter of September 2,
    1994.

    We believe it is very important for Congress to have timely accurate
    information regarding the operation of our programs and the report prepared
    by GAO will be helpful in furthering this aim.

    We concur with your recommendation that a useful strategy for the Department
    and for Congressional staff is to increase dialogue with each other. Good
    communication, shared planning, and mutually agreed upon approaches are
    essential. Improved dialogue would not only strengthen the work that we do
    but ensure that we provide information Congress believes is important. We
    also concur with the recommended strategy of having some core questions that
    will permit Congress and the Department to have information on programs
    during interim years. Our primary concern, however, is that the program
    goals, indicators, and

    outcome data requested for reauthorization be as consistent as possible with
    the requirements of the Government Perfornance and Results Act (GPHA) which
    is driving our strategic planning and budget processes.

    We do have a concern with GAO's call for assessments "side- effects" and
    "comparative advantage". Questions such as these are often beyond the scope
    of the evaluation of particular programs, and difficult to address if the
    programs used for comparison are administered by another agency.

    Page72 GAOL'EMD- 96- 1 ImprovingtheFlowofProgramInformationtotheCongress

    ” ‘:, Appendix IV Comments From the Department of Education

    I' Ms. Terry E. Hedrick Page 2

    Thank you for the opportunity to comment. My staff and I are prepared to
    respond if you or your representatives have any questions.

    <;, pw Thomas W. Payzant TWP: jt cc: Alan Ginsburg

    Page 73 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Appendix V Comments rom the Department of Health and Human Services

    DEPARTMENT OF HEALTH 8 HUMAN SERVICES Olllca al Inspectcw General
    Washington, D. C. 20201

    OCT I8 I994 Ms. Terry E. Hedrick Assistant Comptroller General United States
    General

    Accounting Office Washington, D. C. 20548

    Dear Ms. Hedrick: Enclosed are the Department, '6 comments on your draft
    report, "Congressional Oversight: Obtaining Information for Program
    Evaluation." The comments represent the tentative position of the Department
    and are subject to reevaluation when the final version of this report is
    received.

    The Department appreciates the opportunity to convnent on this draft report
    before its publication.

    Sincerely yours, Inspector General Enclosure

    Page 74 GAO/ PEMD- 96- 1 Improving the Flow of Program Information to the
    Congress

    Appendix V Comments From the Department of Health and Human Services

    COMMENTS OF THE DEPARTMENT OF HEALTH AND HUMAN SERVICES ON THE GENERAL
    ACCOUNTING OFFICE [GAO) DRAFT REPORT

    "CONGRESSIONAL OVERSIGHT: OBTAINING INFORMATION FOR PROGRAM EVALUATION."
    SEPTEMBER 1994

    The Department has reviewed the General Accounting Office (GAO) draft report
    and has the following comments.

    MATTERS FOR CONGRESSIONAL CONSIDERATION This report proposes a strategy to
    obtain information for program oversight and reauthorization that the Senate
    Committee on Labor and Human Resources may wish to adopt. The three
    components of this strategy are

    -- selecting and adapting, from a core list, the descriptive and evaluative
    questions to be asked about a program in interim years and at
    reauthorization;

    -- arranging explicitly to obtain timely oversight information in interim
    years as well as to receive results of evaluation studies at
    reauthorization; and

    -- providing for increased communication with agency staff to help ensure
    that information needs are understood and that requests and reports are
    suitably framed and adapted as needs evolve.

    This strategy can be adapted to take institutional realities into account.
    For example, in view of the many demands on its attention, the Committee
    might select future reauthorization questions for some programs and invite
    agencies to propose questions for others.

    DEPARTMENT COMMENT We generally support the GAO draft report's
    recommendation that congressional committees more clearly express their
    evaluative information needs to agencies and consult with agencies on those
    needs. However, we have concerns about GAO recommending a process that would
    burden agencies with additional evaluation reporting requirements on top of
    the extensive annual reporting of performance required under the recently
    enacted Government Performance and Results Act of 1993 (GPRA).

    From our analysis, much of the information GAO cites as useful to
    congressional committees for both annual oversight and reauthorization
    activities could be part of the annual performance report required of each
    program activity under .GPRA. The GAO draft report, however, does not give
    adequate recognition to this fact or to the point that Congress

    Page 75 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Appendix V Comments From the Department of Health and Human Services

    recently addressed the need for regular evaluative information by enacting
    GPRA.

    Page 76 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Appendix VI Major Contributors to This. Report

    Divisior man, Assistant Director

    I Venkareddy Chekareddy, Referencer

    Page 77 GAOIPEMD- 95- 1 Improving the Flow of Program Information to the
    Congress

    ' Bibliography References on Program Evaluation and Congressional Decision-
    Making

    Abramson, Mark. The Funding of Social Knowledge Production and Application:
    A Survey of Federal Agencies. Washington, D. C.: National Academy of
    Sciences, 1978.

    BarkdoIl, Gerald L., and James B. BeII (eds.). Evaluation and the Federal
    Decision Maker. New Directions in Program Evaluation, No. 41. San Francisco:
    Jossey- Bass, 1989.

    Bimber, Bruce. Congressional Support Agency Products and Services for
    Science and Technology Issues. Report prepared for the Carnegie Commission
    on Science, Technology, and Government, Sept. 17,199O.

    Boyer, John F., and Laura Langbein. “Factors Influencing the Use of
    Health Evaluation Research in Congress.” Evaluation Review, 15: 5
    (Oct. 1991), 507- 32.

    Boruch, Robert, David Cordray, Georgine Pion, and Laura Leviton. “A
    Mandated Appraisal of Evaluation Practices: Digest of Recommendations to the
    Congress and the Department of Education.” Educational Researcher, 10~
    4 (Apr. 1981), 1031.

    Bozeman, Barry, and Julia Melkers (eds.). Evaluating R& D Impacts: Methods
    and Practice. Boston: Kluwer Academic, 1993.

    Caro, Francis G. “Leverage and Evaluation Effectiveness.”
    Evaluation and Program Planning, 3 (1980), 83- 89.

    Chehmsky, Eleanor (ed.). “Program EvaIuation: Patterns and
    Directions.” American Society for Public Administration, 1985.

    Citro, Constance F., and Eric A. Hanushek (eds.). Improving Information for
    Social Policy Decisions: The Uses of Microsimulation Modeling, Volume I,
    Review and Recommendations. Washington, D. C.: National Academy Press, 1991.

    Cook, Thomas D., and William R. Shadish, Jr. “Metaevaluation: An
    Assessment of the Congressional Mandated Evaluation System for Community
    Mental He& h Centers.” In Gerald L. StahIer and WiIIiam Tash (eds.),
    Innovative Approaches to Mental Health Evaluation, pp. 221- 53. New York
    Academic Press, 1982.

    Page 78 GAO/ F'EMD- 96- l Improving the Flow of Program Information to the
    Congress

    Bibliography Cronbach, Lee J., and Associates. Toward Reform of Program
    EvaIuation. San Francisco: Jossey- Bass, 1981.

    Levitan, Sar A, and Gregory Wurzburg. Evaluating Federal Social Programs.
    Philadelphia: W. E. Upjohn Institute for Employment Research, Sept. 1979.

    Lowi, Theodore J. “American Business, Public Policy, Case Studies, and
    Political Theory.” World Politics, 164 (1964), 677- 715.

    Martin, Margaret E., and Miron L. St- raf (eds.). Principles and Practices
    for a Federal Statistical Agency. Washington, D. C.: National Academy Press,
    1992.

    Mohr, Lawrence B. Impact Analysis for Program Evaluation. Newbury Park,
    Calif.: Sage Publications, 1992.

    Rossi, Peter H., and Howard E. Freeman. Evaluation: A Systematic Approach,
    4th Ed. Newbury Park, Calif: Sage Publications, 1989.

    SaIamon, Lester M. (ed.). Beyond Privatization: The Tools of Government
    Action. Washington, D. C.: The Urban Institute Press, 1989.

    Striven, Michael. “Hard- Won Lessons in Program Evaluation.” New
    Directions for Program Evaluation, 58 (Summer 1993).

    Thompson, Mark. Decision Analysis for Program Evaluation. Cambridge, Mass.:
    BaUinger Publication, 1982.

    Weiss, Carol H. Using Social Research in Public Policy Making. Lexington,
    Mass.: Lexington Books, 1977.

    Wholey, Joseph S. Federal Evaluation Policy: Analyzing the Effects of Public
    Programs. Washington, D. C.: Urban Institute, 1970.

    WindIe, Charles, and J. Richard Woy. “From Programs to Systems:
    Implications for Program Evaluation Ihustrated by the Community Mental
    Health Centers Program Experience.” Evaluation and Program Planning, 6
    (1983), 53- 68.

    Woods, Patricia. The Dynamics of Congress: A Guide to the People and Process
    in the Congress. Washington, D. C: Woods Institute, 1987.

    Page 79 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    Bibliography .2s for Case 3grarns

    Advisory Committee on Head Start Quality and Expansion. “Creating a
    21st Century Head Start.” FinaI Report. Washington, D. C.: Dec. 1993.

    Arnold, Judith, Ann Zuvekas, and Nina Teicholz. “A Manual for Using
    Health Outcome Measures to Evaluate the Primary Care System.”
    Discussion Draft, Feb. 1989. Washington, D. C.: Lewin/ ICF, Task Order #340-
    86- 0504.

    Freeman, Howard E., KIJ. Kiecolt, and AlIan Harris, II. “Community
    Health Centers: An Initiative of Enduring Utility.” Milbank Memorial
    Fund Quarterly: Health and Society, 60: 2 (1982), 24567.

    Hawkins, Daniel, Jr., and Sara Rosenbaum. “Lives in the Balance: The
    Health Status of America's Medically Underserved PopuIations.” Special
    report. Washington, D. C.: NACHC, Mar. 1993.

    Puma Michael J., et al. Prospects: The Congressionahy Mandated Study of -
    Educational Growth and Opportunity- The- Interim Report. Washington, D. C.:
    U. S. Department of Education, 1993. Stringfield, Sam, et al. Urban and
    Suburban/ Rural Special Strategies for Educating Disadvantaged Children:
    First Year Report. Washington, D. C.: U. S. Department of Education, 1994.

    U. S. Department of Education. Reinventing Chapter 1: The Current Chapter 1
    Program and New Directions. Washington, D. C.: 1993. i

    ‘This report drew on many other reports that were included in our
    review but are too numerous to be listed here. A full list of those reports
    is available from the Department of Education's Planning and Evaluation
    Service.

    Page 80 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress

    ‘1 1, “I “' !i

    ,, ' 1 ‘i ” I 11

    1. : Page 81 GAO/ PEMD- 96- l Improving the Flow of Program Information to
    the Congress

    1% : ,'

    ‘I Page 82 GAO/ PEMD- 95- 1 Improving the Flow of Program Information
    to the Congresle

    :I' ,; / .‘,,

    ,‘.,, S, ,. T, 8: : ‘-: /, 2,

    I, ,I,:

    ,” ,: Page 83 GAO/ PEMD- 95- 1 Improving the Flow of Program
    Information to the Congress

    Related GAO Products (978766)

    Elementary School ChiIdren: Many Change Schools Frequently, Harming Their
    Education (GAO/ HE& 94- 45, Feb. 4, 1994).

    Public Health Service: Evaluation Set- Aside Has Not Realized Its Potential
    to Inform the Congress (GAO/ PEMD- 93- 13, Apr. 8, 1993).

    Exiting Program Improvement (GAOLBRD- 93- 2R, Mar. 30, 1993). Chapter 1
    Accountability: Greater Focus on Program Goals Needed (GAO/ HRD- 93- 69,
    Mar. 29, 1993).

    Improving Government: Measuring Performance and Acting on Proposals for
    Change (GAO/ T- GGD- 9% 14, Mar. 23 1993).

    Compensatory Education: Difficulties in Measuring Comparability of Resources
    Within School Districts (GAO/ nun- 93- 37, Mar. 11, 1993).

    Improving Government: Need to Reexamine Organization and Performance (GAO/
    T- GGD- 93- 9, Mar. 11, 1993).

    Compensatory Education: Additional Funds Help More Private School Students
    Receive Chapter 1 Services (GAo/ nnn-%- 65, Feb. 26,1993).

    Government Management Issues (GAO/ OCG- 93- 3TR, Dec. 1992). Children's
    Programs: A Comparative Evaluation Framework and Five R. histrations (GAO/
    PEMDS~~ BR, Aug. 31,1988).

    Page 84 GAO/ PEMD- 96- l Improving the Flow of Program Information to the
    Congress
    *** End of document ***