    Juvenile Justice: OJJDP Reporting Requirements for Discretionary
    and Formula Grantees and Concerns About Evaluation Studies
    (30-OCT-01, GAO-02-23).

    Although national rates of violent juvenile crime and youth
    victimization have declined during the past five years, critical
    problems affecting juveniles, such as drug dependency, the spread
    of gangs, and child abuse and neglect, persist. The Office of
    Juvenile Justice and Delinquency Prevention (OJJDP) has funded
    various demonstration, replication, research and evaluation, and
    training and technical assistance programs to prevent and respond
    to juvenile delinquency and juvenile victimization. GAO's review
    of 16 of OJJDP's major programs showed that although virtually
    all grantees must report on their progress twice a year, the
    information they reported varied. Grantees receive standard,
    general guidance for reporting on their projects and provide
    OJJDP information to monitor grantee's projects and
    accomplishments. According to OJJDP officials, such guidance
    needs to be general, because of differences among individual
    projects and local needs and circumstances. GAO identified eight
    programs in which all grantees reported the number of juveniles
    they directly served. OJJDP does not require grantees in all its
    programs to report directly on the number of juveniles served
    directly because many of its programs are not intended to serve
    juveniles directly. GAO's in-depth review of OJJDP's 10 impact
    evaluations undertaken since 1995 raises some concerns about
    whether the evaluations will produce definitive results. In some
    of these evaluations, variations in how the programs are
    implemented across sites make it difficult to interpret
    evaluation results.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-02-23
        ACCNO:   A02064
      TITLE:     Juvenile Justice: OJJDP Reporting Requirements for
    Discretionary and Formula Grantees and Concerns About Evaluation
    Studies
         DATE:   10/30/2001
      SUBJECT:   Crimes or offenses
    	     Grant monitoring
    	     Juvenile offender rehabilitation
    	     Juvenile status offenders
    	     Reporting requirements
    	     Act Now Truancy Program
    	     Child Abuse and Neglect Program
    	     Children's Advocacy Centers Program
    	     Community Prevention Grants Program
    	     Comprehensive Gang Initiative
    	     Court Appointed Special Advocate Program
    	     DOJ Juvenile Mentoring Program
    	     Drug-Free Communities Support Program
    	     Enforcing the Underage Drinking Laws
    	     Program

    	     Formula Grants Program
    	     Internet Crimes Against Children Task
    	     Force Program

    	     Juvenile Accountability Incentive Block
    	     Grants Program

    	     Missing and Exploited Children Program
    	     OJJDP Model Courts Program
    	     Partnerships to Reduce Juvenile Gun
    	     Violence Program

    	     Positive Action Through Holistic
    	     Education Program

    	     Rural Gang Initiative
    	     Safe Futures Program
    	     Tribal Youth Program

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Testimony.                                               **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-02-23

    Report to the Honorable Bob Schaffer, House of Representatives

    United States General Accounting Office

    GAO

    October 2001 JUVENILE JUSTICE OJJDP Reporting Requirements for Discretionary
    and Formula Grantees and Concerns About Evaluation Studies

    GAO- 02- 23

    Page i GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns Letter
    1

    Results in Brief 2 Background 4 Scope and Methodology 7 All Grantees Receive
    Standard Reporting Guidance and Some Are

    Required to Report Specific Data 10 Some Data Exist on the Number of
    Juveniles Served Directly,

    Although Most Grantees Are Not Required to Report This Information 18
    Methodological Concerns Could Adversely Affect Evaluation

    Studies 22 OJJDP Has Funded Other Types of Evaluations 26 Conclusions 26
    Recommendation for Executive Action 27 Agency Comments and Our Evaluation 27

    Appendix I OJJDP Awards Data, Fiscal Years 1996 Through 2000 30

    Appendix II OJJDP?s Process for Disseminating Published Products, Including
    Interim Results of Impact Evaluations 35

    Appendix III Categorical Assistance Progress Report Form 37

    Appendix IV Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information 39

    Appendix V Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees 48

    Performance Data for Training and Technical Assistance Grantees 48
    Performance Data for Research Grantees 51 Contents

    Page ii GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
    Appendix VI States? Compliance With Core Requirements of the

    Juvenile Justice and Delinquency Prevention Act of 1974 55

    Appendix VII Impact Evaluations OJJDP Has Funded of Its Own Programs 56

    Impact Evaluations of OJJDP Programs in Their Formative Stages 56 Impact
    Evaluations of OJJDP Programs Well Into Their

    Implementation 62 Impact Evaluation of OJJDP Program- Summary of Evaluation
    We

    Did Not Assess 68

    Appendix VIII Twenty- Four Other Evaluations Funded by OJJDP 69 OJJDP-
    Funded Programs: 11 Nonimpact Evaluations 69 Non- OJJDP- Funded Programs:
    Nine Impact Evaluations 74 Non- OJJDP- Funded Programs: Four Nonimpact
    Evaluations 78

    Appendix IX Comments from the Department of Justice 80 GAO Comments 93

    Appendix X GAO Contacts and Staff Acknowledgments 95 GAO Contacts 95
    Acknowledgments 95

    Bibliography 96

    Tables

    Table 1: Reasons for Reporting Requirements for 16 Selected Major OJJDP
    Programs 11 Table 2: OJJDP Programs GAO Identified in Which All Grantees

    Reported Data on the Number of Juveniles Directly Served, Fiscal Year 2000
    19 Table 3: OJJDP Funds Awarded by Program Area, Fiscal Years

    1996 Through 2000 31 Table 4: OJJDP Award Recipients, Fiscal Years 1996
    Through 2000 32

    Page iii GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 5: OJJDP Formula/ Block Grant Awards, by State, Fiscal Years 1996
    Through 2000 33 Table 6: Dissemination of Published Products With Interim
    Results

    of Impact Evaluations, as of May 2001 36 Table 7: Eight Programs GAO
    Reviewed With General Reporting

    Requirements Only 40 Table 8: Eight Programs GAO Reviewed With Specific
    Reporting

    Requirements and General Reporting Requirements 44 Table 9: Training and
    Technical Assistance Performance Data

    Reported by OJJDP Grantees 50 Table 10: Active OJJDP Grantees? Research
    Products, Published by

    OJJDP 53 Table 11: Active OJJDP Grantees? Research Products, Published

    Externally 53 Table 12: State- Reported Compliance with Core Requirements,

    Calendar Year 1998 55

    Figures

    Figure 1: OJJDP Award Recipients, Fiscal Year 2000 7 Figure 2: OJJDP
    Formula/ Block Grants Versus Discretionary Grant

    Awards, Fiscal Years 1996 Through 2000 30

    Abbreviations

    CPD Child Protection Division IPR Individual Project Report JUMP Juvenile
    Mentoring Program NCJRS National Criminal Justice Reference Service NCMEC
    National Center for Missing and Exploited Children OJJDP Office of Juvenile
    Justice and Delinquency Prevention OJP Office of Justice Programs PATHE
    Positive Action Through Holistic Education RPDD Research and Program
    Development Division STAD State and Tribal Assistance Division TTAD Training
    and Technical Assistance Division

    Page 1 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    October 30, 2001 The Honorable Bob Schaffer House of Representatives

    Dear Mr. Schaffer: Although the nation?s rates of violent juvenile crime and
    youth victimization have declined dramatically over the past 5 years,
    critical problems affecting juveniles and the juvenile justice system still
    remain, such as drug dependency, the spread of gangs, and child abuse and
    neglect. To address these and other issues, the Office of Juvenile Justice
    and Delinquency Prevention (OJJDP) has funded a variety of demonstration,
    replication, research and evaluation, and training and technical assistance
    programs aimed at preventing and responding to juvenile delinquency and
    juvenile victimization. Questions have been raised, however, regarding what
    these numerous and diverse programs have accomplished.

    You asked us to review OJJDP?s major programs, as well as the evaluations it
    has funded. Specifically, we agreed to provide (1) information on
    programmatic reporting requirements for OJJDP grantees, the reasons for
    these requirements, and examples of the information grantees have reported;
    (2) information on how many juveniles OJJDP grantees reported directly
    serving in fiscal year 2000, and whether OJJDP requires grantees to report
    the number of juveniles they directly serve, and if not, why; and (3)
    analysis of the methodological rigor of the impact evaluations OJJDP has
    funded of its own programs since 1995, and information on the other types of
    evaluations OJJDP has funded. In addition, you asked us to provide
    information on how much OJJDP awarded to various grant programs from fiscal
    years 1996 through 2000 and the types of organizations that received these
    awards, and we have provided this information in appendix I. You asked us
    how OJJDP disseminates published interim results of impact evaluations as
    well as other published publications OJJDP and its grantees produce, and we
    have provided this information in appendix II. You also asked us to review
    how OJJDP monitors its grantees, and we have reported to you on this issue.
    1

    1 Juvenile Justice: Better Documentation of Discretionary Grant Monitoring
    Is Needed

    (GAO- 02- 65, Oct. 10, 2001).

    United States General Accounting Office Washington, DC 20548

    Page 2 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Our review of 16 of OJJDP?s major programs showed that although virtually
    all grantees are required to report on their progress twice a year, the
    information they reported varied. Grantees receive standard, general
    guidance for reporting on their projects and providing OJJDP information
    with which to monitor grantees? projects and accomplishments. According to
    OJJDP officials, such guidance needs to be somewhat general, given the
    variation that can occur among individual projects as grantees try to meet
    local needs and circumstances. For example, one Drug- Free Communities
    Support Program grantee reported that it had helped students produce an
    anti- smoking commercial, while another grantee in the same program reported
    making presentations on drug abuse to young men at the local juvenile
    detention facility. Grantees in 8 of these 16 programs are subject to
    additional reporting requirements tailored to each of their programs, and
    these specific requirements were established primarily to support outside
    evaluations or for other program or project assessment purposes. For
    example, OJJDP requires all Juvenile Mentoring Program grantees to cooperate
    fully with an outside evaluator. Grantees in this program are required to
    report prescribed demographic information for both juveniles and mentors who
    participate in their programs, as well as other specified data.

    We identified eight programs in which all grantees reported the number of
    juveniles they directly served. Grantees in these eight programs reported
    serving about 142,000 juveniles in fiscal year 2000. 2 For example, Court
    Appointed Special Advocate Program grantees reported serving over 70,000
    juveniles through volunteers who advocate for the best interests of abused
    and neglected children who come before the court. Juvenile Mentoring Program
    grantees reported serving about 8, 500 juveniles by providing one- on- one
    mentoring for at- risk youths. OJJDP does not require grantees in all its
    programs to report on the number of juveniles they serve directly for
    several reasons, including that many of its programs are not intended to
    serve juveniles directly. Instead, OJJDP programs have a variety of other
    purposes that can indirectly benefit juveniles, such as helping state and
    local governments improve their juvenile justice systems, providing training
    and technical assistance to juvenile justice professionals, and establishing
    a centralized research effort on problems of juvenile delinquency. In
    commenting on a draft of this report, the Assistant Attorney General stated
    that research projects do not typically

    2 Grantees in three of the eight programs reported data for time periods
    that did not correspond precisely with the fiscal year. Results in Brief

    Page 3 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    provide services directly, yet their results can potentially help thousands
    of juveniles.

    Our in- depth review of 10 of OJJDP?s impact evaluations 3 undertaken since
    1995 of its own programs raises some concerns about whether many of the
    evaluations will produce definitive results. All of these evaluations are
    still ongoing, with half in their formative stages and half well into
    implementation. While program evaluation is an inherently difficult task, in
    some of these evaluations, the effort is particularly arduous because of
    variations across sites in how the programs are implemented. These
    variations will make it difficult to interpret evaluation results. Two of
    the evaluations that are in their later stages and three of those that are
    in their formative stages lack specific plans for comparison groups, which
    would aid in isolating the impacts of the program from the effects of other
    factors that may have influenced change. Furthermore, three of the five
    evaluations that are well into implementation have developed data collection
    problems. We are making a recommendation in relation to the five evaluations
    that are in their formative stages, intending to ensure that potential
    problems related to comparison groups and data collection are mitigated.

    The Assistant Attorney General provided us with written comments on a draft
    of this report in a letter dated October 15, 2001. In general, she agreed
    with our findings and recommendation. However, she disagreed with our focus
    on the use of comparison groups as the only valid evaluation design.
    Although she agreed that the inclusion of comparison groups would strengthen
    the interpretation of evaluation results, she said that sufficient funds are
    not available to include comparison groups in every evaluation and that
    laboratory- like comparison groups may not be possible when evaluating real
    life community- based programs. We recognize that not all evaluation issues
    that can compromise results are resolvable, including the use of comparison
    groups. However, the validity of the evaluation results can be enhanced
    through establishing and tracking comparison groups. If other ways exist to
    effectively isolate the impacts of a program, comparison groups may not be
    needed. However, in the 10 evaluations for which we assessed methodological
    rigor, we saw no evidence of other methods being used. While studies that do
    not have appropriate comparison groups can provide useful information, they

    3 Impact evaluations assess the extent to which a program causes changes in
    the desired direction in the target population.

    Page 4 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    should not be considered impact evaluations. The full text of the Assistant
    Attorney General?s comments and our evaluation of them are presented in
    appendix IX and elsewhere in this report, as appropriate.

    OJJDP, one of the components of the U. S. Department of Justice, Office of
    Justice Programs (OJP), was established by the Juvenile Justice and
    Delinquency Prevention Act of 1974 (Juvenile Justice Act). 4 Its mission is
    to provide national leadership, coordination, and resources to prevent and
    respond appropriately to juvenile delinquency and juvenile victimization.
    OJJDP accomplishes its mission through developing and implementing
    prevention programs and a juvenile justice system that protects the public
    safety, holds juvenile offenders accountable, and provides treatment and
    rehabilitative services based on the needs of juveniles and their families.
    OJJDP funds research and evaluation efforts, statistical studies, and
    demonstration programs; provides technical assistance and training; produces
    and distributes publications and other products containing information about
    juvenile justice topics; oversees activities dealing with missing and
    exploited children; and administers a wide variety of grant programs. OJJDP
    funds programs that serve juveniles directly as well as those that benefit
    juveniles more indirectly by focusing on system- wide changes or by
    increasing the capacity of governmental units or organizations.

    OJJDP awards grants to states, territories, localities, and private
    organizations through five formula and block grant (formula/ block grant)
    programs and numerous discretionary grant programs. OJJDP administers four
    formula grant programs 5 that provide funds directly to states and
    territories on the basis of states? juvenile populations, and one block
    grant program 6 that awards a fixed level of funds to all states and
    territories. Under these formula/ block grant programs, states may, in turn,
    make subawards to other organizations such as units of local government.
    OJJDP awards discretionary grants through a competitive process to state

    4 42 U. S. C. 5601, et seq. 5 OJJDP?s four formula grant programs are the
    Formula Grants Program, the Community Prevention Grants Program, State
    Challenge Activities, and the Juvenile Accountability Incentive Block Grants
    Program.

    6 OJJDP?s one block grant program is the Enforcing the Underage Drinking
    Laws Program. The Enforcing the Underage Drinking Laws Program has both a
    block grant component and a discretionary grant component. Background

    Page 5 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    governments, local governments, or individual agencies and organizations.
    OJP is responsible for the financial monitoring 7 of grantees (i. e., it
    provides policy guidance, control, and support services in the financial
    management of grants), whereas OJJDP is responsible for program monitoring.
    For program monitoring purposes, OJJDP assigns each of its grantees a
    program manager who is responsible for ensuring administrative and
    programmatic compliance with relevant statutes, regulations, policies and
    guidelines of awarded grants. The program manager is also responsible for
    monitoring grantees? performance and progress as related to grantees? stated
    goals and objectives.

    OJJDP?s budget has increased significantly over the last 5 years- from about
    $188 million in fiscal year 1997 to about $596 million in fiscal year 2001.
    During this time, the Congress has created new programs and increased
    appropriations for some existing ones. The Congress has also provided
    direction each year regarding certain program areas OJJDP should fund.
    Overall, in fiscal year 2001, 31 percent of OJJDP?s available funds 8 were
    congressionally earmarked- that is, set aside for an identifiable grantee,
    specified amount, and/ or specific authorized purpose. For fiscal year 2001,
    $180 million of OJJDP?s funds were available for discretionary grant awards,
    of which 77 percent was earmarked.

    OJJDP awards the majority of its funds to grantees in its five formula/
    block grant programs. In fiscal year 2000, the latest year for which awards
    data were available, OJJDP awarded (1) about $354 million, or 64 percent of
    the total funds awarded, to these formula/ block grant programs and (2) just
    over $200 million, or 36 percent of the total funds awarded, to a wide range
    of discretionary grant programs. The programs awarded the most funds in
    fiscal year 2000 were the Juvenile Accountability Incentive Block Grants
    Program ($ 221 million), the Formula Grants Program ($ 70 million), the
    Community Prevention Grants Program ($ 36 million), the Child Abuse and
    Neglect Program 9 ($ 32 million), the Missing and

    7 OJP conducts financial monitoring of grantees through the Monitoring
    Division of its Office of the Comptroller. 8 Of OJJDP?s $597 million dollar
    appropriation, only $583 million were ?available;? about $12. 7 million were
    designated to support OJJDP administrative costs and $1. 3 million (. 0022)
    was rescinded.

    9 This program encompasses several other programs within OJJDP?s Child
    Protection Division including the Court Appointed Special Advocate Program,
    Children?s Advocacy Centers, the Model Courts Program, and the Safe Start
    Initiative. OJJDP?s Fiscal Year 2000

    Awards

    Page 6 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Exploited Children Program 10 ($ 32 million), and the Drug- Free Communities
    Support Program ($ 30 million).

    OJJDP awards funds to a wide range of recipients, with the majority of
    awarded funds going to state governments. As shown in figure 1, 67 percent
    of the funds OJJDP awarded in fiscal year 2000 went to states, 20 percent to
    nonprofit organizations, 6 percent to school districts or educational
    institutions, and 5 percent to local governments. However, because many
    grantees make subawards to other entities, the awards to these grantees do
    not reflect the ultimate recipients of the funds OJJDP awards. For example,
    under the Formula Grants Program, states pass through a minimum of two-
    thirds of their awarded funds to public and private nonprofit organizations.
    (See app. I for data from fiscal years 1996 through 2000 on (1) OJJDP funds
    awarded to formula/ block grant versus discretionary grant programs, (2)
    OJJDP awards by program area, (3) OJJDP award recipients, and (4) OJJDP
    formula/ block grant awards by state.)

    10 This program encompasses several other programs within OJJDP?s Child
    Protection Division including the National Clearinghouse and Resource Center
    for Missing and Exploited Children, the Missing and Exploited Children
    Training and Technical Assistance program, and the Internet Crimes Against
    Children Task Force Program.

    Page 7 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Figure 1: OJJDP Award Recipients, Fiscal Year 2000

    Note: Percentages do not total 100 due to rounding. a Local governments
    include county governments, cities, townships, and special district
    governments.

    b Schools include independent school districts and state and private
    institutions of higher education. c Other includes for- profit
    organizations, individuals, Indian Tribes, and other miscellaneous
    recipients.

    Source: GAO analysis of OJP database.

    To identify programmatic reporting requirements 11 for OJJDP grantees, the
    reasons for these requirements, and examples of information grantees have
    reported, we reviewed all 5 of OJJDP?s formula/ block grant programs and we
    selected 11 of its major discretionary grant programs to review based on
    OJJDP officials? input regarding which programs were ?major?

    (e. g., number of grantees, program funding, and/ or importance of the
    program). To identify grantee reporting requirements, and the purpose of
    these requirements, we met with the OJJDP program managers who monitor each
    of these 16 programs or with other key officials. For those

    11 We did not review requirements associated with OJP?s financial monitoring
    of OJJDP grantees. Scope and

    Methodology

    State Governments

    67%

    Nonprofit Organizations

    20%

    Other c Schools

    6%

    b Local Governments a 5%

    3%

    Page 8 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    programs in which OJJDP funded an outside evaluation, we also met with
    program managers who oversee the evaluations. We reviewed OJP?s Grant
    Management Policies and Procedures Manual (January 19, 2001) and Categorical
    Assistance Progress Report (progress report) form 12 along with the
    instructions for completing the form. (See app. III for a copy of the
    progress report form.) We also reviewed OJJDP program documents for each of
    the 16 programs, including any special reporting conditions. To supplement
    program documents, we reviewed relevant documents from outside evaluators.
    We did not assess the adequacy of reporting requirements established by
    OJJDP or the outside evaluators.

    To identify specific examples of performance data that grantees reported, we
    asked OJJDP officials to provide progress reports for each program
    demonstrating a range of detail and, in some cases, we asked for reports
    from specific grantees. For each program, we then reviewed 3 to 15 progress
    reports (or individual performance reports) submitted between 1998 and 2001.
    We did not review progress reports from all grantees in every program, nor
    did we review grantees? compliance with reporting requirements. For programs
    being evaluated by an outside evaluator, we reviewed performance data that
    program grantees reported to those evaluators, when available.

    To determine whether OJJDP requires grantees to report the number of
    juveniles they serve directly and to identify the number of juveniles OJJDP
    grantees reported serving in fiscal year 2000, we interviewed OJJDP program
    managers for each of the 16 programs we reviewed and examined relevant
    program documents, including selected semiannual progress reports. To
    determine whether other programs directly serve juveniles, obtain available
    data on the numbers of juveniles served, and learn why OJJDP does or does
    not require grantees to report such data, we reviewed OJJDP program
    literature and met with OJJDP division directors. Nevertheless, the list of
    OJJDP programs we identified as directly serving juveniles may not be
    comprehensive. We focused our data collection effort on only those programs
    we identified in which all grantees reported juveniles- served data.

    12 For the Formula Grants Program, we reviewed OJJDP?s instructions for
    completing the annual Performance Report, including the Individual Project
    Report forms, because Formula Grants grantees are required to complete these
    reports in lieu of progress reports.

    Page 9 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    To assess the methodological rigor of the impact evaluations OJJDP has
    funded since 1995 of its own programs, and to provide information on the
    other types of evaluations OJJDP has funded, we asked OJJDP to identify all
    program evaluations it had funded since 1995. For each evaluation, we asked
    OJJDP to indicate whether it was an impact evaluation, whether the program
    being evaluated was funded by OJJDP, and whether the evaluation had been
    completed or was ongoing. Overall, OJJDP identified 35 evaluations funded
    since 1995.

    Eleven of the 35 evaluations were impact evaluations of OJJDP programs, and
    all were ongoing. 13 For each of the 10 impact evaluations we assessed, we
    asked OJJDP to provide any documentation relevant to the design and
    implementation of the evaluation methodologies, such as the initial and
    supplemental proposals, peer review documents, progress reports, reports of
    interim results, and correspondence between OJJDP and the evaluators. In
    addition, we contacted OJJDP officials to resolve any questions that we had
    regarding the documentation and to request any missing documents. We did not
    contact the program manager responsible for each evaluation.

    To assess the methodological rigor of the 10 impact evaluations, we used a
    data collection instrument to collect information systematically on each
    program being evaluated and the features of the evaluation methodology. We
    based our data collection and assessments on generally accepted social
    science standards. We examined such factors as whether evaluation data were
    collected before and after program implementation, how program effects were
    isolated (i. e., the use of nonprogram participant comparison groups or
    statistical controls), and the appropriateness of sampling, outcome
    measures, statistical analyses, and any reported results. A social scientist
    with training and experience in evaluation research and methodology read and
    coded the documentation for each evaluation. A second social scientist
    reviewed each completed data collection instrument and the relevant
    documentation for the impact

    13 During the course of our review, OJJDP officials identified 10 of their
    35 ongoing evaluations as impact evaluations of OJJDP- funded programs. In
    commenting on a draft of this report, the Assistance Attorney General said
    that the Teen Courts evaluation has a comparison group and should have also
    been classified as an impact evaluation of an OJJDP- funded program, along
    with the other 10 evaluations. Because this had not been included as an
    evaluation of an OJJDP- funded program when we initially requested the
    information, we did not assess the methodological rigor of the Teen Courts
    evaluation as we did with the other 10 impact evaluations. On the basis of
    comments provided by the Assistance Attorney General, we have reclassified
    the Teen Courts evaluation as an impact evaluation of an OJJDP- funded
    evaluation.

    Page 10 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    evaluation to verify the accuracy of every coded item. We relied on
    documents OJJDP provided to us in April 2001 in assessing the evaluation
    methodologies and reporting on each evaluation?s status.

    For each of the remaining 24 evaluations, which included nonimpact
    evaluations of OJJDP- funded programs, as well as evaluations of juvenile
    justice programs that OJJDP did not fund, we asked OJJDP to provide general
    descriptive information, such as the type and purpose of the evaluation, the
    number of sites involved, and whether the evaluation included data on all
    participants. We did not assess the methodological rigor of these
    evaluations.

    We conducted our work at OJJDP Headquarters in Washington, D. C., from
    September 2000 to August 2001 in accordance with generally accepted
    government auditing standards.

    OJP requires virtually all OJJDP grantees 14 to submit semiannual progress
    reports, which OJJDP uses to help monitor grantees? project implementation
    and achievement of the goals they identified in their grant applications. To
    this end, OJP provides grantees standard, general guidance on the types of
    program information they are to report, such as narrative information on the
    status of each of their project goals and the quantitative results of their
    projects. In addition to this standard requirement, grantees for some of
    OJJDP?s programs are subject to additional reporting requirements that apply
    only to their respective programs. Our review of 16 major programs 15 showed
    that grantees in 8 of the programs were required to comply only with the
    standard requirement for information, and grantees in the other 8 programs
    were required to report additional specified data. The specific reporting
    requirements were established primarily to help evaluate the results of
    these programs. Table 1 identifies the 16 programs we reviewed and the
    reason for the standard or specific reporting requirements for each program.

    14 All discretionary and all formula/ block grant program grantees, with the
    exception of Formula Grants Program grantees, must submit these reports. 15
    We reviewed all 5 of OJJDP?s formula/ block grant programs and 11 of its
    discretionary grant programs. All Grantees Receive

    Standard Reporting Guidance and Some Are Required to Report Specific Data

    Page 11 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 1: Reasons for Reporting Requirements for 16 Selected Major OJJDP
    Programs Reason for reporting requirement Program Standard

    requirement Specific requirement

    State Challenge Activities Monitoring None Community Prevention Grants
    Program (Title V) Monitoring None Juvenile Accountability Incentive Block
    Grants Program Monitoring None Enforcing the Underage Drinking Laws Program
    (block grant component) a Monitoring None Enforcing the Underage Drinking
    Laws Program (discretionary grant component) a Monitoring None Tribal Youth
    Program Monitoring None Model Courts Program Monitoring None National
    Clearinghouse and Resource Center for Missing and Exploited Children
    Monitoring None Formula Grants Program b Monitoring Statutory compliance,

    OJJDP assessment Comprehensive Gang Initiative Monitoring Outside
    evaluation,

    grantee self- assessment Rural Gang Initiative Monitoring Outside
    evaluation,

    grantee self- assessment Juvenile Mentoring Program (JUMP) Monitoring
    Outside evaluation SafeFutures Monitoring Outside evaluation Drug- Free
    Communities Support Program Monitoring Outside evaluation Internet Crimes
    Against Children Task Force Program Monitoring OJJDP assessment Children?s
    Advocacy Centers Monitoring OJJDP assessment,

    grantee self- assessment Note: Five programs in this table are formula or
    block grant programs: State Challenge Activities, the Community Prevention
    Grants Program, the Juvenile Accountability Incentive Block Grants Program,
    Enforcing the Underage Drinking Laws Program (block grant component), and
    the Formula Grants Program. The remaining 11 programs are discretionary
    grant programs.

    a The Enforcing the Underage Drinking Laws Program has both a block grant
    program component and a discretionary grant program component, and OJJDP
    administers each from separate organizational units. b Rather than reporting
    semiannually through progress reports, grantees under this program report on
    their subgrantees? projects annually through Individual Project Reports. The
    standard requirements for reporting are similar for both types of reports.

    Source: GAO analysis.

    Our review of selected progress reports from the 16 programs showed that, in
    all but the Formula Grants Program, grantees reported information on the
    status of their activities and accomplishments in response to the standard
    requirements, although the details they reported varied as did the projects
    themselves. Grantees in the eight programs with specific reporting
    requirements reported a variety of descriptive information and performance
    data to OJJDP and/ or outside evaluators. (See app. IV for a

    Page 12 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    program description, summary of reporting requirements, and examples of what
    grantees reported in each of the 16 programs we reviewed.) 16

    All OJJDP grantees are required to report on their project activities and
    accomplishments to OJP twice a year using the Office of Justice Programs?
    Categorical Assistance Progress Report (progress report) form. 17 The form
    is unstructured and is to be completed in narrative and/ or chart form. 18
    The standard instructions to grantees for completing the form state that
    grantees should report information on the status of each of their projects?
    goals scheduled to be achieved during the reporting period and set forth in
    their grant application, including quantitative project results based on
    performance measures. Grantees are also instructed to report on actions
    planned to resolve any implementation problems and request any technical
    assistance they might need. OJJDP program managers are to use reported
    information to help monitor grantees? project implementation. OJJDP
    officials explained that because the progress report is intended as just one
    of their monitoring tools, this standard, general guidance meets their basic
    oversight needs. They further explained that guidance needs to be somewhat
    general given the variation that can occur among projects as grantees tailor
    them to meet local needs and circumstances. OJJDP encourages grantees to
    design projects that meet the unique needs of their own communities, and
    therefore grantees do not always report on the same performance measures.

    Although OJJDP program managers have additional ways of keeping abreast of
    grantees? projects, such as phone calls and on- site visits, OJJDP officials
    indicated they would prefer to require and obtain more specific,

    16 In addition, we aggregated performance data from grantees in OJJDP?s
    training and technical assistance programs and research programs because
    such data are quite significant in terms of OJJDP?s outputs, and would not
    be fully captured if we reported only on the 16 programs we selected for
    review. Although training and technical assistance grantees and research
    grantees were typically subject only to standard reporting requirements,
    they reported similar types of quantitative performance data. Thus, we were
    able to report comprehensive output data, rather than examples of output
    data, as we did for selected grantees in the 16 programs. See appendix V for
    a summary of performance data from training and technical assistance
    grantees and research grantees.

    17 As previously noted, the Formula Grants Program is the only exception. In
    lieu of the semiannual Categorical Assistance Progress Report, Formula
    Grants Program grantees are required to report annually on their
    subgrantees? activities and accomplishments using Individual Project
    Reports.

    18 A copy of the progress report form can be found in appendix III. Grantees
    Receiving

    Standard Guidance Reported a Variety of Information Used for OJJDP
    Monitoring

    Page 13 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    and even more frequent, information through the progress reports or other
    reporting mechanisms. However, according to these officials, they are
    reluctant to impose additional reporting requirements on grantees because of
    the Paperwork Reduction Act of 1995, which seeks to ensure that federal
    agencies balance their need to collect information with the reporting and
    paperwork burden they impose. Under the Act, federal agencies have an
    obligation to keep the paperwork burden they impose as low as possible, and
    agencies must receive prior approval from the Office of Management and
    Budget for information collection requests.

    We reviewed selected progress reports that grantees from each of the 16
    programs submitted to OJJDP and found that, in all but the Formula Grants
    Program, grantees reported on the status of their projects. Grantees
    reported input, output, or outcome data related to the process,
    implementation, and/ or accomplishments of their projects. They included
    information such as subgrant awards, specific meetings held, staff hired,
    implementation difficulties, number of project participants, and behavioral
    change in youths. However, the particular information grantees reported
    varied, as did their projects. This variation, coupled with the unstructured
    format of the progress report, makes it difficult to aggregate reported
    data.

    Fourteen of the programs we reviewed had multiple grantees and the
    information these grantees reported in response to the standard guidance
    varied, even within each program. For example, we found the following:

     Under the Tribal Youth Program- a program that recognizes differences
    among tribes and encourages diversity in their projects- grantees must
    implement projects in keeping with at least one of four broad purpose areas.
    19 One tribe reported that it had completed the renovation of a youth
    center; another reported that it had collected examples of other tribes?
    juvenile law enforcement codes 20 and started drafting model codes

    19 Funds are available for projects that support the following four broad
    purposes: (1) to reduce, control, and prevent crime by and against tribal
    youth; (2) to provide interventions for court- involved tribal youth; (3) to
    improve tribal juvenile justice systems; and (4) to provide prevention
    programs focusing on alcohol and drugs.

    20 To address the severe juvenile crime problem in Indian communities, many
    tribal governments have developed juvenile codes that typically include
    provisions relating to the disposition of cases involving Indian youth
    arrested or detained by the Bureau of Indian Affairs or tribal law
    enforcement personnel. Tribal juvenile codes are generally tailored to
    Indian youth by incorporating traditionally accepted cultural methods of
    dealing with juvenile delinquency issues.

    Page 14 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    adapted for each of its villages. A third tribe reported that the
    resignation of its community truancy officer had impacted its ability to
    reduce instances of misbehavior in school.

     Under another program- the Juvenile Accountability Incentive Block Grants
    Program- grantees (states) and their subgrantees (communities) have
    undertaken a variety of projects and, thus, report different information. In
    this program, states and their communities can choose from 12 different
    purpose areas 21 under the broad objective of promoting greater
    accountability in the juvenile justice system. Thus, one state reported that
    one community hired a juvenile court intake officer and included that
    officer?s caseload; the same state reported that another community was
    unable to start a project because a local agency declined to participate in
    its project. Another state reported on the number of youths enrolled in one
    community?s drug testing project and reported the number of drug screening
    tests performed.

     In the Drug- Free Communities Support Program, grantees design projects to
    meet the needs of their local communities; thus, the projects and the
    information grantees reported varied. For example, one grantee reported that
    it helped local students produce a 30- second anti- smoking commercial in
    collaboration with the local health department and further reported that
    only 9 of 50 invited members attended a strategic planning meeting it had
    held. Another grantee reported making presentations on drug abuse to 146
    young men at the local juvenile detention facility, and that its pre- and
    post- assessments continually showed that the young men gained knowledge in
    the harmful effects of alcohol and drugs.

    In the Formula Grants Program, not all grantees reported on the objectives
    and accomplishments of their subgrantees? projects, as required. OJJDP
    requires grantees in this program to complete an Individual Project Report

    21 Funding is available to support the following 12 purpose areas: (1)
    construction of juvenile detention or correctional facilities, including
    training of personnel; (2) accountability- based sanctions programs; (3)
    hiring of judges, probation officers, and defenders, and funding pretrial
    services; (4) hiring of prosecutors; (5) funding of prosecutor- led drug,
    gang, and violence programs; (6) providing technology, equipment, and
    training programs for prosecutors; (7) probation programs; (8) gun courts;
    (9) drug courts; (10) interagency information- sharing systems; (11)
    accountability- based programs for law enforcement referrals or those
    designed to protect students and school personnel from youth violence; and
    (12) controlled substance testing (including interventions) for juveniles in
    the juvenile justice system.

    Page 15 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    (IPR) 22 for each subgrantee. The instructions for completing IPRs are
    similar to the instructions grantees in other programs receive for
    completing semiannual progress reports. Our review of IPRs from selected
    states showed that, for one state, none of the IPRs contained any
    information on subgrantees? accomplishments, and some did not include
    information on subgrantees? program objectives. 23 For another state,
    neither OJJDP nor the state was able to provide us with copies of completed
    IPRs because OJJDP?s automated reporting system for states was inoperable.

    Two of the 16 programs we reviewed had only one grantee each. Although both
    received only standard reporting guidance, they reported more detailed,
    quantitative output and outcome data than grantees in the other programs
    that received only standard guidance. In the first program- the National
    Clearinghouse and Resource Center for Missing and Exploited Children- the
    grantee has voluntarily reported detailed information in a structured
    format. In the second program- the Model Courts Program- OJJDP has
    emphasized that the grantee should include quantitative performance data in
    its progress reports, but did not prescribe the specific indicators on which
    the grantee must report.

    OJJDJP has designated the National Center for Missing and Exploited Children
    (NCMEC) as the grantee for National Clearinghouse and Resource Center for
    Missing and Exploited Children, and NCMEC has developed its own standardized
    reporting format that covers 10 categories. This format collects numbers and
    other information on each category, such as missing children cases,
    exploited children cases, public affairs, and hotline calls. NCMEC reports
    to OJJDP quarterly, rather than semiannually, because this timeframe matches
    the reporting structure of its data management system. For the first quarter
    of 2001, NCMEC reported various output and outcome data that included
    receiving 24,983 calls through its hotline; assisting in the recovery of
    1,610 missing

    22 IPRs are part of annual Performance Reports, which Formula Grants Program
    grantees are required to complete. 23 Although assessing the quality of
    OJJDP?s monitoring efforts was outside the scope of our review, we inquired
    about the lack of information in this particular state?s IPRs. According to
    the responsible OJJDP official, OJJDP staff addressed this issue during a
    site visit made subsequent to the state?s annual submission, and also in a
    follow- up letter to the state.

    Page 16 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    children; 24 receiving 5,291 tips on its online child pornography tip line;
    and displaying pictures of 1,399 missing children, which resulted in
    locating 257 children.

    The sole grantee of the Model Courts Program- the National Council of
    Juvenile and Family Court Judges 25 -- also reports quantitative information
    in its semiannual progress reports. Although OJJDP has not specified the
    performance indicators on which the Council must report, it has emphasized
    to the Council the need for quantitative performance data in the semiannual
    progress reports. As a result, the Council includes specific quantitative
    output data in its progress reports. For example, it reported that during
    the last half of 2000 it distributed 17,818 technical assistance bulletins,
    conducted 96 training presentations, and made 31 site visits to model
    courts. In addition, the Council voluntarily publishes an annual report that
    provides more detailed information on the accomplishments of the individual
    model courts, such as a reduction in the number of children in court
    custody. OJJDP officials told us that they do not require the Council to
    provide this report, but they have instructed it to report detailed
    performance data on the activities of the model courts, when such data
    exist. They further explained that if the Council were to stop publishing an
    annual report, OJJDP would require it to include model court performance
    data in its progress reports.

    In eight of the major programs we reviewed, grantees were given additional,
    more explicit reporting instructions requiring them to report on the same
    specific performance measures as other grantees in the same program. In
    these programs, additional requirements were established to meet the
    evaluative needs of OJJDP or an outside evaluator. In one of the eight
    programs, requirements were also established to ensure grantee compliance
    with certain requirements of the Juvenile Justice Act as well as for program
    assessment. The specific requirements of each of these

    24 ?Missing children? includes nonfamily and family abductions; runaways and

    ?thrownaways?; and lost, injured, or otherwise missing children. The number
    of recovered children includes deceased children.

    25 The Council provides training and technical assistance to 23 ?model
    courts? nationwide. These designated model courts seek to improve how child
    abuse and neglect cases are processed through the court and child protection
    systems. Although their initiatives vary, all the model courts address
    certain core issues, such as how well court processes work in everyday
    practice and how well the child protection system meets the needs of
    children and their families. Grantees in Some

    Programs Are Required to Report Specific Performance Data, Often for
    Evaluative Purposes

    Page 17 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    eight programs varied, as they were tailored to each program. However,
    grantees in all these programs were still required to routinely report
    narrative information on the status of their activities and accomplishments
    through semi- annual progress reports.

    In five of these programs, OJJDP and/ or an evaluator have established
    specific reporting requirements primarily to support an outside program
    evaluation. 26 For example, as a condition of receiving a Juvenile Mentoring
    Program grant, OJJDP requires all grantees to participate fully in the
    evaluation by providing data to the evaluator. This evaluator requires
    grantees to report their data through quarterly progress reports that are
    similar to semiannual progress reports. The required data include
    information on youths participating in each project, participating mentors,
    and youth- mentor matches. For instance, grantees are required to report
    family structure information for participating youths. The evaluator
    aggregates such data from all grantees and has reported, for example, that
    56 percent of participating youths lived with their mother only, 20 percent
    lived with both parents, 4 percent lived with their father only, and 21
    percent were in other living arrangements. 27

    In two of the eight programs- the Internet Crimes Against Children Task
    Force Program and Children?s Advocacy Centers- specific reporting
    requirements were established so that OJJDP could assess program
    accomplishments. The governing board of the Internet Crimes Task Force, in
    agreement with OJJDP, identified monthly performance measures on which
    grantees must report, such as the number of arrests made, search warrants
    issued, subpoenas served, and cases opened by the task forces. Under the
    Children?s Advocacy Centers program, 28 OJJDP prescribed specific
    performance measures on which grantees must report, such as the number of
    practitioners trained, training conferences held, and publications
    distributed. In this program, specific reporting requirements were
    established not only for OJJDP to assess the program?s overall
    accomplishments, but also to help grantees assess their own projects.

    26 In two of these five programs, a secondary purpose for imposing these
    requirements was to allow grantees to assess their own programs. 27 Data
    were reported in a September 2000 OJJDP Juvenile Justice Bulletin entitled

    Juvenile Mentoring Program: A Progress Review.

    28 This program is, in part, a training and technical assistance program.

    Page 18 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    The eighth program- the Formula Grants Program- has requirements that are
    statutorily based and further spelled out by OJJDP in program regulations.
    Program reporting requirements were established to ensure grantees comply
    with the four core requirements of the Juvenile Justice Act and as a basis
    for assessing the effects of the program. These core requirements are (1)
    deinstitutionalization of status offenders, 29 (2) separation of juveniles
    from adult offenders, (3) removal of juveniles from adult jail and lockup,
    and (4) addressing efforts to reduce disproportionate minority confinement.
    OJJDP regulations list in detail the information on which states must
    report. 30 For instance, regarding the deinstitutionalization of status
    offenders, states must report the total number of accused and adjudicated
    status offenders and nonoffenders placed in facilities that are, for
    example, not near their home community. (See app. VI for a summary of
    states? compliance with the core requirements of the Juvenile Justice Act.)
    According to the compliance monitoring coordinator for the Formula Grants
    Program, grantees? reports on compliance with the core requirements also
    provide the basis for OJJDP to assess the effects of the program.

    We identified eight programs that serve juveniles directly and whose
    grantees reported such data for fiscal year 2000. However, OJJDP often does
    not require grantees to provide this information, in large part because not
    all of its programs are intended to provide direct services to juveniles.

    29 A status offender is a juvenile who has been charged with or adjudicated
    for conduct that would not be a crime if committed by an adult. 30 28 C. F.
    R. 31. 303( f)( 5). Some Data Exist on

    the Number of Juveniles Served Directly, Although Most Grantees Are Not
    Required to Report This Information

    Page 19 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    We identified eight programs in which grantees directly serve juveniles and
    in which all grantees report the number of juveniles served to either OJJDP
    or an outside evaluator. About 400 grantees in these eight programs directly
    served close to 142,000 juveniles in one year. For example, in fiscal year
    2000, the Juvenile Mentoring Program reported serving about 8,500 juveniles,
    and in calendar year 2000, the Court Appointed Special Advocate Program
    reported serving 70,348 youths. Table 2 shows the programs we identified as
    directly serving juveniles and reporting such data for fiscal year 2000.

    Table 2: OJJDP Programs GAO Identified in Which All Grantees Reported Data
    on the Number of Juveniles Directly Served, Fiscal Year 2000

    Program Description Juveniles directly served Grantees

    Arts and At- Risk Youth A program that combines the arts with job training
    and conflict resolution skills and provides summer jobs or paid internships
    for youths in the program.

    381 3 The Arts Program for Juvenile Offenders in Detention and Corrections A
    program that works with arts professionals to help youths

    develop art skills to use after release from detention. 1,026 6
    Comprehensive Gang Initiative A demonstration program that utilizes a
    variety of strategies,

    including intensive youth and family intervention, community mobilization,
    and organizational change, to reduce youth gang crime and violence.

    78 a 2 Court Appointed Special Advocate Program A program to help ensure
    that abused and neglected children

    receive timely and effective representation in dependency hearings. Program
    volunteers are appointed by judges to advocate in court for the best
    interests of abused or neglected children.

    70,348 b 207 Juvenile Mentoring Program (JUMP) A one- on- one mentoring
    program for at- risk youths. 8, 509 175 Life Skills Training A training and
    technical assistance program that instructs

    teachers on how to replicate the Life Skills Training program in a large
    number of diverse jurisdictions. Life Skills Training is a tested whole-
    school- immersion model for reducing drug use by middle- school students in
    selected communities.

    53,685 c 1 SafeFutures A program that works to reduce delinquency and youth

    violence through the development of a continuum of services. Grantees in
    this program use a single grant to draw on a variety of OJJDP categorical
    funds.

    5,789 d 6 Truancy Reduction Demonstration Program A program to reduce the
    number of truant children by

    providing services, such as truancy workshops and truancy case workers.

    2,085 7

    Total 141,901 407

    Note: The programs listed are those we identified in which all grantees
    reported juveniles- served data. We identified these programs by reviewing
    OJJDP literature and interviewing OJJDP division directors. Nevertheless,
    the list may not be comprehensive.

    In Some Programs, All Grantees Report the Number of Juveniles They Directly
    Serve

    Page 20 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    a The level of services provided under this program varied widely by
    juvenile- i. e., some juveniles received intensive intervention counseling,
    while others were only monitored by a case worker. b This number represents
    juveniles served in calendar year 2000.

    c This number is associated with the academic year 1999- 2000, which roughly
    corresponds with fiscal year 2000. Because this is a program to teach drug
    prevention skills to all middle- school students in selected communities,
    the grantee reports the number of ?students reached? rather than the number
    of ?juveniles served.? d This number represents a combination of data
    provided by SafeFutures grantees and by the

    program?s national evaluator. In addition, the level of services provided to
    these juveniles varied widely by individual, with some grantees including
    data on juveniles who received a full array of project services, and others
    reporting data on juveniles who were assessed but subsequently denied
    services or referred elsewhere. Furthermore, one grantee reported data for a
    slightly different period- September 1999 through August 2000.

    Source: GAO analysis of OJJDP, evaluator, and grantee data.

    We also identified a program in which all subgrantees directly serve
    juveniles, but not all subgrantees report such data. The national grantee 31
    for the Children?s Advocacy Centers program reported that its subgrantees
    served over 100,000 juveniles in calendar year 2000, but this number
    represents only those juveniles served by subgrantees accredited through a
    national membership council. 32

    For several reasons, OJJDP does not typically require grantees to report the
    number of juveniles their projects directly serve. First and foremost, many
    of OJJDP?s programs are not intended to serve juveniles directly. The
    Juvenile Justice Act established OJJDP for a variety of purposes, many of
    which involve indirect benefits to juveniles, rather than direct services.
    Statutorily- established purposes for OJJDP include the following:

     To provide technical assistance to and training programs for professionals
    who work with delinquents.

     To provide for the evaluations of federally- assisted juvenile justice and
    delinquency prevention programs.

     To establish a centralized research effort on problems of delinquency. 31
    The national grantee (one of five program grantees) awards funds to
    subgrantees, who in turn provide direct services to juveniles. The other
    four grantees, which are regional, provide training and technical assistance
    to local Children?s Advocacy Centers, as does the national grantee.

    32 Accredited subgrantees received 75 percent of funds available to
    subgrantees. Reasons Exist for Not

    Requiring Grantees to Report Number of Juveniles Served

    Page 21 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

     To assist state and local governments in improving the administration of
    justice and services for juveniles who enter the system.

    Some of OJJDP?s programs, in their entirety, provide indirect benefits,
    rather than direct services, to juveniles. OJJDP?s Model Courts Program, for
    example, benefits juveniles indirectly by providing training and technical
    assistance to court personnel to improve their handling of child abuse and
    neglect cases. The Internet Crimes Against Children Task Force Program also
    benefits juveniles indirectly by helping to identify and arrest pedophiles
    and child pornographers who use the Internet to prey on children.
    Furthermore, in commenting on a draft of this report, the Assistant Attorney
    General pointed out that although OJJDP?s research projects do not typically
    provide services directly, their results can potentially help thousands of
    juveniles.

    OJJDP officials provided the following additional reasons for not requiring
    all grantees to report the number of juveniles their projects directly
    serve:

     A common interpretation of ?juveniles served? does not exist across, or
    even within, programs. For example, grantees in one program might consider
    the number of juveniles served as those assessed for services but referred
    elsewhere, while grantees in a different program might consider only
    juveniles who received at least 10 sessions of therapy. Even within the same
    program grantees may not share a common definition of

    ?juveniles served.? One program grantee might report on the number of
    juveniles who received intensive one- on- one drug prevention services over
    an extended period of time, while another grantee in that same program might
    report on the number who attended a one- time presentation on drug
    prevention. Without a common interpretation of ?juveniles served,? the data
    grantees report would be inconsistent and would have little value.

     For some programs, directly serving juveniles may be only one of a number
    of intended program purposes and thus, OJJDP does not typically require all
    grantees within these programs to report such data. For example, in the
    Formula Grants Program, states and their subgrantees can choose from among
    14 different program areas related to preventing and controlling delinquency
    and improving juvenile justice systems. Under the program area of ?planning
    and administration,? for instance, states can fund planning projects that
    benefit juveniles indirectly, such as developing a comprehensive state plan
    to identify juvenile service needs and programs that address those needs
    over the long term. However, under the area of ?illegal drugs and alcohol,?
    a local subgrantee can serve juveniles directly by establishing a drug and
    alcohol abuse prevention project.

    Page 22 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

     Juveniles- served data could be used inappropriately to measure the
    effectiveness of a program whose primary purpose may not be to provide
    direct services to juveniles. For example, the primary purpose of State
    Challenge Activities is to stimulate system- wide change, although many of
    its 10 activity areas also promote projects intended to directly serve
    juveniles. However, grantees in this program are expected to implement
    direct service projects within the broader context of promoting systemwide
    change. For instance, one State Challenge Activities grantee used funds it
    received under the ?deinstitutionalization of status offenders? activity
    area to establish two community projects that provide housing for runaway
    juveniles, many of whom are girls. The grantee intends to use its
    experiences with these two new projects to initiate system- wide change by
    developing a comprehensive model program expressly geared to serving runaway
    girls. By focusing on the number of girls served by this program, one might
    fail to see that its primary purpose was to develop a comprehensive model
    program for serving runaway girls.

    OJJDP has funded 35 evaluations since 1995, including 11 evaluations
    intended to measure the impact of OJJDP- funded programs. We reviewed the
    methodological rigor of 10 of the 11 evaluations. 33 Half of these 10
    evaluations are in formative stages, while the other five are well into
    implementation. 34 None had been completed at the time of our review. Our
    in- depth review of these 10 evaluations shows that although several are
    well- designed and use, or plan to use, sophisticated data analysis methods,
    others raise concerns as to whether the evaluations will produce definitive
    results. We recognize that impact evaluations, such as the types that OJJDP
    are funding, can encounter difficult design and implementation issues. For
    some of the evaluations we reviewed, program flexibility has added to the
    complexity of designing evaluations. A lack of comparison groups to aid in
    isolating the impacts of some programs, and data collection problems could
    compromise some evaluation results.

    According to OJJDP officials, the OJJDP weighs a number of factors when
    deciding which programs to evaluate and what kind of evaluations to fund.

    33 As discussed in the Scope and Methodology section of this report, in
    commenting on a draft of this report, the Assistant Attorney General pointed
    out that the evaluation of the Teen Courts program is also an impact
    evaluation of an OJJDP- funded program. This section addresses only the 10
    impact evaluations OJJDP had initially identified.

    34 We based our analysis of these 10 ongoing evaluations on documentation
    OJJDP provided as of April 2001. Methodological

    Concerns Could Adversely Affect Evaluation Studies

    Page 23 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Given its budget, it considers how much of its discretionary funds to spend
    in support of evaluation activities. In deciding which of its programs to
    evaluate, OJJDP gives priority consideration to programs that have been
    mandated by the Congress. Other criteria OJJDP uses to determine whether a
    program should be evaluated include the program?s level of funding and its
    uniqueness, as well as the feasibility and cost of an evaluation and its
    potential benefits to the field. Similar criteria are also involved with
    decisions to evaluate non- OJJDP funded programs, as well as congressional
    interest and other federal agencies? willingness to co- fund an evaluation.

    The 10 impact evaluations of OJJDP- funded programs that we assessed vary in
    size and scope. The cost to conduct these evaluations ranges from $300,000
    to well over $5 million; however, some of these grants involve both impact
    and process evaluations and the cost of the impact portion alone cannot be
    separated from the total. All 10 evaluations are multi- year, multi- site
    projects. The number of evaluation sites ranges from 2 in the Rural Gang
    Initiative to 175 in the evaluation of the Juvenile Mentoring Program. As of
    April 2001, three evaluations had produced interim findings of some program
    impacts. 35 (See app. II for information on OJJDP?s process for
    disseminating products with interim findings as well as other products.)

    Program evaluation is an inherently difficult task because the objective is
    to isolate the impact of a particular program from all other factors that
    could have caused a change consistent with the intent of the program, or
    mitigated against that change. Given that programs, such as those funded by
    OJJDP, operate in an ever- changing environment and involve juveniles and
    adults who themselves constantly change, producing definitive evaluation
    results can be arduous. For example, the impact of a hypothetical program
    intended to improve students? grades could be confounded by the effects of
    an outside- of- school mentoring program, the transfer of high- performing
    students to a magnet program, changes in school faculty, a new scholarship
    program, a severe flu season that results in widespread student absences
    from school, and a myriad of other factors.

    35 Two other evaluations reported preliminary results of some of the data
    collected as part of the impact evaluations; however, these did not comprise
    impact results. Evaluations of OJJDP

    Programs Are Difficult to Successfully Design and Implement

    Page 24 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Our in- depth review of the 10 impact evaluations of OJJDP programs showed
    that a number of these evaluations are particularly complex because local
    grantees design their own projects to fit their communities? needs. (See
    app. VII for a description of the impact evaluations OJJDP has funded of its
    own programs since 1995). Although this customization may make sense from a
    program perspective, it makes it more difficult to evaluate the program.
    Instead of assessing a single, homogeneous program with multiple grantees,
    the evaluation must assess the effects of multiple configurations of a
    program. Although all of the grantees? projects under each program being
    evaluated are intended to achieve the same or similar goals, an aggregate
    analysis could mask differences in individual projects? effectiveness and,
    thus, not result in information about which configurations of projects work
    and which do not.

    OJJDP?s evaluation of the Enforcing the Underage Drinking Laws Program
    (discretionary grant component) exemplifies this situation. In implementing
    their projects, states and local communities have substantial latitude to
    employ media campaigns, merchant education, compliance checks, youth
    leadership training, or a variety of other activities to deter underage
    drinking. Similarly, under the Positive Action Through Holistic Education
    program, local educators develop their own ways to prevent student violence
    and behavior problems based on their assessments of the causes of these
    problems in their schools. Because of the limited number of sites (two
    school districts) being evaluated and the likely differences in how each
    school has developed its own project, the resulting evaluation may not
    provide information that could be generalized to a broader implementation of
    the program.

    A standard way for evaluators to isolate the impacts of a program from other
    potential factors that could have influenced change is to use a comparison
    group as a benchmark. In the hypothetical example cited above concerning a
    program to improve students? grades, a second set of students who are not in
    the program but are matched in academic performance and exposed to all of
    the same factors (except the program) could provide a baseline from which to
    assess the impact of the program. The grades of students in the two groups
    before and after the program would provide the data from which to measure
    program impacts. Without the benefit of the comparison group as a baseline,
    it is difficult or impossible to isolate changes resulting from the program
    from changes due to other factors.

    The designs of two of the five evaluations that are well into implementation
    lack an appropriate comparison group. The evaluation of Project Variation
    Within a

    Program Can Complicate Evaluation Design

    Lack of Appropriate Comparison Groups Could Compromise Some Results

    Page 25 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    the Juvenile Mentoring Program- a one- on- one mentoring program for youths-
    compares youths entering the program to those completing it. However, a
    variety of other factors, including the fact that youths in the program are
    likely to mature and, thus, improve somewhat spontaneously, cannot be ruled
    out as a rival cause of change from the beginning to the end of the program.
    Although the evaluators are employing multiple and innovative strategies to
    determine the effectiveness of the program in achieving its objective, the
    lack of a comparison group of nonparticipant youths is an obstacle to
    identifying definitive outcomes.

    An evaluation of the Partnerships to Reduce Juvenile Gun Violence Program
    includes a comparison of before and after crime statistics in project
    communities with crime statistics for the same time frames for the cities in
    which the projects operate. However, citywide crime statistics would no
    doubt include data from communities that are similar to the project
    community as well as from those that are not. Thus, the differences between
    citywide and project community baselines make it difficult to attribute
    potential findings to the program.

    Of the five programs for which evaluations are still being developed, two
    (the Safe Start Initiative and the Rural Gang Initiative) did not seem to
    have plans for comparison groups at the time of our review. Another (Parents
    Anonymous) anticipates using a comparison group, but as yet had not
    developed specific plans for one.

    Regardless of the quality of a program evaluation design, data collection
    problems can compromise the validity of findings. Data collection problems
    may affect the validity of the findings for three of the five evaluations
    that are currently completing or have completed data collection. The
    Juvenile Mentoring Program evaluation has experienced problems obtaining
    behavioral measures and school performance data with which to gauge program-
    driven change. The Comprehensive Gang Initiative evaluation has also
    experienced data collection problems such as the lack of fully adequate
    comparison youth data at all or most sites, missing police histories, and
    missing self- reported data.

    The Intensive Aftercare evaluation has experienced survey response rate
    shortfalls, in some cases obtaining response rates of less than 30 percent,
    which may affect the validity of the findings. In commenting on a draft of
    this report, the Assistant Attorney General said that the poor response
    rates for some elements at different sites were particularly disappointing
    because this evaluation had a strong random assignment design; however, the
    strategies for obtaining adequate data turned out to be insufficient. She
    Data Collection Problems

    Could Compromise Some Results

    Page 26 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    added that the program staff who were required to collect data did not give
    data collection adequate priority in comparison to their other duties. This
    was particularly true of data regarding the comparison groups.

    In addition to funding impact evaluations of OJJDP programs, OJJDP has
    funded 24 other evaluations since 1995- 11 nonimpact evaluations of OJJDP
    programs, 9 impact evaluations of programs that were not supported by OJJDP
    funds, and 4 nonimpact evaluations of programs that were not funded by
    OJJDP. The nonimpact evaluations are not intended to determine the outcomes
    of the various programs, but rather how well the programs have been
    implemented. For example, OJJDP has funded a process evaluation of its
    SafeFutures program to learn more about the process of community
    mobilization and collaboration in building a comprehensive program of
    prevention and intervention for at- risk youths and juvenile offenders.

    OJJDP has also funded evaluations of programs that are funded by entities
    other than OJJDP. For example, although OJJDP does not fund the Act Now
    Truancy Program, it has funded a nonimpact evaluation of this program. The
    Act Now Truancy Program grew out of a unique Arizona law that allowed
    prosecutors to issue citations to parents whose children were chronically
    truant. Because there was a great deal of interest in this approach and
    OJJDP believed it provided a unique opportunity to learn about the impact of
    an unusual approach, it funded an evaluation of the program. Appendix VIII
    contains brief descriptions of these 24 other evaluations.

    Although there is great interest in assessing results of programs, it is
    extremely difficult to design and execute evaluations that will provide
    definitive information. Our in- depth review of 10 OJJDP- funded evaluations
    of OJJDP's own programs undertaken since 1995 has shown that, in some cases,
    the flexibility that can be beneficial to grantees in tailoring programs to
    meet their communities' needs has added to the complexities of designing
    impact evaluations that will result in valid findings. Furthermore, the lack
    of an appropriate comparison group or sites and/ or problems in data
    collection may compromise the reliability and validity of some of these
    evaluations. Because half of these 10 evaluations are in relatively early
    stages, any potential problems with comparison group issues or data
    collection shortfalls could still be resolved over the course of the
    evaluation. We recognize that not all evaluation issues that can compromise
    results are resolvable, including OJJDP Has Funded

    Other Types of Evaluations

    Conclusions

    Page 27 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    many involving comparison groups and data collection. However, to the extent
    that appropriate comparison groups can be established and tracked and data
    collection issues can be overcome, the validity of the evaluation results
    can be enhanced.

    Our review of the recent OJJDP program evaluations has shown that, of the
    five that are in or near their final stages, some problems with valid
    comparison groups and/ or data collection could compromise the usefulness of
    some of their results. Five other program evaluations are in a formative
    stage where comparison group issues and data collection strategies are not
    yet finalized. Accordingly, we recommend that the Attorney General direct
    the Administrator of OJJDP to assess the five impact evaluations in the
    formative stages to address potential comparison group and data collection
    problems and, on the basis of that assessment, initiate any needed
    interventions to help ensure that the evaluations produce definitive
    results.

    We provided a copy of this report to the Attorney General for review and
    comment. In an October 15, 2001 letter, the Assistant Attorney General
    commented on a draft of this report. Her comments are summarized below and
    are presented in their entirety in appendix IX. Her detailed comments have
    been addressed in the report as appropriate.

    The Assistant Attorney General said that the draft report provides useful
    information that highlights areas warranting attention. She added that the
    draft report would be an important tool that OJP will use to improve the
    quality of its evaluations and to design programs that will achieve greater
    impact. Furthermore, OJP will assess the five impact evaluations that are
    currently in their formative stages to address potential comparison group
    and data collection problems. On the basis of that assessment, OJJDP will
    initiate any needed interventions to help ensure that evaluations produce
    definitive results.

    The Assistant Attorney General said that OJP agrees that it should always
    strive for more rigorous and scientifically sound evaluation designs and
    that the inclusion of comparison groups would certainly strengthen the
    interpretation of evaluation results. However, she disagreed with our
    reliance on the use of comparison groups as the only valid evaluation design
    for two primary reasons. First, OJJDP seeks to conduct juvenile justice
    evaluations in a real- world setting, where laboratory- like comparison
    groups may not be possible. Second, sufficient funding is not Recommendation
    for

    Executive Action Agency Comments and Our Evaluation

    Page 28 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    available for including comparison groups in every evaluation. The Assistant
    Attorney General also said that given the choice between conducting far
    fewer evaluations, all with comparison groups, and conducting a greater
    number of evaluations under less- than- ideal conditions, OJJDP?s Research
    and Program Development Division works hard to tread a middle ground that
    satisfies needs for both quality and quantity. She further pointed out that
    a growing number of policy makers and evaluators firmly believe that
    community- based initiatives do not lend themselves to the kind of
    traditional evaluations that this draft report proposes. Accordingly, some
    researchers have strongly urged that new approaches to evaluation be
    developed.

    In addition, the Assistant Attorney General said that our report suggests
    that more evaluations using experimental or quasi- experimental evaluation
    designs should be funded. She added that many communities reject
    participation in programs that are evaluated in this way (i. e., with
    control or comparison groups) because they feel that it requires them to
    purposely exclude youths from receiving services.

    In her comments, the Assistant Attorney General seemed to be using the terms
    ?comparison group? and ?control group? interchangeably. However, control
    groups are commonly associated with experiments involving random assignment.
    We do not intend our statements regarding the need for comparison groups in
    impact evaluations to imply that random assignment is necessary for studies
    to be valid. Furthermore, we recognize that groups can be compared after
    controlling for differences by methods other than random assignment,
    including statistical methods and various methods of matching. For impact
    evaluations, comparisons should be made, and should involve individuals who
    were not subject to the program or treatment being evaluated. However, not
    all the evaluations we assessed made such comparisons.

    We also recognize that not all evaluation issues that can compromise results
    are resolvable, even with the use of comparison groups. We also recognize
    that designing evaluations with comparison groups can be expensive and
    funding limitations could preclude their use in all evaluations. In
    addition, obtaining participants can be troublesome, as the Assistant
    Attorney General pointed out. However, the validity of evaluation results
    can be enhanced through establishing and tracking comparison groups. If
    other ways exist to effectively isolate the impacts of a program, comparison
    groups may not be needed. However, we saw no evidence of other methods being
    used in the 10 impact evaluations we assessed. While studies that do not
    have appropriate comparison groups

    Page 29 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    can provide useful information, they should not be considered impact
    evaluations. Furthermore, we recognize the fact that communities may not
    favor withholding treatments or programs from individuals in control or
    comparison groups, however, this problem is commonly handled by phasing in
    the treatment or program and offering it to comparison group members
    following the evaluation period.

    As we agreed with your office, unless you publicly announce the contents of
    this report earlier, we plan no further distribution of it until 30 days
    from the date of this letter. At that time, we will send copies to the
    Senate Judiciary Committee, the Senate Subcommittee on Youth Violence, the
    House Committee on Education and the Workforce, the House Subcommittee on
    Early Childhood, Youth and Families, the Attorney General, and the Director
    of the Office of Management and Budget.

    If you or your staff have any questions about this report, please contact
    James M. Blume or me at (202) 512- 8777. Key contributors to this report are
    acknowledged in appendix X.

    Sincerely yours, Laurie E. Ekstrand Director, Justice Issues

    Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000

    Page 30 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    This appendix provides information on the awards the Office of Juvenile
    Justice and Delinquency Prevention (OJJDP) made each year from fiscal years
    1996 through 2000. It contains data on OJJDP funds awarded to formula/ block
    grant programs versus discretionary grant programs (see fig. 2), OJJDP funds
    awarded by more specific program areas (see table 3), types of OJJDP award
    recipients (see table 4), and OJJDP formula/ block grant awards by state
    (see table 5).

    We relied on the Office of Justice Program?s (OJP) awards database to
    analyze data on all OJJDP- administered awards made during this 5- year
    period. We analyzed awards by the year the award was made- not the year in
    which the funds were appropriated. We worked with OJJDP officials to
    identify awards by major program or program area, as the database did not
    provide sufficiently detailed information. OJP officials advised us that
    they perform daily quality control checks on all data entered into their
    database, however, we did not verify the accuracy of the database.

    Figure 2: OJJDP Formula/ Block Grants Versus Discretionary Grant Awards,
    Fiscal Years 1996 Through 2000

    Note: Dollar amounts are rounded to the nearest million. Source: GAO
    analysis of OJP database.

    Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000

    Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000

    Page 31 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 3: OJJDP Funds Awarded by Program Area, Fiscal Years 1996 Through 2000
    Program area FY 1996 FY 1997 FY 1998 FY 1999 FY 2000 Formula/ block grant
    programs

    Formula Grants Program $68,795,296 $77,469,250 $92,069,600 $74,468,599
    $69,699,750 State Challenge Activities 9,352,900 10,363,264 9, 726,800
    9,011,429 8,227,500 Community Prevention Grants Program (Title V) 19,762,000
    18,368,000 18,707,000 39,921,000 36,158,000 Enforcing the Underage Drinking
    Laws Program (block grant component)

    a a 18,360,000 17,640,000 19,080,000 Juvenile Accountability Incentive Block
    Grants Program

    a a 232,250,000 232,250,000 221,094,775

    Discretionary grant programs National Institute for Juvenile Justice and
    Delinquency Prevention b

    Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders
    900,730 900,055 1, 835,597 1,842,240 2,640,000 Causes and Correlates of
    Delinquency Research 299,999 900,000 599,999 600,000 750,000 Statistics and
    Systems Development Program 349,978 249,996 449,998 549,923 649,996 Other
    research, statistics, evaluation, information dissemination, and training
    and technical assistance grants 12,382,088 15,689,329 21,566,738 40,816,294
    42,783,923

    Demonstration and replication programs

    Comprehensive Gang Initiative 2,349,597 2,848,881 3,224,480 2,249,742
    1,149,498 Rural Gang Initiative a a 215,000 1, 112,754 1,474,473 SafeFutures
    2, 159,950 5,540,000 4,715,979 9,462,844 10,829,102 Community Assessment
    Centers a 1,074,552 1,699,992 125,000 875,000 Juvenile Mentoring Program
    (JUMP) a 9,688,488 3,363,281 21,059,220 5, 760,000 Partnerships to Reduce
    Juvenile Gun Violence Program

    a 551,060 1, 050,230 1,299,975 799,996 Enforcing the Underage Drinking Laws
    Program (discretionary grant component)

    a a 6,608,019 2,344,986 6,272,003 Drug- Free Communities Support Program a a
    8,903,433 19,039,326 29,677,581 Truancy Reduction Demonstration Program a a
    a 693,849 711,328 Tribal Youth Program a a aa 8,252,019 Other demonstration
    and replication grants 7, 070,330 8,846,421 17,178,085 26,309,408 24,058,149

    Missing and Exploited Children Program

    National Clearinghouse and Resource Center for Missing and Exploited
    Children 3,294,648 4,576,640 7,585,000 8,925,000 19,490,263 Missing and
    Exploited Children Training & Technical Assistance program

    a 3,448,575 1,749,803 1,848,947 1,949,741 Internet Crimes Against Children
    Task Force Program

    a a 2,351,829 199,997 8, 486,056 Other missing and exploited children grants
    1, 189,639 1,488,935 1,759,887 978,356 2, 074,318

    Child Abuse and Neglect Program

    Court Appointed Special Advocate Program 6, 040,000 5,904,400 7,062,100
    9,034,200 9,796,000 Children?s Advocacy Centers 3, 199,856 3,099,914
    5,691,000 4,733,000 5,824,302

    Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000

    Page 32 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program area FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

    Model Courts Program 742,500 984,100 1, 964,900 2,108,097 2,109,200 Safe
    Start Initiative a a aa 8,038,573 Other child abuse and neglect grants
    1,725,000 3,199,878 4,838,152 6,508,938 6,499,976

    Total $139,614,511 $175,191, 738 $475,526, 902 $535,133, 124 $555,211, 522

    Note: Award amounts are not adjusted for de- obligated funds, except in
    cases in which an entire award was de- obligated and later re- awarded, as
    occurred in rare instances of formula/ bock grant awards. a Either the
    program did not receive an appropriation or OJJDP did not award funds.

    b Awards listed under this subheading comprise a variety of research,
    statistics, evaluation, information dissemination, and training and
    technical grants. However, the funds awarded under this subheading do not
    represent a comprehensive accounting of all such activities funded by OJJDP.
    In some cases these activities are accounted for under other subheadings.
    For example, the Missing and Exploited Children Training and Technical
    Assistance award, which is primarily a training and technical assistance
    grant, is listed under the Missing and Exploited Children?s Program. Also,
    evaluations for many of the demonstration and replication programs are
    accounted for under their respective programs, rather than under the
    National Institute for Juvenile Justice and Delinquency Prevention. In
    commenting on a draft of this report, the Assistant Attorney General pointed
    out that combining training and technical assistance funding under the
    National Institute of Justice and Delinquency Prevention does not highlight
    the importance of training and technical assistance. Although we recognize
    the importance of training and technical assistance, OJP?s awards database
    did not readily lend itself to identifying all training and technical
    assistance- either awards or portions of awards- because they are part of
    many programs across OJJDP divisions.

    Source: GAO analysis of OJP database.

    Table 4: OJJDP Award Recipients, Fiscal Years 1996 Through 2000 FY 1996 FY
    1997 FY 1998 FY 1999 FY 2000 Recipients Amount

    awarded % a Amount awarded % a Amount

    awarded % Amount awarded % a Amount

    Awarded % a

    State governments $98,854,634 71 $109,350,562 62 $377,103,675 79
    $377,819,401 71 $369,196,062 67 Local governments b 2,584,003 2 7, 230,243 4
    13,534,664 3 18,766,350 4 27,078,615 5 Schools c 4,936,373 4 14,425,127 8
    18,292,717 4 20,926,891 4 34,975,373 6 Nonprofit organizations 32,289,708 23
    41,026,686 23 60,476,935 13 106,044,793 20 109,431,488 20 Other d 949,793 1
    3,159,120 2 6, 118,911 1 11,575,689 2 14,529,984 3

    Total $139,614,511 100 $175,191,738 100 $475,526,902 100 $535,133,124 100
    $555,211,522 100

    a Percentages may not total 100 due to rounding. b Local governments include
    county governments, cities, townships, and special district governments. c
    Schools include independent school districts and state and private
    institutions of higher education. d Other includes for- profit
    organizations, individuals, Indian tribes, and other miscellaneous
    categories.

    Source: GAO analysis of OJP database.

    Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000

    Page 33 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 5: OJJDP Formula/ Block Grant Awards, by State, Fiscal Years 1996
    Through 2000 Funds awarded State FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

    Alabama $1,424,000 $1,648,900 $5,896,400 $5,855,300 $5,638,100 Alaska
    787,500 832,500 2, 822,300 2,839,800 2,594,200 Arizona 1, 502,000 1,821,000
    6,197,500 6,714,900 6,327,800 Arkansas 869,000 1, 014,500 4,222,700
    4,432,700 4,222,100 California 11,440,000 13,417,100 37,575,000 37,547,300
    36,119,800 Colorado 1,279, 000 1,497,000 5,579,400 5,649,200 5,474,219
    Connecticut 1,017,000 1,197,000 4,744,200 4,824,300 4,612,400 Delaware
    787,500 829,500 2, 133,100 3,469,600 2,726,200 District of Columbia 788,546
    187,500 1, 785,400 1,778,000 1,699,100 Florida 4, 302,000 5,144,000
    15,440,600 15,631,200 15,089,600 Georgia 2, 467,000 2,907,000 9,432,800
    9,534,600 9,188,600 Hawaii 787,500 860,500 2, 984,800 3,402,400 3,096,800
    Idaho 787,500 869,500 3, 280,000 3,388,200 3,264,700 Illinois 3, 605,000
    1,626,064 17,824,400 14,320,800 12,886,850 Indiana 1, 942,000 2,269,000
    7,615,300 7,608,129 7,304,900 Iowa 961,000 1, 106,000 4,446,700 4,641,700
    4,438,300 Kansas 909,000 1, 057,000 4,316,400 4,517,200 4,360,300 Kentucky
    278,000 0 4,117,800 5,117,100 6,659,600 Louisiana 1, 553,900 1,816,100
    6,518,200 6,317,100 6,010,400 Maine 787,500 859,500 2, 965,650 3,027,850
    3,076,500 Maryland 1,665,000 1,941,000 6,751,400 6,678,100 6,418,700
    Massachusetts 1,537,500 893,000 7, 789,450 7,375,900 7,035,600 Michigan
    5,572,000 3,808,000 11,800,200 11,613,500 11,207,319 Minnesota 1, 636,000
    1,899,000 6,592,150 6,604,300 6,312,800 Mississippi 997,000 857,000 4,
    902,400 4,736,500 4,327,600 Missouri 1, 797,000 2,088,000 7,169,800
    7,188,300 6,835,700 Montana 787,500 843,500 2, 957,900 2,619,400 3,223,100
    Nebraska 664,500 910,500 3, 558,900 3,700,700 3,563,000 Nevada 795,500
    888,500 3, 481,600 3,695,300 3,630,700 New Hampshire 637,500 1, 006,500
    3,133,550 3,192,800 3,089,400 New Jersey 2,546,000 2,996,000 7,096,000
    7,676,900 9,528,200 New Mexico 830,500 939,500 3, 739,300 3,884,500
    3,748,900 New York 5,815,000 6,798,000 19,858,900 19,779,400 18,544,600
    North Carolina 2, 260,000 2,693,000 8,924,300 9,016,600 8,746,900 North
    Dakota 787,500 827,500 2, 779,400 2,766,400 2,682,900 Ohio 3,763,000
    4,363,000 13,101,700 13,060,100 9, 405,700 Oklahoma 1, 160,000 1,340,000
    5,102,900 5,170,600 4,946,500 Oregon 1,027,000 1,217,000 4,809,400 4,934,700
    3,769,500 Pennsylvania 3, 821,000 4,439,000 13,292,600 13,163,000 12,471,600
    Rhode Island 787,500 843,500 2, 965,000 2,992,300 2,905,700

    Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000

    Page 34 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Funds awarded State FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

    South Carolina 1,240,000 1,427,000 5,319,300 5,166,300 4,938,550 South
    Dakota 787,500 100,000 2, 013,500 1,993,800 1,921,300 Tennessee 1,710,000
    1,999,000 6,897,100 6,905,800 5,332,000 Texas 6, 910,000 8,165,000
    23,614,200 24,006,200 22,961,500 Utah 885,000 1, 030,000 4,281,400 4,518,500
    4,372,000 Vermont 787,500 822,500 2, 717,300 2,711,200 2,626,619 Virginia
    2,113,000 2,460,000 8,155,800 8,205,799 7,794,800 Washington 1,856,000
    2,164,000 7,362,500 7,433,800 6,784,350 West Virginia 810,500 899,500 3,
    497,100 3,590,800 3,432,100 Wisconsin 1, 755,000 2,045,000 6,538,150
    7,368,750 6,621,600 Wyoming 0 0 1, 933,950 1,836,800 1,759,100 American
    Samoa 148,000 133,000 617,391 601,962 472,248 Guam 115,000 192,000 842,350
    841,700 807,800 Northern Mariana Islands 148,000 151,000 381,959 381,738
    370,552 Puerto Rico 1,367,750 1,917,000 5,856,900 5,854,100 5,548,800
    Republic of Pulau 15,000 21,350 4, 300 0 0 Virgin Islands 100,000 123,000 1,
    374,700 1,407,100 1,331,818

    Total $97,910,196 $106,200, 514 $371,113,400 $373,291, 028 $354,260, 025

    Source: GAO analysis of OJP database.

    Appendix II: OJJDP?s Process for Disseminating Published Products, Including
    Interim Results of Impact Evaluations

    Page 35 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    The Office of Juvenile Justice and Delinquency Prevention (OJJDP) has a
    process for disseminating published interim results of impact evaluations as
    well as other publications produced by OJJDP and its grantees. OJJDP
    publications are available through the Juvenile Justice Clearinghouse. 1
    According to an OJJDP official, OJJDP develops a specific strategy for each
    publication that includes the number of copies to be printed, the methods
    for announcing availability, and the target audience that will automatically
    receive copies. OJJDP promotes products through the

    National Criminal Justice Reference Service (NCJRS) Catalog, OJJDP?s

    Juvenile Justice journal, the NCJRS and OJJDP Web sites, e- mail lists, the
    Office of Justice Programs press announcements, conference displays,
    criminal/ juvenile justice newsletters and journals, and flier mailings.
    Almost all of OJJDP?s publications are made available to the public through
    OJJDP?s Web sites, which is administered by the Juvenile Justice
    Clearinghouse. Many publications, depending on their length, are also
    available through the Clearinghouse?s fax- on- demand service. Individuals
    can also order copies of publications online or by calling the
    Clearinghouse?s toll- free number. In addition, the Clearinghouse
    automatically sends publications to targeted constituents (e. g., juvenile
    justice policymakers, practitioners, researchers, and community- based
    organizations) and to individuals who have registered to receive
    publications based on their specific areas of interest.

    As of May 2001, OJJDP had used this dissemination process to share interim
    results from 5 of the 10 ongoing impact evaluations 2 of OJJDP programs that
    we assessed. 3 In total, OJJDP had distributed over 400,000 copies of 9
    products that contained interim results from the 5 evaluations. Table 6
    provides additional information on the distribution of these publications.

    1 OJJDP established the Juvenile Justice Clearinghouse to provide
    individuals and organizations easy access to information and resources on
    juvenile justice topics. The Clearinghouse is a component of NCJRS, a
    federally- sponsored information clearinghouse that offers a range of
    services including online newsletters, access to publications by partner
    agencies, and a calendar of upcoming conferences. Its federal sponsors
    include the Office of Justice Programs (including OJJDP) and the Office of
    National Drug Control Policy. NCJRS maintains a Web site where publications
    by OJJDP and other federal sponsors can be accessed.

    2 Although these interim results were based on data collected from impact
    evaluations, they did not necessarily comprise impact results. 3 These 10
    impact evaluations were ongoing as of April 2001. Appendix II: OJJDP?s
    Process for

    Disseminating Published Products, Including Interim Results of Impact
    Evaluations

    Appendix II: OJJDP?s Process for Disseminating Published Products, Including
    Interim Results of Impact Evaluations

    Page 36 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 6: Dissemination of Published Products With Interim Results of Impact
    Evaluations, as of May 2001 Program Product a Automatic

    mailing b Other distribution c Other

    availability

    Juvenile Mentoring Program (JUMP) Juvenile Mentoring Program: 1998 Report to
    the

    Congress (Report, December 1998). 41,000 7, 100 Online Juvenile Mentoring
    Program: A Progress Review (Bulletin, September 2000). 44,000 7, 000 Online

    Fax- on- demand FY 2000 OJJDP Discretionary Program Announcement: Juvenile
    Mentoring Program (Solicitation, March 2000).

    63,700 9, 000 Online Fax- on- demand

    Comprehensive Gang Initiative Gang- Free Schools and Communities Initiative:
    FY

    2000 OJJDP Discretionary Program Announcement (Solicitation, July 2000).

    48,700 6, 700 Online Fax- on- demand

    Youth Gang Program and Strategies (Summary, August 2000). 45,000 2, 500
    Online Intensive Aftercare Implementation of the Intensive Community- Based

    Aftercare Program (Bulletin, July 2000). 32,100 2, 500 Online Fax- on-
    demand

    Reintegrating Juvenile Offenders Into the Community: OJJDP?s Intensive
    Community- Based Aftercare Demonstration Program (National Institute of
    Justice Research Preview, December 1998).

    Not available 664 Online

    Fax- on- demand Partnerships to Reduce Juvenile Gun Violence Program

    Fighting Juvenile Gun Violence (Bulletin, September 2000). 44,000 2, 300
    Online

    Fax- on- demand Enforcing the Underage Drinking Laws Program OJJDP Research
    2000 (Report, May 2001). 46,000 1, 366 Online

    Total 364,500 39,130

    a Full citation information for these products is contained in the
    bibliography. b ?Automatic mailing? refers to publications disseminated to
    targeted and registered individuals and groups. The Juvenile Justice
    Clearinghouse maintains 290 targeted mailing lists of key constituents that
    include criminal/ juvenile justice policymakers, practitioners, researchers,
    and community- based organizations. In addition, individuals can register
    with the Clearinghouse to automatically receive information based on their
    specific areas of interest. c ?Other distribution? refers to orders for
    specific publications. Anyone can order a publication by calling

    the toll- free line or using the NCJRS or OJJDP Web site. Source: OJJDP.

    Appendix III: Categorical Assistance Progress Report Form Page 37 GAO- 02-
    23 OJJDP Grantee Reporting Requirements and Concerns

    Appendix III: Categorical Assistance Progress Report Form

    Appendix III: Categorical Assistance Progress Report Form Page 38 GAO- 02-
    23 OJJDP Grantee Reporting Requirements and Concerns

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 39 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Twice a year, the Office of Juvenile Justice and Delinquency Prevention
    (OJJDP) grantees are required to complete a Categorical Assistance Progress
    Report- a narrative report that is to include a summary of the status of
    their particular projects? goals, quantitative project results based on
    performance measures set forth in their grant applications, actions planned
    to resolve any implementation problems, and any technical assistance they
    might need. In 8 of the 16 major programs we reviewed, grantees received
    only this general guidance, and were not subject to any additional reporting
    requirements. In the other eight programs we reviewed, grantees were
    required to follow this standard guidance and, in addition, report more
    specific information. Grantees in all 16 programs reported input, output,
    and/ or outcome data related to the process, implementation, and/ or
    accomplishments of their projects, such as acquisition of additional funding
    for a project evaluation, the number of project participants, or the number
    of missing children recovered.

    Table 7 provides summary information on the eight programs in which grantees
    are not subject to additional reporting requirements and examples from
    grantees? progress reports. Table 8 provides similar information regarding
    the other eight programs in which grantees are additionally required to
    report specified data to OJJDP or outside evaluators, as well as the
    specific performance measures on which grantees are required to report.
    Unless otherwise noted, the examples of reported information represent
    individual grantee or subgrantee data 1 for a 6- month period. Information
    provided regarding the specific data on which grantees are required to
    report do not necessarily include all performance data required.

    1 Examples from OJJDP?s five formula/ block grant programs were typically
    reported at the subgrantee or community level. Appendix IV: Descriptions of
    OJJDP

    Programs, Reporting Requirements, and Examples of Reported Information

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 40 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 7: Eight Programs GAO Reviewed With General Reporting Requirements
    Only Program name

    Active grantees (FY2000) Program description

    Examples of information reported by grantee( s) per general requirements

    State Challenge Activities 52 This program, established in 1992 under

    Title II, Part E of the Juvenile Justice and Delinquency Prevention Act of
    1974, as amended, provides incentive grants to states to improve their
    juvenile justice system. States must use funds to develop, adopt, or improve
    policies in 10 specified challenge activity areas, including (1) basic
    system services, (2) access to counsel, (3) community- based alternatives,
    (4) violent juvenile offender facilities, (5) genderspecific policies and
    programs, (7) state ombudsman, (8) deinstitutionalization of status
    offenders, (9) alternatives to school suspension and expulsion, and (10)
    aftercare services/ state agency coordination/ case review system. To
    receive these grants, states must also participate in the Title II, Part B
    Formula Grants Program. States may subaward State Challenge Activities funds
    to communities.

     Implemented an aftercare program for state wards returning to their
    community that provided community reintegration services including
    counseling, employment readiness services, family mediation, and weekly
    check- in and monitoring programs.

     Hired a disproportionate minority confinement coordinator.

     Home visitation research contract delayed due to paperwork complications.

     Held weekly meetings with female juveniles identified by the court to
    receive counseling and support through a gender- specific services program.

     Selected and made subaward to private provider with experience in
    developing intake services for youths cited by law enforcement with a focus
    on preventing unnecessary placement in attendant care or detention.

     Provided quarterly training to all case management staff on more rapidly
    and successfully returning youths in state custody from out- of- home
    placement into community and in- home placement. Community Prevention Grants
    Program (Title V)

    54 This program funds collaborative, community- based delinquency prevention
    efforts. The program integrates six fundamental principles- comprehensive
    and multidisciplinary approaches, research foundation for planning,
    community control and decision making, leveraging of resources and systems,
    evaluation to monitor program effectiveness, and a longterm perspective-
    that combine to form a strategic approach to reducing juvenile delinquency.

     Exceeded goal of providing career exploration and training on work
    awareness skills to 150 high school students.

     Trained 11 students to become mentors.

     Provided substance abuse education to six middle school girls.

     Experienced difficulties starting a delinquency prevention program to
    develop self- esteem and reduce inappropriate school behavior in at- risk
    seventh- grade students.

     Hired two outreach workers to work with elementary and junior high
    schools.

     Created an instructional video on quality daycare issues and distributed
    it to parents and the community.

     Enrolled and ?graduated? 13 children in a 6- week kindergarten preparation
    class. Juvenile Accountability Incentive Block Grants Program

    54 The Congress established this formula grant program in 1997 to promote
    greater accountability among juveniles who are involved in the juveniles
    justice system. To qualify for funds, states must demonstrate a commitment
    to increasing such accountability. Funds are allocated to states based on
    their relative population of youths under age 18, and states must pass on at

     Hired a case manager to monitor attendance and grades of juveniles in a
    youthful offender program.

     A local office declined to participate in the mentoring program and the
    subgrantee is considering alternative participants.

     Completed a study of achieving a system of graduated sanctions within the
    state.

     Enrolled 30 youths in a juvenile drug testing program and performed 52
    drug screens.

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 41 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program name

    Active grantees (FY2000) Program description

    Examples of information reported by grantee( s) per general requirements

    least 75 percent of their funds to local governments, absent a waiver. Funds
    can be used for 12 program purpose areas.

     Expanded a drug abuse program for juveniles referred to the court to
    include a second group involving both parents and juveniles. To date, the
    project has held 3 groups of 6- week classes.

     No juveniles completed a 6- week intensive educational program for
    substance abuse during the reporting period because of problems starting the
    program.

     The placement of a full- time police officer at a high school was delayed
    due to problems involving the officer?s training.

     Contracted with a part- time prosecutor to assist fulltime prosecutor with
    juvenile pre- trial and trial services, including compiling case
    information, preparing witnesses, and tracking cases. The parttime
    prosecutor works an average of 25 cases per week. In total, about 234 cases
    have been handled.

     Experienced difficulties in recruiting a data analyst due to problems with
    applicants meeting the minimum requirements. Enforcing the Underage Drinking
    Laws Program (block grant component)

    51 The block grant component of the Enforcing the Underage Drinking Laws
    Program helps states and the District of Columbia develop comprehensive and
    coordinated efforts to enforce state laws that prohibit the sale of
    alcoholic beverages to minors and to prevent the purchase or consumption of
    alcoholic beverages to those under age 21. Grantees may use funds to support
    activities in one or more of three priority areas, including (1) law
    enforcement and prosecution efforts that target establishments suspected of
    selling alcohol to minors; (2) public advertising campaigns to educate
    businesses and youths about the prohibitions against illegal alcohol sales
    and purchases; and (3) other innovative programs to prevent and combat
    underage drinking.

     Formed 15 youth task groups in order to examine the community norms and
    messages that young people get regarding alcohol.

     Placed billboards to educate the public about alcohol use in 20
    communities for 4 weeks.

     Produced a 60- second radio spot aimed at preventing underage alcohol use.

     Trained 20 community teams about prevention approaches to reduce underage
    drinking.

     Established a support position to improve coordination and communication
    between enforcement, prevention, and treatment systems.

     Held a full- day conference for established task force groups.

     Experienced implementation problems due to problems with students?
    schedules and limitations of the school year schedule.

     The local public television station worked with a youth task force group
    to design and film a 1- hour documentary about underage drinking. Enforcing
    the Underage Drinking Laws Program (discretionary grant component) a

    23 The discretionary grant component of the Enforcing the Underage Drinking
    Laws Program provides grants to states and territories to support the block
    grant component of the same program. Its purpose is to expand the number of
    communities taking a comprehensive approach to the problem of underage
    drinking, with an emphasis on increasing law enforcement activity regarding
    the sale of alcohol to minors. States that receive

     Three communities developed a compliance check database form to be used by
    law enforcement agencies.

     Requested technical assistance for a 1- day training session for local
    coalitions on environmentallybased activities.

     Published two reports on status of underage drinking in targeted
    community.

     Held two advisory committee meetings.

     Implemented new diversion program for underage alcohol violators and
    achieved a 32- percent success rate for youths completing the program.

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 42 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program name

    Active grantees (FY2000) Program description

    Examples of information reported by grantee( s) per general requirements

    these competitive awards select between two and eight diverse jurisdictions
    as subgrantees.

     Experienced difficulty in bringing one subgrantee on board; the subgrantee
    has not yet attended any planning sessions.

     Four communities hired part- time program coordinators.

     Conducted six monthly conference calls with four communities.

     Youth coalition placed 4,000 warning stickers on alcoholic beverages in 13
    stores.

     Youth coalition designed placemats containing underage drinking facts and
    figures for restaurant and grocery store distribution.

     Published two newsletters.

     Was not able to hold banquet recognizing top 100 businesses that had
    discouraged underage alcohol sales due to preparation demands of an
    antialcohol youth conference.

     Arranged for more than 700 youths to sign a pledge not to drink. Tribal
    Youth Program 81 Under this discretionary award program,

    federally recognized Tribes can apply for funds to (1) reduce, control, and
    prevent crime both by and against tribal youths; (2) provide interventions
    for court- involved tribal youths; (3) improve tribal juvenile justice
    systems; and (4) provide prevention programs focusing on alcohol and drugs.
    The program is intended to provide a flexible funding stream for tribes.

     Began keeping juvenile crime data for future analysis of whether targeted
    reductions have been achieved.

     Drafted court diversion protocol.

     Identified tribal judges and court clerks.

     Experienced set- backs in establishing four functional tribal courts to
    deal with juvenile crime for reasons including the resignation of the
    general counsel and competing demands of the commercial and subsistence
    fishing season.

     Reported conferences and training events attended.

     Met with state representatives about possible transfers of cases from
    state to tribal court.

     Began identifying courtroom equipment needs and researching purchases.

     Obtained examples of tribal juvenile codes.

     Taught conflict resolution skills to families upon request.

     Suffered a set- back in reducing cases of misbehavior and rules violations
    in the schools because the community truancy officer resigned and had not
    yet been replaced.

     Referred five youths for counseling as a result of conflict resolution
    curriculum presented in schools. Model Courts Program b 1 This program
    awards one earmarked c grant

    annually to the National Council of Juvenile and Family Court Judges to
    provide training and technical assistance to 23 ?model

    courts? around the nation that process child abuse and neglect cases. The
    program?s goal is to design and deliver technical

    Information reported by grantee through progress reports:

     Made 31 on- site staff visits to model courts.

     Held 96 national, regional, state, and local training events for nonmodel
    court jurisdictions.

     Supported the establishment of two model courts.

     Trained 9,729 model court and nonmodel court

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 43 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program name

    Active grantees (FY2000) Program description

    Examples of information reported by grantee( s) per general requirements

    assistance and training programs that improve dependency courts? handling of
    such cases. Model courts serve as national

    ?laboratories? for implementing systemswide change in the processing of
    abuse and neglect cases. Reforms and initiatives vary across courts. The
    Council also supports a mentoring program in which experienced model court
    personnel ?mentor? and train personnel from new model courts.

    personnel.

     Distributed 17,818 technical assistance bulletins.

     Conducted 134 telephone consultations.

     Distributed 2,341 resource packets. Information reported by grantee
    through annual status report: d

     Reported one court reduced its caseload of 58, 000 abused and neglected
    children in 1995 (prior to program implementation) to 27, 001 children in
    1999.

     Reported one court reduced the average length of abuse and neglect
    complaint cases from 3.5 years in court year 1998 to 2.4 years in court year
    1999.

     Reported one court established a child protection division with two judges
    who hear all child protection cases.

     Following this change, the average case length (from petition to
    adjudication) for one section of the division dropped from 140 days to
    between 29 and 41 days. National Clearinghouse and Resource Center for
    Missing and Exploited Children e

    1 The National Center for Missing and Exploited Children (NCMEC) is the
    national resource center and clearinghouse dedicated to missing and
    exploited children?s issues. By law, NCMEC is to (1) establish a toll- free
    hotline, (2) provide information on free or low- cost transportation for
    missing children and their families, (3) coordinate programs that reunite
    missing children with their families, (4) disseminate information that
    benefits missing and exploited children, and (5) provide training and
    technical assistance to law enforcement and other agencies.

     Upgraded toll- free hotline.

     Received 24,983 hotline calls.

     Assisted in recovering 1,610 missing children.

     Described missing children cases in which NCMEC assisted.

     Distributed 5,306 pictures of missing children.

     Described public awareness campaigns.

     Trained 1,708 law enforcement and health care personnel.

     Described joint projects with other agencies.

     Completed 48 age progressions of photographs.

     Received 5,291 child pornography tips online, and 492 tips through the
    hotline.

     Described technical assistance provided to other agencies. a This program
    was established in fiscal year 1998 under the name Combating Underage
    Drinking.

    b OJJDP has emphasized to the single grantee for this program that it should
    include quantitative performance data in its progress reports, but did not
    prescribe the specific indicators on which the grantee must report. c
    Earmarked grants are those for which the Congress sets aside a specified
    amount for an identifiable

    grantee and/ or a specific authorized purpose. d The grantee chooses to
    provide performance data regarding model court activities and

    accomplishments in an annual status report that is available to nonmodel
    courts and to the general public. e Although the grantee in this program is
    only required to submit progress reports semiannually, it chooses to submit
    progress reports on a quarterly basis. Examples presented in this table were
    excerpted from a recent quarterly progress report.

    Source: GAO analysis of OJJDP and grantee data.

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 44 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 8: Eight Programs GAO Reviewed With Specific Reporting Requirements
    and General Reporting Requirements Program name

    Active grantees (FY2000) Program description

    Specific data required from and reported by program grantee( s)

    Examples of information reported by grantee( s) per general requirements

    Formula Grants Program 54 OJJDP has administered this

    program since 1975. The program provides funds to help states, U. S.
    territories, and the District of Columbia implement a comprehensive state
    juvenile justice plan based on a detailed study of needs to support state
    and local delinquency prevention and intervention efforts and juvenile
    justice system improvements. To be eligible for a formula grant, a state
    must designate an agency to prepare a 3- year plan, establish a State
    Advisory Group, and commit to complying with the four core requirements of
    the Juvenile Justice and Delinquency Prevention Act of 1974, as amended.

    Grantees (states) must report data including but not limited to the
    following:

     Number of accused status offenders and nonoffenders held in any secure
    detention or correctional facility for more than 24 hours.

     Number of accused status offenders and nonoffenders securely detained in
    any adult jail, lockup, or nonapproved collocated facility for any period of
    time.

     Number of accused status offenders and nonoffenders placed in facilities
    not near their home, not the least restrictive appropriate alternative, and
    not community- based.

     Number of juvenile offenders and nonoffenders not separated from adult
    criminal offenders in facilities used for secure detention and confinement
    of both juveniles and adults.

     Number of juveniles detained in collocated facilities not approved by the
    state that were not sight and sound separated from adult criminal offenders.

     For 21 youths involved in a truancy abatement program, school attendance
    increased by an average of 89 percent.

     A runaway youth program provided counseling to 129 runaway youths and
    their families; of 57 cases closed, 86 percent of youths returned to home, a
    friend, or a relative.

     Discontinued a violence prevention curriculum for middle and high school
    students and a conflict resolution training program for eighth- grade
    students due to the programs? lack of effectiveness.

     Twenty- six status offenders and their families participated in family
    counseling activities that focused on parenting skills, communication, and
    prosocial behaviors.

     A mentoring program has continued to have difficulty in recruiting
    appropriate mentors.

     Enrolled 40 youths pending adjudication and/ or disposition in home
    detention programs in lieu of incarceration, shelter, or detention
    placement.

     Despite efforts, was unable to gather and analyze data on youths
    participating in a tutoring program due to a lack of school system
    cooperation. Comprehensive Gang Initiative 2 This program assists

    community coalitions in implementing and testing the comprehensive gang
    model- a collection of strategies aimed at the prevention, intervention, and
    suppression of youth gang crime and violence. The model requires communities
    to assess local gang problems, identify key risk factors among youths,
    develop program plans, and

     Number of juveniles served.

     Age, sex, and income level of participating youths.

     School participation and academic performance for participating youths.

     Criminal activity, gang activity, drug use, and alcohol use by
    participating youths.

     Project staff?s perceptions of the seriousness of gang and nongang
    problems in

     Received university grant to fund local project evaluation.

     Developed action team to address problem of female gang members.

     Project youths were not participating in tattoo removal program; project
    staff need to better promote benefits of tattoo removal.

     Recruited five new community leaders to join project.

     Held biweekly staff meetings to discuss progress of participating

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 45 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program name

    Active grantees (FY2000) Program description

    Specific data required from and reported by program grantee( s)

    Examples of information reported by grantee( s) per general requirements

    evaluate program outcomes. Coalitions focus on five core strategies:
    community mobilization, academic and economic opportunities, social
    intervention, gang suppression, and organizational change and development.

    the community.

     Nature and frequency of services provided to youths and families.

     Youth workers? (probation officers, case workers) perceptions of the
    effectiveness of services provided to youths and families.

    youths.

     Inadequate staffing levels caused delays in processing higher than
    expected numbers of youth referrals and prevented project expansion to
    additional neighborhoods.

     Revised staff procedures for delivering and coordinating warrants.

     Recruited 25 girls for selected projects. Rural Gang Initiative 4 A spin-
    off of the

    Comprehensive Gang Initiative that adapts the comprehensive gang model to
    rural communities. Two sites have incorporated the full version of the model
    and two sites follow a modified version of the Comprehensive Gang
    Initiative.

     Aggregated demographic data for community (unemployment rates, education
    attainment, race).

     Community attitude data (community?s perception of gang problem).

     Youth attitude data (perceptions about ganginvolved youths from at- risk
    youths).

     Law enforcement crime data (data on nature of criminal gang activity in
    community- number of assaults, property crimes, etc.)

     Community resource data (inventory of services available to gang and
    community members- e. g., conflict resolution, individual counseling, etc.)

     Experienced delay in hiring a project coordinator and outreach staff due
    to a lack of qualified applicants; project is preparing for second
    recruitment effort.

     Developed steering committee composed of representatives from community
    organizations involved with youths.

     Attended training on implementation of the comprehensive gang model.

     Located and rented a building to house project staff.

     Insufficient staff to follow up on court orders against targeted youths
    who were at- risk for joining a gang.

     In the process of forming an intervention team composed of law
    enforcement, school, and social services personnel.

     Hired a gang crime specialist.

     Computer failure prevented police department from providing crime data.
    Juvenile Mentoring Program (JUMP)

    175 JUMP provides one- to- one mentoring for youths at risk of delinquency,
    gang involvement, educational failure, or dropping out of school. Awards are
    made to local educational agencies or private or nonprofit agencies that
    have clearly defined relationships with local schools. Grantees must conduct
    thorough background checks on volunteer mentors to ensure a safe environment

    Youth information:

     Demographic information, such as age, gender, and ethnicity.

     Living situation.

     Medical or mental health problems.

     Total number of school absences. Mentor information:

     Demographic information such as age, gender, and ethnicity.

     Reason for becoming

     Exceeded goal of matching 50 atrisk girls with mentors.

     Mentors and mentees participated in classes highlighting other cultures.

     Retained 50 percent of mentors from the previous year.

     Experienced problems with parents who were unwilling to complete project
    requirements, such as interviews and home visits.

     Continued to recruit mentors from local businesses.

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 46 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program name

    Active grantees (FY2000) Program description

    Specific data required from and reported by program grantee( s)

    Examples of information reported by grantee( s) per general requirements

    for each child. JUMP also emphasizes projects that are established in
    communities where 60 percent or more of the youths qualify to receive a free
    or reduced- price lunch.

    involved in the mentoring program.

     Highest education completed.

     Previous mentoring experience.

     Twenty out of 26 mentors who responded to a survey said the program met
    their expectations and 23 said they would participate again.

     Twenty- five youths dropped out of a project. SafeFutures 6 This program,
    based on

    OJJDP?s comprehensive strategy model, targets communities that have already
    begun to reform their juvenile care systems. Program goals include
    developing a full range of graduated sanctions, ensuring community safety,
    and providing treatment services for juveniles. Communities use one grant to
    tap into nine OJJDP program components (e. g., juvenile mentoring,
    aftercare, mental health, delinquency prevention, gang programs, etc.)

     Demographic data (age, race, sex) for youth participants.

     Family/ household characteristics (employment, number of family members,
    number of juveniles/ adults living at home, school history, family rules)
    for youth participants.

     Self- reported data by youth participants (neighborhood characteristics,
    gang affiliation, number of delinquent acts, characterization of family
    relationships, drug and alcohol abuse, emotional problems, participation in
    project activities, drug and alcohol abuse by family members).

     Developed and submitted applications for additional funding.

     Experienced problems providing substance abuse treatment services due to
    the lack of a fulltime, trained substance abuse counselor.

     Hired a new local evaluator.

     Referred 249 juveniles for direct services and care coordination.

     Completed 13 community service projects involving 394 students and
    families.

     Experienced problems implementing a mentoring program- i. e., recruiting
    mentors, retaining matches, and motivating mentors to attend sessions.

     Continued coordination between law enforcement agencies on gang
    intelligence resulted in the arrest of two arson suspects.

     Installed a computer system to track gang members and gangrelated
    activities in the community. Drug- Free Communities Support Program

    307 This program is intended to increase citizen participation and
    strengthen community anti- drug coalition efforts to reduce substance abuse
    among youths in communities throughout the United States and, over time, to
    reduce substance abuse among adults.

     Number of organizations represented on the grantee?s coalition.

     Types of organizations represented on the coalition.

     Number of individuals who are coalition members.

     Number of youths, parents, and community leaders who are coalition
    members.

     Number of recruitment activities held.

     Steps taken to engage members in drug prevention planning.

     Activities sponsored by the coalition.

     Number of participants in activities.

     Exceeded goal of 100 community organizations participating in workshops.

     Produced anti- smoking commercial.

     Invited health professionals to a strategic planning meeting.

     Staff worked with drug- free clubs.

     Hired first full- time staff.

     Printed quarterly newspaper.

     Distributed 20,000 copies of a family resource directory.

     Experienced low attendance at a strategic planning meeting.

     Provided drug prevention education to 78 after- school detention students.

    Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
    Examples of Reported Information

    Page 47 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program name

    Active grantees (FY2000) Program description

    Specific data required from and reported by program grantee( s)

    Examples of information reported by grantee( s) per general requirements

     Target population for activities. Internet Crimes Against Children Task
    Force Program

    50 This program encourages communities to develop regional, multi-
    jurisdictional, and multi- agency responses to online enticement and child
    pornography cases. These task forces provide forensic, prevention and
    investigative assistance to parents, law enforcement, prosecutors,
    educators, and other professionals working on child victimization issues.

    Grantees in this program must complete a monthly performance measures report
    that collects information including, but not limited to:

     Number of arrests.

     Number of child pornography producer cases.

     Number of child pornography seller cases.

     Number of search warrants.

     Number of subpoenas.

     Number of referrals from other agencies.

     Number of law enforcement personnel trained.

     Number of presentations given to students, parents, teachers, citizens or
    the media.

     Provided summaries of 24 cases during the reporting period.

     Hired a child victim service specialist.

     Achieved goal of implementing training seminars for local, states and
    federal law enforcement officers.

     Acquired necessary equipment for investigating online offenders such as
    desktop and laptop computers, cameras, and investigative software.

     Made presentations on Internet safety to teachers, parents, community
    groups, and schoolage children.

    Children?s Advocacy Centers

    5 This program provides training and technical assistance to local
    children?s advocacy centers and promotes their expansion nationwide.
    Children?s advocacy centers coordinate law enforcement and protective
    services responses to victims of child abuse through multidisciplinary
    teams. Under this program four regional centers and one national association
    provide training and technical assistance to local centers; the national
    association also awards subgrants to local centers for training, program
    development, and other purposes.

    Training and technical assistance grantees are required to report on:

     Number of training sessions conducted.

     Number of practitioners trained.

     Number of technical assistance requests filled.

     Number of consultations (on- site, electronic, telephone) provided.

     Number of publications developed and distributed. National grantee is
    required to report subgrantee data on:

     Number of new and developing local centers.

     Number of on- site monitoring visits.

     Number of grant applications approved.

     Type of grants approved (program development, program support, training,
    and state chapter).

    Training and technical assistance grantees:

     Finalized a diversity plan to ensure that children receive services in a
    culturally appropriate setting.

     Experienced difficulties agreeing on research methods to evaluate the
    children?s advocacy centers? impact on case outcomes.

     Developed a ?multidisciplinary

    team development? curriculum.

     Identified and provided assistance to underserved rural communities.
    National grantee:

     Reviewed subgrantees? biannual reports on activities held during the
    reporting period.

     Unexpected resignation of grants coordinator hampered data collection
    efforts and the awarding and processing of grants.

     Approved 409 applications for subgrants and notified recipients.

     Surveyed local accredited centers and found that 49,350 juveniles had been
    served during the reporting period.

    Source: GAO analysis of OJJDP, grantee, and evaluator data.

    Appendix V: Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees

    Page 48 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    The Office of Juvenile Justice and Delinquency Prevention?s (OJJDP) training
    and technical assistance programs and research programs are unique in that
    they cut across many of OJJDP?s other programs. Also, grantees 1 in each of
    these two areas typically report the same types of quantitative performance
    data as other grantees in their area, even though OJJDP does not usually
    prescribe the specific performance measures on which the grantees should
    report. Training and technical assistance grantees maintain the same types
    of data due to the common support services they provide, and research
    grantees do the same because they share a common goal of producing research
    products.

    OJJDP awards grants to training and technical assistance providers to
    support grantees in many of OJJDP?s grant programs. OJJDP administers the
    vast majority of its training and technical assistance grants through three
    of the Office?s divisions: (1) the Training and Technical Assistance
    Division (TTAD), (2) the State and Tribal Assistance Division (STAD), and
    (3) the Child Protection Division (CPD). 2 Many of the training and
    technical assistance providers are required to report information on their
    projects? activities and accomplishments semiannually using OJP?s
    Categorical Assistance Progress Report form, as do all OJJDP grantees. 3 OJP
    provides standard guidance on information to be reported, such as
    information on the status of each of the grantees? project goals and
    quantitative results of their projects. STAD has not imposed additional or
    more specific reporting requirements on its training and technical
    assistance providers and, for the most part, neither have TTAD 4 nor CPD. 5

    1 OJJDP awards grants, cooperative agreements, and contracts to training and
    technical assistance providers, and award grants and cooperative agreements
    to research providers. Officials commonly refer to all of these as ?grants?
    and to all award recipients as

    ?grantees.?

    2 In commenting on a draft of this report, the Assistant Attorney General
    noted that these divisions do not always provide the same types of services.
    3 All training and technical assistance providers that receive grants are
    required to complete progress reports. Providers that are under contract or
    have cooperative agreements with OJJDP must also report on their
    accomplishments, but not necessarily through a progress report. Often, they
    are required to report their activities on a monthly basis.

    4 In commenting on a draft of this report, the Assistant Attorney General
    noted that TTAD requires Juvenile Accountability Incentive Block Grants
    Program grantees to submit monthly reports. These grantees comprise
    approximately one- fifth of all TTAD grantees and account for approximately
    $5 million annually. Appendix V: Training and Technical

    Assistance and Research Performance Data Reported by OJJDP Grantees

    Performance Data for Training and Technical Assistance Grantees

    Appendix V: Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees

    Page 49 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Officials explained that OJJDP does not require all grantees to routinely
    report prescribed data because it is reluctant to place additional reporting
    requirements on grantees due to the Paperwork Reduction Act of 1995, 6 which
    set goals to reduce the federal government?s reporting and paperwork burden.
    Although most of these providers are not subject to additional reporting
    requirements for prescribed data, it is not unusual for them to report on
    the same or similar quantitative performance measures. Because of the nature
    of the services they provide, training and technical assistance providers
    tend to maintain like data that can readily be counted, such as numbers of
    training events held, practitioners who attended those events (?
    practitioners trained?), and technical assistance requests filled. Some of
    these providers also produce publications or materials, such as bulletins,
    surveys, curricula, brochures, and other support materials, and report such
    information to OJJDP. Table 9 summarizes performance data we obtained
    regarding training and technical assistance grants. 7

    5 CPD has prescribed additional, specific reporting requirements for
    providers in two of its programs- the Children?s Advocacy Centers and the
    Missing and Exploited Children Training and Technical Assistance program-
    such as reporting the number of training events held.

    6 The Paperwork Reduction Act applies to identical reporting requirements
    imposed on 10 or more parties. CPD?s training and technical assistance
    providers that have specific reporting requirements are not affected by the
    Act since they involve less than 10 providers.

    7 We attempted to obtain fiscal year 2000 data from all grantees; however,
    in some cases OJJDP provided calendar year 2000 or fiscal year 2001 data due
    to a lack of available or representative data. Because we obtained
    publications data that pertained only to some of the grantees, we did not
    include such information in the table. For example, State Challenge
    Activities training and technical assistance providers reported producing 10
    bulletins in fiscal year 2001and Juvenile Accountability Incentive Block
    Grant providers reported that in fiscal year 2000 they developed or were
    developing 628 supportive materials, such as reports, curricula, and
    newsletters.

    Appendix V: Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees

    Page 50 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 9: Training and Technical Assistance Performance Data Reported by
    OJJDP Grantees

    Division/ program supported Training events Practitioners trained

    Technical assistance

    requests filled a Training and Technical Assistance Division (FY2000)

    Juvenile Accountability Incentive Block Grants Program 227 13,816 1, 374
    Life Skills Training b 64 858 16 National Training and Technical Assistance
    Center c d d 329 Other programs e f 36,000 1, 633 g

    State and Tribal Assistance Division (FY2001) h

    Community Prevention Grants Program (Title V) 55 2,171 4 Formula Grants
    Program 10 1, 148 265 State Challenge Activities 2 69 71 Comprehensive
    Strategy for Serious, Violent, and Chronic Juvenile Offenders

    76 2,252 232

    Child Protection Division (Calendar Year 2000)

    Children?s Advocacy Centers 183 10,939 6, 501 i Court Appointed Special
    Advocate Program 194 5,312 21,323 Model Courts Program 161 42,537 401
    Permanency Planning 124 38,888 4, 264 Prosecutor Training f 9,300 1,800
    Parents Anonymous 3 149 f Missing and Exploited Children Training and
    Technical Assistance program j

    68 2,950 f a Data can include the number of in- depth, on- site visits and/
    or the number of responses to telephone and e- mail requests. b Data for
    this program were from the academic year 2000- 2001, which roughly
    corresponds with fiscal

    year 2001. c The Center receives requests from the field for technical
    assistance, which it then fills in- house or refers to other technical
    assistance providers. In fiscal year 2000, it either filled or referred 329
    requests for technical assistance. The Center does not provide direct
    training to the field. d Not applicable.

    e Data reported from 19 other training and technical assistance grantees. f
    Data not available. g Reflects the number of communities that received
    technical assistance.

    Appendix V: Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees

    Page 51 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    h At the suggestion of OJJDP officials responsible for the following
    programs, data for the Community Prevention Grants Program are prorated from
    the first 9 months of the fiscal year, training and technical assistance
    data for State Challenge Activities are prorated from the first 8 months of
    the fiscal year, and data for the Comprehensive Strategy for Serious,
    Violent, and chronic Juvenile Offenders are prorated from the first 6 months
    of the fiscal year. i Includes data from the national grantee and three of
    the four regional grantees.

    j Data are for fiscal year 2000. Source: GAO analysis of OJJDP and grantee
    data.

    OJJDP officials cautioned that not all providers share common definitions of
    ?training? and ?technical assistance.? 8 For one thing, the difference
    between the two is not always clear and, therefore, it is sometimes
    difficult to definitively categorize a provided service as training versus
    technical assistance. Furthermore, not all training events are equal. For
    example, some providers might characterize both a 1- day training conference
    and a 10- day training workshop as a training event; others might
    differentiate between the two. Furthermore, one provider might consider a
    telephone request from a grantee as merely a query, while another might
    consider it a request for technical assistance.

    OJJDP administers its research grants out of its Research and Program
    Development Division (RPDD). 9 RPDD sponsors empirical studies on an array
    of topics related to juveniles and delinquency, from the causes of violence
    to the impact of victimization. The overall goal of these research grants is
    to generate credible and useful information to help prevent and reduce
    juvenile delinquency and victimization. Research grantees are not only
    expected to collect data but to analyze and disseminate their analyses to
    the public. RPDD requires all research grantees to produce publishable
    products and, in some instances, RPDD specifies the type of products to be
    published depending on the results of the research. Thus, according to OJJDP
    officials, one measure of a research grantee?s performance is the number of
    products the grantee has published.

    Like most OJJDP grantees, research grantees must report information on their
    projects? activities and accomplishments semiannually through

    8 In commenting on a draft of this report, the Assistant Attorney General
    pointed out that TTAD published the Training, Technical Assistance, and
    Evaluation Protocols booklet 2 years ago, which defined both training and
    technical assistance.

    9 Some grants administered through other OJJDP divisions have a research
    component, but they are not considered ?research grants.? Performance Data
    for

    Research Grantees

    Appendix V: Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees

    Page 52 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    progress reports. RPDD does not impose additional, specific reporting
    requirements on grantees, but it does encourage them to report on products
    produced through private publishers (as opposed to those published through
    OJJDP). 10 The division director told us that it is not necessary to impose
    specific requirements on grantees in addition to the semiannual progress
    report requirements because officials work closely with grantees throughout
    the life of the grants.

    OJJDP research grantees produce products based on their OJJDP- funded
    research. Some of these products are approved and published by OJJDP, who in
    turn disseminates the products through its own distribution process (see
    app. II for a description of OJJDP?s product dissemination process).
    Grantees also publish many products that are based on their OJJDP- funded
    research through private publishers. OJJDP officials told us they give their
    research grantees latitude to privately publish products because the
    majority of their research grantees are academics whose funding depends on
    the number of products they publish, and because grantees often have funding
    sources in addition to OJJDP.

    Tables 10 and 11 summarize the products that active OJJDP research grantees
    published through OJJDP and private publishers as a direct result of OJJDP-
    funded research. Table 10 describes the number of research products
    published by OJJDP from 1993 through September 2000, by topic. Table 11
    shows the number of products, by topic, that grantees with active research
    grants privately published between 1986 and June 2001, or were in the
    process of publishing in June 2001.

    10 Officials told us that they ask grantees to submit products for review
    prior to external publication, but often grantees do not do this. In the
    past, grant applicants have turned down funding when it was contingent on
    OJJDP?s review of privately published products, citing the importance of
    academic freedom. Also, grantees oppose submitting external products to
    OJJDP prior to publication because many academic journals will not publish
    research findings that have been reviewed or disseminated prior to
    submission.

    Appendix V: Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees

    Page 53 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 10: Active OJJDP Grantees? Research Products, Published by OJJDP Topic
    Summaries Reports Fact sheets Bulletins

    Aftercare (post- detention) 2 1 1 3 Causes and Correlates of Delinquency a 1
    2 46 Crimes against juveniles 0 0 0 7 Delinquency prevention 1 5 1 6
    Detention/ corrections 3 2 7 2 Gangs 5 0 4 8 Juvenile courts 1 3 9 4
    Juvenile health 0 1 1 2 Juvenile justice system reform 1 2 0 2 Juvenile
    offenders 3 3 5 10 School safety 0 2 0 0 Other 0 1 1 0

    Total 17 22 33 50

    Note: Table includes some products published as a result of OJJDP evaluation
    and statistics grants. a This topic is comprised of products from Causes and
    Correlates of Delinquency grants. In commenting on a draft of this report,
    the Assistant Attorney General noted that these products can be categorized
    under a number of the topics listed.

    Source: GAO analysis of OJJDP data.

    Table 11: Active OJJDP Grantees? Research Products, Published Externally
    Topic Journal

    articles Book chapters Other publications a

    Causes and Correlates of Delinquency b 57 43 17 Crimes against juveniles 12
    c 6 2 Delinquency prevention 0 0 2 Detention/ corrections 3 0 0 Gangs 13 d 9
    2 Juvenile courts 5 1 1 Juvenile health 51 e 5 7 Juvenile offenders 3 0 2
    School safety 16 0 4 Other 0 0 5

    Total 160 64 42

    Note: Table includes some products published as a result of OJJDP evaluation
    and statistics grants. a Includes books, reports, and other publications.

    b This topic is comprised of products from Causes and Correlates of
    Delinquency grants. In commenting on a draft of this report, the Assistant
    Attorney General noted that these products can be categorized under a number
    of the topics listed.

    Appendix V: Training and Technical Assistance and Research Performance Data
    Reported by OJJDP Grantees

    Page 54 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    c Includes one paper submitted but not yet accepted for publication. d
    Includes one paper presented at an academic conference. e Includes three
    papers submitted but not yet accepted for publication.

    Source: GAO analysis of OJJDP and grantee data.

    Appendix VI: States? Compliance With Core Requirements of the Juvenile
    Justice and Delinquency Prevention Act of 1974

    Page 55 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Table 12: State- Reported Compliance with Core Requirements, Calendar Year
    1998 Core requirements Number of

    jurisdictions Jurisdictions a Deinstitutionalization of status offenders

    Full compliance - zero violations 9 ME, MT, NY, VT, AS, GU, MP, PR, VI Full
    compliance - de minimis exceptions b 39 AL, AK, AR, CA, CO, CT, DE, DC, FL,
    GA, HI, ID, IL, IN,

    IA, KS, KY, LA, MD, MA, MN, MO, NE, NV, NH, NM, NC, ND, OK, OR, PA, RI, TX,
    UT, VA, WV, WI Not in compliance 4 MS, SC, TN, WA Funds withheld pending
    additional compliance data 2 NJ, OH Not participating 2 SD, WY

    Separation of juvenile and adult offenders

    Full compliance - zero violations 41 AL, AZ, AR, CA, CO, CT, DE, DC, FL, IL,
    IN, KS, KY, ME, MD, MA, MI, MN, MO, MT, NE, NV, NH, NM, NC, ND, OK, OR, PN,
    RI, SC, UT, VT, VA, WA, WI, AS, GU, MP, PR, VI Full compliance - exception
    provision 11 AL, GA, HI, ID, IA, LA, MS, NY, TN, TX, WV Funds withheld
    pending additional compliance data 2 NJ, OH Not participating 2 SD, WY

    Jail and lockup removal

    Full compliance - zero violations 11 AL, DC, NC, OR, SC, VT, AS, GU, MP, PR,
    VI Full compliance - de minimis exceptions 38 AZ, AR, CA, CO, CT, DE, FL,
    GA, HI, ID, IN, IA, KS, KY,

    LA, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NM, NY, ND, OK, PA, RI, TN, TX,
    UT, VA, WA, WV, WI Not in compliance 3 AK, IL, ME Funds withheld pending
    additional compliance data 2 NJ, OH Not participating 2 SD, WY

    Disproportionate minority confinement c

    Completed identification and assessment, implementing intervention and
    monitoring 21 AR, CO, CT, FL, GA, ID, IN, IA, LA, MO, NV, NC, ND, OK,

    PA, SC, TN, UT, VA, WA, WV Completed identification and assessment,
    implementing intervention 15 AK, AZ, HI, IL, KS, MD, MA, MI, MN, NE, NM, NY,
    RI, TX,

    WI Completed identification and assessment/ updating data and revising
    intervention plan 2 CA, D E Completed identification/ implementing
    intervention/ conducting formal assessment 3 AL, MS, MT Conducting
    identification phase 1 NH Exempt from requirement 7 ME, VT, AS, GU, MP, PR,
    VI Status under review 5 DC, KY, NJ, OH, OR Not participating 2 SD, WY

    Note: OJJDP based its initial determination of states? eligibility to
    receive fiscal year 2000 Formula Grants on 1998 monitoring reports- the most
    recent data available as of March 2001. a AS = American Samoa; GU = Guam; MP
    = Northern Mariana Islands; VI = Virgin Islands.

    b Fewer than 29.4 violations per 10,000 persons under age 18 in the state. c
    States reported data on disproportionate minority confinement in their
    fiscal year 2000 comprehensive plans for compliance.

    Appendix VI: States? Compliance With Core Requirements of the Juvenile
    Justice and Delinquency Prevention Act of 1974

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 56 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    This appendix contains information on the 10 impact evaluations that the
    Office of Juvenile Justice and Delinquency Prevention (OJJDP) has funded of
    its own programs since 1995 and for which we have assessed the
    methodological rigor, as well as information on one impact evaluation- Teen
    Courts- that we did not assess. Five of the 10 evaluations are in their
    formative stages, and five are well into their implementation. For each of
    the 10, we have included a description of the program being assessed, the
    evaluating organization, a description of the evaluation and its findings,
    and our assessment of the evaluation. 1 As discussed in the Scope and
    Methodology section of this report, we did not assess the methodological
    rigor of the Teen Courts evaluation. However, we have included a summary of
    this evaluation at the end of this appendix. 2

    Program Description: Parents Anonymous is a national child abuse prevention
    program that began in 1970. It consists of 32 state and local organizations
    and over 1,000 weekly mutual support groups. The principal participants are
    at- risk parents, though complementary projects exist for children. The
    cornerstones of the program are mutual support and shared leadership.

    Evaluator: National Council on Crime and Delinquency.

    Evaluation description: This evaluation, which is in the beginning stages,
    is based on a proposal to conduct a process evaluation in year 1, and an
    outcome evaluation in years 2 and 3. The researchers will determine how
    Parents Anonymous is staffed and operated in different settings, how it
    attempts to change the behavior and attitudes of parents, and what factors
    are related to its effectiveness. While the specifics of an outcome
    evaluation design are yet to be determined, the researchers

    1 We relied on documents OJJDP provided to us in April 2001 in assessing the
    evaluation methodologies and reporting on each evaluation?s status. 2 OJJDP
    provided us with this evaluation description. Appendix VII: Impact
    Evaluations OJJDP Has

    Funded of Its Own Programs Impact Evaluations of OJJDP Programs in Their
    Formative Stages

    Parents Anonymous

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 57 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    indicate that they will most likely compare the Parents Anonymous
    participants with a control group and with Parents Anonymous dropouts. The
    process evaluation received $300, 000 for a 3- year period.

    Evaluation findings: It is too early in this evaluation to have reported
    results.

    GAO assessment: No assessment of the impact evaluation is possible because
    it has not yet been planned. More fully developed proposals will need to be
    made to OJJDP to obtain funding for the impact evaluation portion of the
    study.

    Program Description: This program replicates and evaluates Project PATHE,
    which was first implemented in the Charleston County School District in
    South Carolina between 1980 and 1983. Project PATHE is a comprehensive
    school- based program that combines services to students who are at elevated
    risk for developing problem behaviors with schoolwide organizational changes
    intended to improve both school climate and students? behavior. Local
    educators are encouraged to develop their own (1) explanations for the
    causes of their schools? violence and behavior problems and (2) specific
    local objectives and ways to prevent these problems by empowering teachers?
    decision- making and fostering collaborative and nonhierarchical efforts.
    For various reasons, the grantee has had difficulty in selecting a school
    district for the replication. Funding for this effort began in October 1999
    with funds provided by the Centers for Disease Control and Prevention.

    In commenting on a draft of this report, the Assistant Attorney General
    pointed out that all funds for this effort come from the Centers for Disease
    Control and Prevention through an interagency agreement. OJJDP awarded this
    grant, which the two agencies jointly manage. OJJDP originally identified
    Project PATHE as an OJJDP- funded program. However, on the basis of the
    Assistant Attorney General?s comments, this program appears to be a non-
    OJJDP- funded impact evaluation. Since we assessed its methodological rigor,
    we have included it with the other OJJDP- funded evaluations.

    Evaluator: Institute of Behavioral Science, University of Colorado.

    Evaluation description: Although the impact evaluation was expected to be
    completed in July 2001, it has yet to begin. Original plans called for one
    school district to be selected for the replication and evaluation. In a
    Positive Action Through

    Holistic Education (Project PATHE)

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 58 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    school district, one high school and one middle school would be selected to
    receive the program, once school principals had been informed and staff
    surveys had been conducted to determine interest in participating.
    Comparison schools (one high school and one middle school) in the same
    district would be selected with similar demographic characteristics of
    students, levels of problem behaviors, and other unspecified organizational
    characteristics, as well as a low probability of mounting other school- wide
    efforts to reduce problem behavior during the study period. Plans are to
    collect data before and after project implementation. All students and staff
    in all schools would be surveyed in September and May for 3 consecutive
    school years. In addition, 200 students from each high school- 100 seniors
    and 100 sophomores- and 200 students from each middle school- 100 eighth
    graders and 100 sixth graders- would be sampled in the first year of the
    study for followup for 3 years. Each sample is to include 25 students
    identified as ?high risk.? It is not clear, however, how these samples will
    be selected. Schools are to be visited three times yearly during the study,
    and school records and teacher ratings of student behavior in each school
    will be used in establishing differences between the program and comparison
    schools. Program outcomes are to be selected after determining goals and
    objectives collaboratively with local school officials. Multivariate
    statistical analyses, such as logistic regression, are planned. Because of
    the difficulty in selecting a school district the latest progress report
    indicates some changes in this design. Agreements have been signed with two
    school districts (Charleston, SC and Baltimore, MD), instead of one, and
    plans are to conduct the replication and evaluation in four middle schools
    (two program and two comparison schools) in each district- high schools have
    been excluded. The amount of the grant is $875,000. However, additional
    funds have been requested.

    Evaluation findings: It is too early in this evaluation to have reported
    results.

    GAO assessment: The evaluation, as designed, is basically sound. The
    variation in program structure and implementation between schools may limit
    generalizability. While the researchers do suggest awareness of potential
    problems due to students switching schools, they do not clearly indicate, at
    this early design stage, how possible contamination will be handled.

    Program Description: The Rural Gang Initiative is a comprehensive strategy
    to ameliorate gang problems in rural areas. The program was Rural Gang
    Initiative

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 59 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    adapted from the comprehensive gang model, developed at the University of
    Chicago, and implementation began in two rural areas in fall 2000. The
    program consists of five elements: community mobilization, opportunities
    provision, social intervention, suppression, and organizational change and
    development.

    Evaluator: National Council on Crime and Delinquency.

    Evaluation description: The impact evaluation of this program began in
    January 2001 and is expected to be completed in December 2003. It focuses on
    two of the four sites, Mount Vernon (IL) and Glenn County (CA), that were
    part of a year- long feasibility study that began in April 1999. The other
    two sites are not part of the evaluation because they did not fully
    implement the model. 3 Information will be collected from these two sites on
    gang- involved youths and youths at- risk of gang involvement, all of whom
    have participated in the program. However, it is unclear how the youths will
    be sampled or whether all participants will be included. Data will be
    obtained from a variety of sources, including interviews, organizational
    surveys, and administrative data from schools and the justice system. No
    comparison groups are planned, though the researchers indicate that attempts
    will be made to collect data that will permit an assessment of alternative
    explanations for program effects. The researchers plan to collect data
    before and after program implementation; then they will measure any changes
    and follow- up with program participants for at least 12 months after their
    participation in the program. Individual sites are to identify specific
    program outcomes. Although the researchers have described prospective
    outcome measures, such as the reduction of gang- related crime and the
    prevention or reduction of gang involvement, they (1) have not chosen the
    outcome measures that they will use and (2) have not provided information on
    the types of statistical analyses planned. However, the evaluation is still
    in the formative stages. At the time of our review, this evaluation had
    received $525,000 in funding.

    Evaluation findings: It is too early in this evaluation to have reported
    results.

    3 In commenting on a draft of this report, the Assistant Attorney General
    pointed out that these two sites did not implement the comprehensive gang
    model because their gang crime problem did not warrant the model?s
    prescribed intervention approach.

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 60 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    GAO assessment: It is too early to tell from this evaluation how effective
    it will be. However, the absence of plans for comparison groups and the lack
    of specificity regarding any other control mechanisms, make it unclear how
    program effects will be distinguished from alternative explanations at this
    stage of the evaluation.

    Program Description: The Safe Schools/ Healthy Students program has funded
    77 school districts nationwide, with grants ranging up to $3 million, to
    develop services and activities to promote healthy childhood development and
    prevent violence and drug abuse. The program also aims to develop greater
    collaboration and cooperation between communities and schools to enhance
    their effectiveness in responding to and reducing violence. Each project
    model is intended to evolve over time.

    Evaluator: Research Triangle Institute.

    Evaluation description: This impact evaluation began in October 1999 and
    data will be collected through the Spring of 2005. School and community-
    based archival records, surveys of key coalition personnel, teachers,
    superintendents, principals and other school staff, and teacher behavioral
    checklists for students in selected grades will be gathered in all 77 school
    districts. The evaluation will compare data from participating sites with
    national norms and with similar information from matched nonparticipating
    (comparison) sites that the researchers surveyed in each of two large,
    nationally representative studies of school districts. The matching will be
    based on unspecified socio- demographic characteristics and responses to
    policy- related questions in the baseline survey. Archival data will be
    collected yearly over a 5- year period. Survey data will generally be
    collected at three points in time, about 2 years apart. The survey items
    will be drawn from established instruments and will provide, in conjunction
    with the archival or administrative data, information on behavioral
    outcomes, risk factors and inhibiting factors, and indicators of positive
    development and mental health. At the time of our review, this evaluation
    was funded at approximately $5.6 million.

    Evaluation findings: It is too early in this evaluation to have reported
    results.

    GAO assessment: This evaluation, as designed, is basically sound. Safe
    Schools/ Healthy

    Students

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 61 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program Description: This demonstration program seeks to prevent and reduce
    the impact of family and community violence on young children, primarily
    aged 0 to 6, in up to 12 communities. The program plans to create a
    comprehensive service delivery system that integrates service providers (in
    the fields of early childhood education/ development, health, mental health,
    and all manner of prevention, intervention, and treatment programs), law
    enforcement, legal services, and the courts. It also seeks to improve the
    access, delivery, and quality of services to children exposed to, and at
    high risk of, violence. Project sites are to be selected through a
    competitive grant process. Funding for the program began in October 1999.

    Evaluator: Caliber Associates.

    Evaluation description: This evaluation began in May 2000, and is expected
    to end in September of 2005. The effect of the Safe Start Initiative will be
    measured within and across all participating communities at both the
    community and individual levels. Multiple data collection methods, including
    focus groups, service agency usage logs and documents, and random- digit-
    dialing telephone surveys, will be used. Plans are to collect data before
    program implementation and for 4 years after the program begins in each
    site. Although specific outcome measures have not yet been identified, they
    are expected to address such areas as agency referral levels and quality of
    service, increased interagency collaboration, knowledge and perceptions of
    police and child protective services, rates of child maltreatment, physical
    injuries, and mental health problems. No comparison communities are to be
    studied. Analyses are to include regression and time series models. At the
    time of our review, this initiative had received $1 million.

    Evaluation findings: It is too early in this evaluation to have reported
    results.

    GAO assessment: The absence of any appropriate comparison communities and
    the variability in program implementation and components across the 12 study
    sites will make it difficult to find compelling evidence of program effects.
    Safe Start Initiative

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 62 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program Description: This program aims, in the five sites in which it is
    being implemented and evaluated (Mesa and Tucson, AZ; Bloomington, IL; San
    Antonio, TX; and Riverside, CA) to reduce gang- related crime through five
    interrelated strategies: community mobilization; provision of social,
    educational, and economic opportunities; suppression of gang violence;
    social intervention; and organizational innovation. It involves the
    collaborative efforts of the police, probation officers, prosecutors,
    judges, schools, youth agencies, churches, housing authorities, and
    governmental agencies. It targets youths at strong risk of gang membership
    and crime, and youths already involved in serious gang crime. The evaluation
    began in 1995.

    Evaluator: University of Chicago.

    Evaluation description: The impact evaluation of the program began in May
    1995, and is expected to be completed in April 2002. Each project site is to
    be matched with a comparison site. In four of the five sites, the program
    participants and comparison groups were selected from similar gang problem
    areas within the same city; in the fifth site, a separate comparison
    community was selected. Between 100 and 115 youths, ages 12 to 21, who were
    involved in gangs or at risk of involvement, were selected to participate in
    the program in each site, and between 77 and 134 similar youths in each site
    were selected for comparison purposes. Neither program nor comparison group
    youths were selected randomly. A large, complex, communitywide data
    collection effort is being employed in each site, through a variety of
    methods and sources, including organizational surveys, youth surveys,
    reports from service workers, police and school records, local newspaper
    reports, and census data. Data were to be collected at baseline and after
    the first and third years of the program. The principal outcomes to be
    measured are gang crime patterns at the individual, gang, and community
    levels. The evaluation will also consider changes in opportunities, as well
    as integration in and alienation from conventional individuals and
    institutions. A variety of analyses of the data are planned, including time-
    series analyses and hierarchical linear models. At the time of our review,
    the evaluation had received approximately $3 million. Impact Evaluations of

    OJJDP Programs Well Into Their Implementation

    Comprehensive Communitywide Approach to Gang Prevention, Intervention, and
    Suppression Program (Comprehensive Gang Initiative)

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 63 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Evaluation findings: Preliminary results have been reported for program
    participants, but not comparison groups. Because of missing data and other
    problems, reporting deadlines may not be met, and two of the five project
    sites and associated comparison sites have been deferred from the current
    analyses.

    GAO assessment: The evaluation, as designed, is basically sound. However,
    numerous difficulties in obtaining data threaten parts of the evaluation.
    Also, the way in which subjects were selected for the study may be
    problematic, and it is unclear how program and comparison youths were
    matched within sites.

    Program Description: Since 1998, 85 communities and 4 colleges in at least
    10 states have been awarded subgrants under the discretionary grant
    component of this program to enforce underage drinking laws. In most states,
    a diverse group of stakeholders are involved in planning a variety of
    projects under this program that can include media campaigns, merchant
    education, compliance checks and other enforcement, youth leadership
    training, school- based education, and the development of local coalitions
    and interventions aimed at reducing underage drinking. States and
    communities are given substantial latitude in planning their projects;
    interventions are not standardized across communities.

    Evaluator: Wake Forest University School of Medicine.

    Evaluation description: The effort to evaluate the discretionary grant
    component of this program began October 1, 1998, and is expected to be
    completed December 31, 2001. Data are to be collected- from telephone
    surveys of police chiefs, sheriffs, and youths in participating communities
    and matched comparison communities in at least 9 states- before or early on
    in project implementation and at least 1 year after project initiation.
    Project sites to be evaluated were initially chosen from all states
    participating in the program. It is unclear whether these sites are
    representative of all participating project sites. In the first year,
    surveys were conducted in 52 participating communities and a similar number
    of comparison communities. In the second and third years, surveys will be
    conducted in those same communities and 34 others- 17 in each group. The
    participant and comparison communities were matched on median income, liquor
    law violations, percentage attending college, and population size. The
    surveys of the top one or two law enforcement officials in each community
    will provide information on local law enforcement efforts, including the
    number of compliance checks Enforcing the Underage

    Drinking Laws Program

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 64 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    conducted in each year. A small number of youths from each site are to be
    selected at random for the surveys each year. The youth surveys will obtain
    data on perceptions of alcohol availability, peer and personal alcohol use,
    and alcohol- related problem behaviors including binge drinking and drunk
    driving. At the time of our review, this evaluation had received
    approximately $945,000.

    Evaluation findings: While some demographic data have been reported from the
    baseline survey, no results have been reported involving program effects.

    GAO assessment: The researchers suggest aggregating all program communities
    together and all comparison communities together to diminish community
    sample size problems, which may mask program effects. In addition, the wide
    variation allowed in program implementation may compromise the
    interpretation and generalizability of any findings.

    Program Description: The Intensive Aftercare program provides intensive
    supervision and services to serious juvenile offenders for 6 months
    following their release from secure confinement. The goal is to facilitate
    reintegration and reduce recidivism. The program was implemented, beginning
    in June of 1993, by various youth service offices and departments of
    corrections in four states: Colorado, Nevada, New Jersey, and Virginia. New
    Jersey was eventually dropped because of implementation problems, so the
    evaluation of the program is being completed in the other three states.

    Evaluator: National Council on Crime and Delinquency.

    Evaluation description: Beginning in 1995, youths entering correctional
    facilities in the three states (four counties in Colorado including
    Metropolitan Denver; Clark County, NV; and Norfolk County, VA) were screened
    for eligibility and randomly assigned, within each site, to the treatment
    group (whose members participated in the Intensive Aftercare program upon
    release) or control group. Between 1995 and 1999, 82 youths were assigned to
    the program and 68 to the control group in Colorado, 120 youths were
    assigned to the program and 127 to the control group in Nevada, and 75
    youths were assigned to the program and 45 to the control group in Virginia.
    Information was collected for study participants at baseline (that is, upon
    entry into the institution), before release from the institution (9 to 12
    months after baseline, or entry), immediately after completing the program
    (6 months after release), and 6 Intensive Aftercare

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 65 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    months after completing the program (12 months after release). The data
    collected, using survey instruments, standardized tests, monthly case
    management forms, and administrative (police and court) databases, included
    social and criminal history and demographic data, information on the extent
    of supervision and services, and the extent of criminal activity following
    institutional release. Many of the measures being employed in the study,
    according to the researchers, are standard and have been validated. At the
    time of our review, the evaluation had received approximately $932,000 in
    funding.

    Evaluation findings: The preliminary findings offered from this evaluation
    suggest that the Intensive Aftercare participants did receive greater
    supervision and more services after release than the control group, which
    suggests some success in implementing the program. Outcome results related
    to reintegration and recidivism are not complete, and the interim results
    are mixed as to whether the program is associated with positive outcomes.

    GAO assessment: This is a well- designed study, though serious missing data
    problems, if not corrected, may make it difficult to determine the outcome
    of this program.

    Program Description: JUMP was established by Part G of the Juvenile Justice
    and Delinquency Prevention Act of 1974, as amended in 1992. Through that
    legislation, the Congress authorized OJJDP to award 3- year grants to
    community- based not- for- profit organizations or to local educational
    agencies. The grantees are to support one- on- one mentoring projects that
    match volunteer adult mentors with youths at risk of delinquency, gang
    involvement, educational failure, and dropping out of school. The
    legislation also provided funding for a national, cross- site evaluation of
    JUMP. OJJDP guidelines emphasize the need for projects to recruit, train,
    supervise, and do thorough background checks for all volunteer mentors;
    develop procedures for appropriately matching youths and mentors; define the
    population of at- risk youths to be served; develop guidelines for the type,
    frequency, and duration of youth and mentor project activities; and
    establish procedures for gathering and reporting data to support the
    evaluation process. As of November 2000, 175 JUMP projects had been funded,
    in amounts ranging from $180,000 to $210,000 over a 3- year period.

    Evaluators: Information Technology International and Pacific Institute for
    Research and Evaluation. Juvenile Mentoring

    Program (JUMP)

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 66 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Evaluation description: This evaluation began in May 1997 and is expected to
    conclude on September 30, 2002. 4 Three approaches are being taken to
    determine how well JUMP is accomplishing its objectives. The first is a
    modified pre- post design that involves a within- subject comparison of the
    characteristics of youths at the time they enter and exit the program and
    between- subject comparisons of youths entering and exiting the program at
    the same time. The second approach is a best practices approach that will
    use structural equation models to estimate what program features or
    activities, including success in matching mentors and youths, are most
    likely to contribute to program success in reducing the risk of school and
    family problems, delinquency, and drug use among youths. The third approach
    relies on combined youth outcome data and community data to determine
    community cost offsets. The evaluation was funded at $3. 3 million.

    Evaluation findings: In their November 2000 JUMP annual report, the
    evaluators provided considerable descriptive information about the various
    JUMP projects, the characteristics of the youths and mentors, and
    information on youth- mentor matching. The only ?outcome? information thus
    far provided, however, is information on how satisfied youths and mentors
    were with the mentoring experience and how much benefit each perceived was
    derived from the experience. None of the three analytic approaches described
    above has been successfully applied to study outcomes because of a variety
    of pitfalls experienced by the national evaluation team, most notably
    insufficient data on school performance and behavioral measures (e. g.,
    delinquent behavior and arrests).

    GAO assessment: The researchers are employing multiple and innovative
    strategies to determine the effectiveness of JUMP in achieving its
    objectives. It is not clear, however, whether definitive evaluation results
    can be reached in the absence of outcome data on youths who, in the same
    project areas at the same points in time, do not receive the program. In
    addition, data limitations, if not corrected, may be serious enough to
    compromise findings.

    4 In commenting on a draft of this report, the Assistant Attorney General
    pointed out that the latest award sets project and budget end- dates of
    September 30, 2002. However, she expects the evaluation will not be
    completed before September 30, 2004. OJP does not move the project period
    forward beyond the budget period as this may imply an assurance of future
    funding.

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 67 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Program Description: The Partnerships to Reduce Juvenile Gun Violence
    Program is a multi- year demonstration program planned for four sites (Baton
    Rouge and Shreveport, LA; Syracuse, NY; and Oakland, CA). It began in 1997
    and is expected to conclude in 2001. However, one site, Shreveport, was
    dropped from the program early. The program aims to reduce youth gun
    violence by enhancing, in specific target areas of these cities, prevention
    and intervention strategies and strengthening partnerships among community
    residents, law enforcement agencies, and the juvenile justice system. The
    program involves mobilizing the community, establishing agency linkages, and
    planning case management for juveniles with gun charges in year 1, linking
    at- risk youths to services in year 2, and expanding opportunities for
    youths in year 3.

    Evaluator: COSMOS Corporation.

    Evaluation description: The strategy for evaluating the impact of this
    program has evolved as the program has unfolded. An impact evaluation was
    planned for three sites and was to include (1) a comparison of changes in
    crime rates in target areas of these cities before and after the
    implementation of the program, (2) a comparison of responses from highrisk
    youths in targeted areas surveyed before and after the program was
    implemented and services were provided, and (3) information on changes in
    policies and caseloads revealed through focus group meetings and interviews
    with agency officials. Crime rate information has thus far been reported
    only for Oakland and Baton Rouge, and surveys have been conducted only in
    Baton Rouge. In Baton Rouge, surveys were given to 92 high- risk youths in
    the criminal justice system identified through a variety of processes. The
    sampling strategies for surveying these high- risk youths were unlikely to
    yield generalizable results. In addition, fifth-, seventh-, and ninth- grade
    students in six schools in the target area were surveyed in March of 1999.
    It is unclear why these students and schools were sampled and what response
    rates were. In 2000, a small sample of 50 youths in Baton Rouge was
    identified as a possible matched comparison group for arrest rate
    comparisons. At the time of our review, this evaluation had received $1.2
    million in funding, although a process evaluation is also being conducted
    with these funds.

    Evaluation findings: The researchers report decreases in gun- related
    homicides and arrests in Oakland that were larger in the target area than
    for the city as a whole. They also report decreases in gun- related
    homicides in Baton Rouge. No analyses of results from the survey data have
    been reported to date. Partnerships to Reduce

    Juvenile Gun Violence Program

    Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

    Page 68 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    GAO assessment: Comparisons between crime rates in the target community and
    the city as a whole may not be appropriate. Student and school selection
    criteria are unclear, making it difficult to assess their appropriateness
    for obtaining definitive results. In addition, if supporting survey and
    administrative data are only gathered in one site, it will be very difficult
    to generalize findings whether they appear positive or not.

    The purpose of the Teen Courts evaluation is to measure the effect of
    handling young, relatively nonserious violators of the law in teen courts,
    rather than in traditional juvenile family courts. Although teen courts
    often include many of the same steps used by formal juvenile courts (for
    example, intake, preliminary review of charges, court hearing, and
    sentencing), they differ from formal courts in that young people are able to
    assist in the community decision- making process for dealing with juvenile
    offenders. Youths may act as prosecutors, defense counsel, jurors, court
    clerks, bailiffs, and judge (or as a panel of judges). To evaluate teen
    courts, both a process and impact evaluation are used, with case studies and
    comparison groups as part of the research design. In each of the four case
    study sites (Anchorage, AK; Independence, MO; Maricopa County, AZ; and
    Rockville, MD), data are collected on about 100 youths handled in teen
    courts (experimental group) and 100 youths handled in the traditional
    juvenile justice system (comparison group). Data are also collected on
    several dimensions of program outcomes, including post- program changes in
    teens? perceptions of justice and their ability to make more mature
    judgements as a result of the program. A process evaluation of the projects-
    exploring legal, administrative, and case- processing factors that hinder
    the achieving of project goals- is also being conducted. Impact Evaluation
    of

    OJJDP Program- Summary of Evaluation We Did Not Assess Teen Courts

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 69 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    This appendix contains summaries of the 24 evaluations the Office of
    Juvenile Justice and Delinquency Prevention (OJJDP) has funded since 1995
    (excluding the 11 impact evaluations it has funded of its own programs
    discussed in app. VII). For 22 summaries, we used descriptions of
    evaluations that were provided to us by OJJDP; for 2 summaries, 1 we wrote
    the descriptions based on OJJDP documents. OJJDP categorized the 24
    evaluations into the following three groups:

     OJJDP- funded programs: nonimpact evaluations (11).

     Non- OJJDP- funded programs: impact evaluations 2 (9).

     Non- OJJDP- funded programs: nonimpact evaluations (4). The purpose of
    this evaluation is to test the feasibility and effectiveness of the OJJDP
    community assessment center concept in different environments. Community
    assessment centers seek to facilitate earlier and more efficient delivery of
    prevention and intervention services at the front end of the juvenile
    justice system. The evaluation uses a two- phase process to (1) measure some
    outcomes at the two enhancement sites, with quasi- experimental design, 3
    and (2) achieve more and better outcome measures. But, according to OJJDP,
    implementation and data problems will limit the effectiveness of the
    quantitative methods employed. OJJDP also believes that the attempt to
    implement a random assignment study at one project site will probably need
    to be abandoned. The first phase covers the four project sites that
    comprised the Community Assessment Centers program- two of these sites
    funded enhancements to existing

    1 We used OJJDP documents to develop summaries for SafeFutures and Coping
    With Life Course. 2 Impact evaluations assess the extent to which a program
    causes changes in the desired direction in the target population. 3 In a
    quasi- experimental design, methods other than random assignment are used to
    create a comparison group. When randomly assigning individuals to a control
    group is not feasible, quasi- experimental impact evaluations can be used to
    compare the performance of program participants on various measures to
    individuals not in the program. Appendix VIII: Twenty- Four Other

    Evaluations Funded by OJJDP OJJDP- Funded Programs: 11 Nonimpact Evaluations

    Community Assessment Centers

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 70 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    programs and the other two funded the planning and implementation of new
    programs. The second phase covers the two project sites- one enhancement and
    one planning- in which the program is being continued after the end of the
    first funding cycle. 4 Many of the evaluation measures are at the project or
    community level rather than at the participant level.

    The purpose of this evaluation is to examine the viability and effectiveness
    of the community- based delinquency prevention model used by grantees in the
    Community Prevention Grants Program. The Community Prevention Grants Program
    encourages communities to develop comprehensive, collaborative plans to
    prevent delinquency. The evaluation focuses on two main questions: (1) What
    is the impact of the program on community planning, service delivery, risk
    factors, protective factors, and juvenile problem behaviors? (2) What
    factors and activities lead to the effective implementation of the Community
    Prevention Grants Program model and to positive program outcomes? This
    evaluation employs a case study approach supplemented by a basic profile of
    communities that are participating in the program. Case studies are to be
    implemented in 11 communities in 6 states. Evaluation measures are to be
    applied at the project, community, and program levels.

    The purpose of this process evaluation 5 is to address the following
    questions about the Comprehensive Strategy program: (1) What are the factors
    associated with successful Comprehensive Strategy planning and
    implementation? (2) To what extent do project sites adhere to the prescribed
    Comprehensive Strategy framework? (3) What are the major implementation
    challenges program grantees face in implementing the Comprehensive Strategy?
    (4) To what extent does the training and technical assistance provided to
    project sites help them acquire the knowledge, skill, and tools necessary to
    develop the Comprehensive Strategy? (5) What role should OJJDP play in the
    future implementation of the Comprehensive Strategy? The Comprehensive
    Strategy is OJJDP?s

    4 In commenting on a draft of this report, the Assistant Attorney General
    explained that the other two project sites were not included in the second
    phase of the evaluation because they did not receive additional funding. One
    site is no longer a separate project and the other project- a planning site-
    continues to move toward operational status.

    5 A process evaluation uses empirical data to assess delivery of a program
    and verifies whether it was delivered as intended to the targeted
    recipients. Community Prevention

    Grants Program (Title V) Comprehensive Strategy for Serious, Violent, and
    Chronic Juvenile Offenders (Comprehensive Strategy)

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 71 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    approach for addressing juvenile violence and delinquency at the community,
    state, and national levels through a systematic plan. It advocates the use
    of local planning teams to assess the factors and influences that put youths
    at risk of delinquency, determine available resources, and establish
    prevention programs to either reduce risk factors or provide protective
    factors that buffer juveniles from the impact of risk factors. This
    evaluation uses a multilevel design to assess how project sites implement
    the Comprehensive Strategy. The evaluation began with telephone interviews
    with site coordinators from all 48 project sites; 25 of the 48 project sites
    were randomly selected for a stakeholder survey. One year later, 10 of the
    25 project sites are being given a second stakeholder survey. Subsequently,
    five sites are to be selected for visits, and intensive case studies are
    being done in three cities.

    The purpose of this evaluation is to examine (1) community coalitions?
    developmental processes from the early planning and adoption stages through
    implementation and later stages and (2) the impact of coalitions? prevention
    efforts concerning risk and resiliency factors and, to the extent feasible,
    alcohol, tobacco, and other drug use. The Drug- Free Communities Support
    Program provides grants to community coalitions to strengthen their efforts
    to prevent and reduce young people?s illegal use of drugs, alcohol, and
    tobacco. The evaluation is studying two cohorts of program grantees- those
    that received grants in 1998 or 1999 (cohort 1) and those that received
    grants in 2000 (cohort 2). The national evaluation sample is comprised of a
    total of 213 grantees. The sample is divided by years of operation: 1- 5
    years, 6- 9 years, and more than 9 years. Semiannually, cohort 1 grantees
    are required to submit progress reports to OJJDP and the evaluator that
    include a special section (Part II), which provides information about the
    compositions of the coalitions and outcome data collection. Cohort 2
    grantees do not have a Part II reporting requirement and submit progress
    reports semiannually only to OJJDP. In addition, 21 grantees (15 from cohort
    1 and 6 from cohort 2) serve as intensive study sites, where interviews with
    staff and stakeholders provide greater detail about coalition development
    and local program evaluation.

    The purpose of this process evaluation is to provide feedback to OJJDP on
    the implementation of the Juvenile Accountability Incentive Block Grants
    Program. The program encourages states and local jurisdictions to implement
    accountability- based programs and services in 54 states and U. S.
    territories. The evaluator is surveying state and local practitioners,
    policy makers, and grant program administrators about their perceptions
    Drug- Free Communities

    Support Program Juvenile Accountability Incentive Block Grants Program

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 72 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    and attitudes about the program and its administration. Specifically, the
    evaluation focuses on (1) understanding how states and local units of
    government plan for and administer program funds and (2) examining the
    perceptions of states and local units of government about how well the
    program is achieving congressional intent. In addition, in- depth case
    studies are conducted at a limited number of sites.

    The purpose of this evaluation is to (1) provide feedback to the
    Performance- based Standards Project team on improving design and
    implementation support to the sites, (2) assist the project team in refining
    the Performance- based Standards Project model and in maximizing
    responsiveness to the needs of the participants, that is, those who are
    implementing the project model, and (3) chronicle the development of the
    project and summarize lessons learned. OJJDP established the Performance-
    based Standards Project to improve the quality and conditions of juvenile
    corrections facilities. Specifically, the project develops and implements
    outcome standards and an assessment tool. Corrections facilities can use
    both to monitor progress towards meetings goals in areas of operations, such
    as health and safety. The evaluation uses a case study approach. This
    approach consists of the collection of both quantitative and qualitative
    data describing the processes used to implement the project model in 80
    juvenile detention and correctional facilities across the country. Site
    visits are made and in- depth case studies are planned. An all- site survey
    is distributed to key participants to determine satisfaction with the
    supports provided to them in the implementation of the project model. In
    addition, the survey seeks the participants? assessment of (1) the impact
    the project has made on conditions of confinement and management of the
    facilities and (2) the overall utility of the project model.

    The purpose of this evaluation is to (1) determine the extent to which
    replication project sites have been able to conform to the original program
    model, and (2) assess the ?prosocial? (that is, positive, socially- oriented
    behavior) outcomes for mothers and their babies. The Prenatal and Early
    Childhood Nurse Home Visitation Program consists of intensive and
    comprehensive home visitation by nurses during a woman?s pregnancy and the
    first 2 years following the birth of her first child. The evaluation
    involves six project sites and employs a quasi- experimental design with
    matched comparison groups. Performance- based

    Standards Project Prenatal and Early Childhood Nurse Home Visitation Program

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 73 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    This purpose of this process evaluation is to document and understand the
    process of community mobilization and collaboration. SafeFutures is designed
    to build a comprehensive program of prevention and intervention strategies
    for at- risk youths and juvenile offenders. The program comprises six
    project sites that represent urban, rural, and Native American communities.
    The evaluation is examining all six project sites. Project sites collect and
    record performance data on program operations and client outcomes using the
    Client Indicator Data Base. The project sites are required to collect
    extensive information from selected SafeFutures program components on
    individual participants? risk and protective factor profiles, youths?
    service utilization, and agencies? coordination of services for youth during
    the course of their involvement in the SafeFutures program. In addition,
    they collect information on outcome measures regarding youths? educational
    commitment (that is, school attendance, achievement, and behavior), youths?
    involvement in delinquency and crime, and any changes in youths? risk
    profiles. Analysis of these data will provide a picture on program
    performance in three key areas: reaching the intended high- risk youth
    clientele, coordinating services for youths with multiple problems, and
    monitoring subsequent school performance problems and involvement in the
    juvenile justice system.

    The purpose of this evaluation is to document the lessons learned and
    factors associated with the successful development and implementation of the
    Safe Kids/ Safe Streets program. The Safe Kids/ Safe Streets program is
    designed to (1) help communities break the cycle of early childhood
    victimization and later criminality and (2) reduce child abuse and neglect,
    as well as the child fatalities that often result. The evaluation is
    surveying five project sites through five data collection strategies: agency
    administrative data, case tracking, key informant interviews, surveys of
    agency professionals, and surveys of stakeholders.

    The purpose of this evaluation is to (1) support culturally appropriate
    process and outcome evaluations of activities funded under Tribal Youth
    Program grants and (2) build the capacity of tribes to better evaluate their
    own juvenile justice programs and activities. The Tribal Youth Program
    assists grantees in developing projects, within tribal communities, for the
    prevention and control of youth violence and substance abuse. The evaluation
    is participatory in nature, that is, project personnel and stakeholders will
    be involved in developing the evaluation designs, with the assistance and
    guidance of an evaluation facilitator. The five project sites are
    implementing different projects and have not yet completed their
    Partnerships to Reduce

    Youth Violence and Delinquency Program (SafeFutures)

    Safe Kids/ Safe Streets Tribal Youth Program

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 74 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    evaluation designs. According to OJJDP, it is too early in the evaluation to
    tell exactly what designs are to be used. OJJDP has required that all
    evaluations be designed to examine both program implementation and program
    outcomes.

    The purpose of this process evaluation 6 is to (1) determine how community
    collaboration can affect truancy reduction and lead to systemic reform and
    (2) assist OJJDP in the development of a model for a truancy reduction
    program, including identifying the essential elements of that model. The
    Truancy Reduction Demonstration Program encourages communities to develop
    comprehensive approaches- involving schools, parents, the justice system,
    law enforcement, and social service agencies- in identifying and tracking
    truant youths. The evaluation is employing site visits, interviews with key
    personnel, and case studies of individual sites. Process data are gathered
    from all seven project sites participating in the evaluation and, from some
    sites, limited outcome data are gathered.

    The purpose of this evaluation is to evaluate the efficacy of three
    Adolescent Female Offenders programs in Wayne County, Michigan. The three
    programs are (1) a program incorporating gender- specific programming, home-
    based intervention, and community involvement, including pregnant and
    parenting adolescents; (2) an intensive probation program with limited
    gender- specific programming; and (3) a traditional, female- only
    residential program that provides limited gender- specific training. The
    evaluation is using a quasi- experimental design. Using random assignment,
    the home- based intervention model is to be compared with the established
    intensive probation model; the outcomes of these models are then to be
    compared with outcomes of the traditional,

    6 The Truancy Reduction Demonstration Program and its evaluation are jointly
    funded by OJJDP, the Office of Justice Program?s Executive Office of Weed
    and Seed, and the U. S. Department of Education. Truancy Reduction

    Demonstration Program Non- OJJDP- Funded Programs: Nine Impact Evaluations

    Adolescent Female Offenders (Three Programs)

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 75 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    female- only residential program. The comparison analysis involves at least
    50 young women in each of the 3 programs. A wide range of outcomes-
    including recidivism, substance use, depression, community integration,
    academic performance and career aspirations, parenting readiness, and
    responsible sexual behavior- is to be examined. The evaluator is also
    exploring the relationship of specific program components to these outcomes.

    The purpose of this impact evaluation is to evaluate the effectiveness of a
    cognitive- behavioral group intervention. The Coping With Life Course is
    aimed at enhancing prosocial coping and problem solving for adolescents
    incarcerated in youth correctional facilities. To evaluate the program, a
    minimum of 120 adolescents in one youth correctional facility are randomly
    assigned to either the Coping with Life Course intervention group or a
    standard- care control group. Six Coping with Life Course cohort groups of
    10 each are followed. The evaluation is allowing for attrition (from the
    initial 60 participants down to 48) in each of the intervention and control
    groups. Participant functioning is assessed before and after intervention
    through a battery of questionnaires. Recidivism, return to close custody,
    and service utilization are tracked through databases and statewide records.

    The purpose of the evaluation is to (1) document the implementation of a new
    ?family index? case management system (through a process evaluation) and (2)
    examine the impact of the family index on juvenile court case processing
    (through an impact evaluation). The family index system allows cross-
    referencing to identify all family members involved in family law; juvenile
    dependency; juvenile delinquency; and criminal, civil, and probate matters.
    For the process evaluation, a case study approach is used to describe the
    implementation of the family index at one project site, the Riverside,
    California, Court. For the impact evaluation, a pre- post design is used to
    examine how the family index has affected juvenile court matters (for
    example, court processing time, coordination between courts, and content of
    hearings).

    The purpose of this evaluation is to evaluate the effects of Flashpoint on
    the antisocial patterns of juvenile offenders? thoughts and actions and high
    school students? thoughts and actions. Specifically, changes are assessed
    for (1) media use and literacy, (2) violence- supporting beliefs and
    behavior, and (3) substance use and abuse. The Flashpoint program is Coping
    with Life Course

    Creation and Implementation of a Family Index in Riverside County,
    California, Courts

    Flashpoint

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 76 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    designed to build critical thinking skills young people need to (1) see
    through false media portrayals that glorify violence and drug use and (2)
    apply decision- making in their own lives. The evaluation is using
    comparison groups, pre- post interventions, and case studies. Participants
    include 264 juveniles, ages 14 to 17. Treatment groups are compared with no-
    treatment control groups for baseline- to- posttest changes. Three groups
    and project sites are involved: (1) repeat and serious offenders in a
    correctional program, (2) first- time offenders in a diversion program, and
    (3) students in a public high school.

    The purpose of this evaluation is to determine if Free to Grow can reduce
    substance abuse (alcohol use, smoking, and illegal drug use). The Free to
    Grow program builds on existing Head Start programs, adding
    communitystrengthening and family- intervention components to address the
    problem of substance abuse. The evaluation is attempting to determine the
    independent effects of these two components on substance abuse prevention.
    It involves 16 project sites and 16 comparison sites and employs a
    multistage experimental research design.

    The purpose of this evaluation is to provide a process and outcome
    evaluation of the Gaining Insight into Relationships for Lifelong Success
    Project. The project involves two primary levels of intervention: (1) a
    psycho- educational counseling group, dealing with relationships and
    involving girls in four relational domains (relation to self, family, peers,
    and teachers), and (2) a focus on individual consultations, educational
    workshops and the policies and procedures of the local juvenile justice
    system, and the involvement of court service workers from the system.
    Specifically, the evaluation will (1) investigate the applicability of a
    relational approach to the treatment of female juvenile offenders; (2)
    examine the components of the relational approach that deal with
    relationships to self, family, peers and teachers; (3) evaluate the impact
    of increasing the knowledgebase of professionals involved in the local
    juvenile justice system; and (4) provide an empirically based, alternative
    treatment model that can be replicated in other settings. The evaluation of
    the first level of intervention- the counseling group- focuses on each of
    the four relational domains through the use of multimethod data collection;
    this collection includes self- reports and other reports, school records,
    and recidivism data. The evaluation of the second level of intervention
    focuses on the court services that workers use, specifically gender-
    sensitive treatment recommendations and referrals; qualitative observational
    data, gathered from monthly meetings, will be used. There is Free to Grow:
    Head Start

    Partnerships to Promote Substance- Free Communities

    Gaining Insight Into Relationships for Lifelong Success Project

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 77 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    random assignment between girls referred to either the project intervention
    or to the standard intervention currently being used by the Clark County
    Court in Athens, Georgia. Approximately 180 girls- 90 referred to the
    project intervention and 90 referred to the standard court intervention- are
    to be evaluated.

    The purpose of this evaluation is to assess the implementation and the
    impact of Quantum Opportunities. The Quantum Opportunities program is
    designed to reduce the incidence of delinquency, criminal behavior, and
    subsequent involvement in the criminal and juvenile justice systems amongst
    educationally at- risk inner city youths. The evaluation is using an
    experimental design with random assignment. Ninth- grade students at six
    sites are randomly assigned to treatment and control groups, with the
    treatment group enrolled in the Quantum Opportunities program. The students
    are followed through their high school careers and 2 years beyond.
    Information is collected from academic achievement tests, administered each
    year, and from two questionnaires.

    The purpose of this evaluation is to test the impact- on recidivism, program
    completion, and victim satisfaction- of the Restorative Justice Conferences
    for a population of youthful offenders (aged 14 and under) in an urban
    setting (Indianapolis, IN). Restorative Justice Conferences bring together
    the offender, victim, and supporters of each so as to provide an opportunity
    for fuller discussion of the offense; the effect of the offense on the
    victim, the offender?s family, and greater community; and steps the offender
    can take to make amends. The evaluation is using a single- site evaluation
    with an experimental design. As part of the design, youths are randomly
    assigned to a treatment group (Restorative Justice Conferences) or a matched
    control group.

    The purpose of this evaluation is to determine if the program reduces the
    amount of delinquency in a city. The Risk- Focused Community Policing
    program increases protection by the community police, potentially reducing
    delinquency. The evaluation is using an experimental research design. The
    project site (city) is divided into approximately 40 census blocks, with 20
    blocks randomly selected as program blocks and the other 20 designated as
    control blocks. Quantum Opportunities

    Restorative Justice Conferences

    Risk- Focused Community Policing

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 78 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    The purpose of this evaluation is to study the Act Now Truancy Program. The
    program is a prosecutor- led truancy reduction program. The evaluation is
    using a pre- post intervention design involving one project site.
    Information is collected and aggregated (for example, truancy rates rather
    than individual truancy behavior) for all participants.

    The purpose of this evaluation, conducted in two schools, is to assess the
    impact of the Childhood Violence Prevention Program. The program is designed
    to prevent the legitimization of aggression among pre- adolescent,
    elementary, and middle school children, with special focus on victims of
    child maltreatment. The evaluation is using a pre- post intervention design,
    with comparison groups. The study involves having elementary school students
    participate in a class activity using a workbook designed to encourage
    problem solving action rather than aggressive behavior in interactions with
    peers.

    This project is not an evaluation, per se, but rather a synthesis of
    existing evidence on community- level interventions and service programs.
    Its purpose is to identify the strengths and weaknesses of community- level
    evaluations and to provide recommendations to the field about how to
    structure and carry out such evaluations. Community- level programs for
    youths are designed to promote positive youth development. To evaluate the
    programs, a committee- experts from several disciplines (child and
    adolescent development, child health, sociology, psychology, evaluation
    research, youth services, and community development)- is assessing the
    strengths and limitations of measurements and methodologies that have been
    used to evaluate these interventions. Non- OJJDP- Funded

    Programs: Four Nonimpact Evaluations

    Act Now Truancy Program Childhood Violence Prevention Program (Let?s Talk
    About...)

    Community- Level Programs for Youth

    Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP

    Page 79 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce

    The purpose of this evaluation is to assess the activities undertaken by
    project sites, determine whether they can be evaluated, and ultimately
    assess the impact of these activities on the youthful offenders
    participating in the program. The program is intended to (1) enhance school-
    to- work education and training in juvenile correctional facilities and (2)
    improve youthful offenders? transition into the community. The evaluation
    design has not been completed, but random assignment study is strongly
    preferred, if feasible. At the time of our review, only one of the three
    potential sites could be evaluated. One more project site is to be awarded
    and, if it can be evaluated, it will be added as a second evaluation site.
    Department of Labor?s

    Education and Training for Youthful Offenders Initiative

    Appendix IX: Comments from the Department of Justice

    Page 80 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Appendix IX: Comments from the Department of Justice

    Note: GAO comments supplementing those in the report text appear at the end
    of this appendix.

    Appendix IX: Comments from the Department of Justice

    Page 81 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now on pp. 23- 24. Now on pp. 3, 8- 10, and

    22- 25.

    Appendix IX: Comments from the Department of Justice

    Page 82 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Appendix IX: Comments from the Department of Justice

    Page 83 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now on p. 10. Now on p. 9.

    Now on p. 6. Now on p. 4. Now on p. 21.

    Appendix IX: Comments from the Department of Justice

    Page 84 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    See comment 1. Now on p. 22. Now on pp. 10- 18.

    Appendix IX: Comments from the Department of Justice

    Page 85 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Appendix IX: Comments from the Department of Justice

    Page 86 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now on p. 25.

    Appendix IX: Comments from the Department of Justice

    Page 87 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now on p. 31. Now on p. 26. Now on p. 26. Now on pp. 25- 26.

    See comment 2. Now on p. 25.

    Appendix IX: Comments from the Department of Justice

    Page 88 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Appendix IX: Comments from the Department of Justice

    Page 89 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now on pp. 48- 51 Now on p. 48.

    Now on p. 36, table 6. Now on p. 35. Now on p. 35.

    Appendix IX: Comments from the Department of Justice

    Page 90 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now on p. 66. Now on p. 57- 58.

    Now on p. 59. Now on p. 53, tables 10

    and 11. See comment 3.

    Now on p. 51. Now on p. 51.

    Appendix IX: Comments from the Department of Justice

    Page 91 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now on p. 73. Now on p. 71.

    Now on p. 70. Now on p. 64.

    See comment 4.

    Appendix IX: Comments from the Department of Justice

    Page 92 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Now in app. VII, p. 68.

    Appendix IX: Comments from the Department of Justice

    Page 93 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    The following are GAO?s comments on the Department of Justice?s October 15,
    2001, letter.

    1. As we indicated in our report, impact evaluations, such as the types that
    OJJDP is funding, can encounter difficult design and implementation
    challenges. (See section titled, Evaluations of OJJDP Programs are Difficult
    to Successfully Design and Implement.) Also, we are aware that virtually all
    impact evaluations have limitations. However, where possible, impact
    evaluations should be designed to mitigate as many rival explanations of
    program effects as feasible, and potential limitations of the chosen
    research design should be acknowledged.

    2. Our statement that the Juvenile Mentoring Program evaluation ?has

    experienced problems obtaining behavioral measures and school performance
    data? was not intended to criticize the evaluators? level of effort, but
    rather to indicate that their inability to obtain data from school and law
    enforcement officials in many of the study sites makes it more difficult to
    evaluate how well the program is achieving its objectives of diminishing
    delinquency, gang involvement, and school failure. While enhanced analysis
    of sites with the best data may be warranted, it does not overcome the
    problem of having a large number of sites with little or no reliable data
    from school and law enforcement officials. This problem was explicitly
    recognized by OJJDP in its November 2000 report.

    3. During the course of our review, OJJDP officials told us that one measure
    of a research grantee?s performance is the number of products the grantee
    has published. These officials provided us a listing of all products
    published by active research grantees through OJJDP and private publishers
    as a direct result of OJJDP- funded research. We summarized these voluminous
    data by topic to facilitate the presentation.

    4. Our report points out that the Enforcing the Underage Drinking Laws
    Program evaluation documents OJJDP provided to us were not clear on whether
    the sites chosen were representative. Our report does not suggest ?there is
    no way of achieving a legitimate representative sample.? However, we agree
    with OJJDP that the evaluation may not be evaluating the Enforcing the
    Underage Drinking Laws Program because there may be no program components
    common to all project cities. The Assistant Attorney General states that the
    evaluation will be GAO Comments

    Appendix IX: Comments from the Department of Justice

    Page 94 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    able to measure impacts on several program areas across each site. However,
    our point is that the evaluator?s plan to aggregate data across sites may be
    inappropriate because wide variation allowed by the program means that
    program activities are not common across all sites. Therefore, interpreting
    and generalizing results may be problematic.

    Appendix X: GAO Contacts and Staff Acknowledgments

    Page 95 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns

    Laurie E. Ekstrand, (202) 512- 8777 James M. Blume, (202) 512- 8777

    In addition to the above, Lori A. Weiss, Barbara A. Guffy, Michele J. Tong,
    Leslie C. Bharadwaja, David P. Alexander, Douglas M. Sloane, Shana B.
    Wallace, Michele C. Fejfar, Charity J. Goodman, and Jerome T. Sandau made
    key contributions to this report. Appendix X: GAO Contacts and Staff

    Acknowledgments GAO Contacts Acknowledgments

    Bibliography Page 96 GAO- 02- 23 OJJDP Grantee Reporting Requirements and
    Concerns

    Howell, J. C. Youth Gang Programs and Strategies. U. S. Department of
    Justice, Office of Justice Programs, Office of Juvenile Justice and
    Delinquency Prevention. Washington, D. C.: Aug. 2000.

    Novotney, L. C., E. Mertinko, J. Lange, and T. K. Baker. ?Juvenile

    Mentoring Program: A Progress Review.? Juvenile Justice Bulletin, Sept.
    2000.

    Sheppard, D., H. Grant, W. Rowe, and N. Jacobs. ?Fighting Juvenile Gun
    Violence.? Juvenile Justice Bulletin, Sept. 2000.

    U. S. Department of Justice, Office of Justice Programs, National Institute
    of Justice. ?Reintegrating Juvenile Offenders Into the Community: OJJDP?s
    Intensive Community- Based Aftercare Demonstration Program.? Research
    Preview, Dec. 1998.

    U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
    Justice and Delinquency Prevention. FY 2000 OJJDP Discretionary Program
    Announcement: Juvenile Mentoring Program. Mar. 2000.

    U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
    Justice and Delinquency Prevention. Gang- Free Schools and Communities
    Initiative: FY 2000 OJJDP Discretionary Program Announcement. July 2000.

    U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
    Justice and Delinquency Prevention. Juvenile Mentoring Program: 1998 Report
    to the Congress. Washington, D. C.: Dec. 1998.

    U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
    Justice and Delinquency Prevention. OJJDP Research 2000. Washington, D. C.:
    May 2001.

    Wiebush, R. G., B. McNulty, and T. Le. ?Implementation of the Intensive
    Community- Based Aftercare Program.? Juvenile Justice Bulletin, July 2000.
    Bibliography

    (185012)

    The first copy of each GAO report is free. Additional copies of reports are
    $2 each. A check or money order should be made out to the Superintendent of
    Documents. VISA and MasterCard credit cards are also accepted.

    Orders for 100 or more copies to be mailed to a single address are
    discounted 25 percent.

    Orders by mail:

    U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013

    Orders by visiting:

    Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
    20013

    Orders by phone:

    (202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537

    Each day, GAO issues a list of newly available reports and testimony. To
    receive facsimile copies of the daily list or any list from the past 30
    days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
    will provide information on how to obtain these lists.

    Orders by Internet

    For information on how to access GAO reports on the Internet, send an email
    message with ?info? in the body to:

    Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
    gao. gov

    Contact one:

     Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm

     E- mail: fraudnet@ gao. gov

     1- 800- 424- 5454 (automated answering system) Ordering Information

    To Report Fraud, Waste, and Abuse in Federal Programs
    *** End of document. ***