Juvenile Justice: OJJDP Reporting Requirements for Discretionary
and Formula Grantees and Concerns About Evaluation Studies
(30-OCT-01, GAO-02-23).
Although national rates of violent juvenile crime and youth
victimization have declined during the past five years, critical
problems affecting juveniles, such as drug dependency, the spread
of gangs, and child abuse and neglect, persist. The Office of
Juvenile Justice and Delinquency Prevention (OJJDP) has funded
various demonstration, replication, research and evaluation, and
training and technical assistance programs to prevent and respond
to juvenile delinquency and juvenile victimization. GAO's review
of 16 of OJJDP's major programs showed that although virtually
all grantees must report on their progress twice a year, the
information they reported varied. Grantees receive standard,
general guidance for reporting on their projects and provide
OJJDP information to monitor grantee's projects and
accomplishments. According to OJJDP officials, such guidance
needs to be general, because of differences among individual
projects and local needs and circumstances. GAO identified eight
programs in which all grantees reported the number of juveniles
they directly served. OJJDP does not require grantees in all its
programs to report directly on the number of juveniles served
directly because many of its programs are not intended to serve
juveniles directly. GAO's in-depth review of OJJDP's 10 impact
evaluations undertaken since 1995 raises some concerns about
whether the evaluations will produce definitive results. In some
of these evaluations, variations in how the programs are
implemented across sites make it difficult to interpret
evaluation results.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-23
ACCNO: A02064
TITLE: Juvenile Justice: OJJDP Reporting Requirements for
Discretionary and Formula Grantees and Concerns About Evaluation
Studies
DATE: 10/30/2001
SUBJECT: Crimes or offenses
Grant monitoring
Juvenile offender rehabilitation
Juvenile status offenders
Reporting requirements
Act Now Truancy Program
Child Abuse and Neglect Program
Children's Advocacy Centers Program
Community Prevention Grants Program
Comprehensive Gang Initiative
Court Appointed Special Advocate Program
DOJ Juvenile Mentoring Program
Drug-Free Communities Support Program
Enforcing the Underage Drinking Laws
Program
Formula Grants Program
Internet Crimes Against Children Task
Force Program
Juvenile Accountability Incentive Block
Grants Program
Missing and Exploited Children Program
OJJDP Model Courts Program
Partnerships to Reduce Juvenile Gun
Violence Program
Positive Action Through Holistic
Education Program
Rural Gang Initiative
Safe Futures Program
Tribal Youth Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-23
Report to the Honorable Bob Schaffer, House of Representatives
United States General Accounting Office
GAO
October 2001 JUVENILE JUSTICE OJJDP Reporting Requirements for Discretionary
and Formula Grantees and Concerns About Evaluation Studies
GAO- 02- 23
Page i GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns Letter
1
Results in Brief 2 Background 4 Scope and Methodology 7 All Grantees Receive
Standard Reporting Guidance and Some Are
Required to Report Specific Data 10 Some Data Exist on the Number of
Juveniles Served Directly,
Although Most Grantees Are Not Required to Report This Information 18
Methodological Concerns Could Adversely Affect Evaluation
Studies 22 OJJDP Has Funded Other Types of Evaluations 26 Conclusions 26
Recommendation for Executive Action 27 Agency Comments and Our Evaluation 27
Appendix I OJJDP Awards Data, Fiscal Years 1996 Through 2000 30
Appendix II OJJDP?s Process for Disseminating Published Products, Including
Interim Results of Impact Evaluations 35
Appendix III Categorical Assistance Progress Report Form 37
Appendix IV Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information 39
Appendix V Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees 48
Performance Data for Training and Technical Assistance Grantees 48
Performance Data for Research Grantees 51 Contents
Page ii GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Appendix VI States? Compliance With Core Requirements of the
Juvenile Justice and Delinquency Prevention Act of 1974 55
Appendix VII Impact Evaluations OJJDP Has Funded of Its Own Programs 56
Impact Evaluations of OJJDP Programs in Their Formative Stages 56 Impact
Evaluations of OJJDP Programs Well Into Their
Implementation 62 Impact Evaluation of OJJDP Program- Summary of Evaluation
We
Did Not Assess 68
Appendix VIII Twenty- Four Other Evaluations Funded by OJJDP 69 OJJDP-
Funded Programs: 11 Nonimpact Evaluations 69 Non- OJJDP- Funded Programs:
Nine Impact Evaluations 74 Non- OJJDP- Funded Programs: Four Nonimpact
Evaluations 78
Appendix IX Comments from the Department of Justice 80 GAO Comments 93
Appendix X GAO Contacts and Staff Acknowledgments 95 GAO Contacts 95
Acknowledgments 95
Bibliography 96
Tables
Table 1: Reasons for Reporting Requirements for 16 Selected Major OJJDP
Programs 11 Table 2: OJJDP Programs GAO Identified in Which All Grantees
Reported Data on the Number of Juveniles Directly Served, Fiscal Year 2000
19 Table 3: OJJDP Funds Awarded by Program Area, Fiscal Years
1996 Through 2000 31 Table 4: OJJDP Award Recipients, Fiscal Years 1996
Through 2000 32
Page iii GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 5: OJJDP Formula/ Block Grant Awards, by State, Fiscal Years 1996
Through 2000 33 Table 6: Dissemination of Published Products With Interim
Results
of Impact Evaluations, as of May 2001 36 Table 7: Eight Programs GAO
Reviewed With General Reporting
Requirements Only 40 Table 8: Eight Programs GAO Reviewed With Specific
Reporting
Requirements and General Reporting Requirements 44 Table 9: Training and
Technical Assistance Performance Data
Reported by OJJDP Grantees 50 Table 10: Active OJJDP Grantees? Research
Products, Published by
OJJDP 53 Table 11: Active OJJDP Grantees? Research Products, Published
Externally 53 Table 12: State- Reported Compliance with Core Requirements,
Calendar Year 1998 55
Figures
Figure 1: OJJDP Award Recipients, Fiscal Year 2000 7 Figure 2: OJJDP
Formula/ Block Grants Versus Discretionary Grant
Awards, Fiscal Years 1996 Through 2000 30
Abbreviations
CPD Child Protection Division IPR Individual Project Report JUMP Juvenile
Mentoring Program NCJRS National Criminal Justice Reference Service NCMEC
National Center for Missing and Exploited Children OJJDP Office of Juvenile
Justice and Delinquency Prevention OJP Office of Justice Programs PATHE
Positive Action Through Holistic Education RPDD Research and Program
Development Division STAD State and Tribal Assistance Division TTAD Training
and Technical Assistance Division
Page 1 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
October 30, 2001 The Honorable Bob Schaffer House of Representatives
Dear Mr. Schaffer: Although the nation?s rates of violent juvenile crime and
youth victimization have declined dramatically over the past 5 years,
critical problems affecting juveniles and the juvenile justice system still
remain, such as drug dependency, the spread of gangs, and child abuse and
neglect. To address these and other issues, the Office of Juvenile Justice
and Delinquency Prevention (OJJDP) has funded a variety of demonstration,
replication, research and evaluation, and training and technical assistance
programs aimed at preventing and responding to juvenile delinquency and
juvenile victimization. Questions have been raised, however, regarding what
these numerous and diverse programs have accomplished.
You asked us to review OJJDP?s major programs, as well as the evaluations it
has funded. Specifically, we agreed to provide (1) information on
programmatic reporting requirements for OJJDP grantees, the reasons for
these requirements, and examples of the information grantees have reported;
(2) information on how many juveniles OJJDP grantees reported directly
serving in fiscal year 2000, and whether OJJDP requires grantees to report
the number of juveniles they directly serve, and if not, why; and (3)
analysis of the methodological rigor of the impact evaluations OJJDP has
funded of its own programs since 1995, and information on the other types of
evaluations OJJDP has funded. In addition, you asked us to provide
information on how much OJJDP awarded to various grant programs from fiscal
years 1996 through 2000 and the types of organizations that received these
awards, and we have provided this information in appendix I. You asked us
how OJJDP disseminates published interim results of impact evaluations as
well as other published publications OJJDP and its grantees produce, and we
have provided this information in appendix II. You also asked us to review
how OJJDP monitors its grantees, and we have reported to you on this issue.
1
1 Juvenile Justice: Better Documentation of Discretionary Grant Monitoring
Is Needed
(GAO- 02- 65, Oct. 10, 2001).
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Our review of 16 of OJJDP?s major programs showed that although virtually
all grantees are required to report on their progress twice a year, the
information they reported varied. Grantees receive standard, general
guidance for reporting on their projects and providing OJJDP information
with which to monitor grantees? projects and accomplishments. According to
OJJDP officials, such guidance needs to be somewhat general, given the
variation that can occur among individual projects as grantees try to meet
local needs and circumstances. For example, one Drug- Free Communities
Support Program grantee reported that it had helped students produce an
anti- smoking commercial, while another grantee in the same program reported
making presentations on drug abuse to young men at the local juvenile
detention facility. Grantees in 8 of these 16 programs are subject to
additional reporting requirements tailored to each of their programs, and
these specific requirements were established primarily to support outside
evaluations or for other program or project assessment purposes. For
example, OJJDP requires all Juvenile Mentoring Program grantees to cooperate
fully with an outside evaluator. Grantees in this program are required to
report prescribed demographic information for both juveniles and mentors who
participate in their programs, as well as other specified data.
We identified eight programs in which all grantees reported the number of
juveniles they directly served. Grantees in these eight programs reported
serving about 142,000 juveniles in fiscal year 2000. 2 For example, Court
Appointed Special Advocate Program grantees reported serving over 70,000
juveniles through volunteers who advocate for the best interests of abused
and neglected children who come before the court. Juvenile Mentoring Program
grantees reported serving about 8, 500 juveniles by providing one- on- one
mentoring for at- risk youths. OJJDP does not require grantees in all its
programs to report on the number of juveniles they serve directly for
several reasons, including that many of its programs are not intended to
serve juveniles directly. Instead, OJJDP programs have a variety of other
purposes that can indirectly benefit juveniles, such as helping state and
local governments improve their juvenile justice systems, providing training
and technical assistance to juvenile justice professionals, and establishing
a centralized research effort on problems of juvenile delinquency. In
commenting on a draft of this report, the Assistant Attorney General stated
that research projects do not typically
2 Grantees in three of the eight programs reported data for time periods
that did not correspond precisely with the fiscal year. Results in Brief
Page 3 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
provide services directly, yet their results can potentially help thousands
of juveniles.
Our in- depth review of 10 of OJJDP?s impact evaluations 3 undertaken since
1995 of its own programs raises some concerns about whether many of the
evaluations will produce definitive results. All of these evaluations are
still ongoing, with half in their formative stages and half well into
implementation. While program evaluation is an inherently difficult task, in
some of these evaluations, the effort is particularly arduous because of
variations across sites in how the programs are implemented. These
variations will make it difficult to interpret evaluation results. Two of
the evaluations that are in their later stages and three of those that are
in their formative stages lack specific plans for comparison groups, which
would aid in isolating the impacts of the program from the effects of other
factors that may have influenced change. Furthermore, three of the five
evaluations that are well into implementation have developed data collection
problems. We are making a recommendation in relation to the five evaluations
that are in their formative stages, intending to ensure that potential
problems related to comparison groups and data collection are mitigated.
The Assistant Attorney General provided us with written comments on a draft
of this report in a letter dated October 15, 2001. In general, she agreed
with our findings and recommendation. However, she disagreed with our focus
on the use of comparison groups as the only valid evaluation design.
Although she agreed that the inclusion of comparison groups would strengthen
the interpretation of evaluation results, she said that sufficient funds are
not available to include comparison groups in every evaluation and that
laboratory- like comparison groups may not be possible when evaluating real
life community- based programs. We recognize that not all evaluation issues
that can compromise results are resolvable, including the use of comparison
groups. However, the validity of the evaluation results can be enhanced
through establishing and tracking comparison groups. If other ways exist to
effectively isolate the impacts of a program, comparison groups may not be
needed. However, in the 10 evaluations for which we assessed methodological
rigor, we saw no evidence of other methods being used. While studies that do
not have appropriate comparison groups can provide useful information, they
3 Impact evaluations assess the extent to which a program causes changes in
the desired direction in the target population.
Page 4 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
should not be considered impact evaluations. The full text of the Assistant
Attorney General?s comments and our evaluation of them are presented in
appendix IX and elsewhere in this report, as appropriate.
OJJDP, one of the components of the U. S. Department of Justice, Office of
Justice Programs (OJP), was established by the Juvenile Justice and
Delinquency Prevention Act of 1974 (Juvenile Justice Act). 4 Its mission is
to provide national leadership, coordination, and resources to prevent and
respond appropriately to juvenile delinquency and juvenile victimization.
OJJDP accomplishes its mission through developing and implementing
prevention programs and a juvenile justice system that protects the public
safety, holds juvenile offenders accountable, and provides treatment and
rehabilitative services based on the needs of juveniles and their families.
OJJDP funds research and evaluation efforts, statistical studies, and
demonstration programs; provides technical assistance and training; produces
and distributes publications and other products containing information about
juvenile justice topics; oversees activities dealing with missing and
exploited children; and administers a wide variety of grant programs. OJJDP
funds programs that serve juveniles directly as well as those that benefit
juveniles more indirectly by focusing on system- wide changes or by
increasing the capacity of governmental units or organizations.
OJJDP awards grants to states, territories, localities, and private
organizations through five formula and block grant (formula/ block grant)
programs and numerous discretionary grant programs. OJJDP administers four
formula grant programs 5 that provide funds directly to states and
territories on the basis of states? juvenile populations, and one block
grant program 6 that awards a fixed level of funds to all states and
territories. Under these formula/ block grant programs, states may, in turn,
make subawards to other organizations such as units of local government.
OJJDP awards discretionary grants through a competitive process to state
4 42 U. S. C. 5601, et seq. 5 OJJDP?s four formula grant programs are the
Formula Grants Program, the Community Prevention Grants Program, State
Challenge Activities, and the Juvenile Accountability Incentive Block Grants
Program.
6 OJJDP?s one block grant program is the Enforcing the Underage Drinking
Laws Program. The Enforcing the Underage Drinking Laws Program has both a
block grant component and a discretionary grant component. Background
Page 5 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
governments, local governments, or individual agencies and organizations.
OJP is responsible for the financial monitoring 7 of grantees (i. e., it
provides policy guidance, control, and support services in the financial
management of grants), whereas OJJDP is responsible for program monitoring.
For program monitoring purposes, OJJDP assigns each of its grantees a
program manager who is responsible for ensuring administrative and
programmatic compliance with relevant statutes, regulations, policies and
guidelines of awarded grants. The program manager is also responsible for
monitoring grantees? performance and progress as related to grantees? stated
goals and objectives.
OJJDP?s budget has increased significantly over the last 5 years- from about
$188 million in fiscal year 1997 to about $596 million in fiscal year 2001.
During this time, the Congress has created new programs and increased
appropriations for some existing ones. The Congress has also provided
direction each year regarding certain program areas OJJDP should fund.
Overall, in fiscal year 2001, 31 percent of OJJDP?s available funds 8 were
congressionally earmarked- that is, set aside for an identifiable grantee,
specified amount, and/ or specific authorized purpose. For fiscal year 2001,
$180 million of OJJDP?s funds were available for discretionary grant awards,
of which 77 percent was earmarked.
OJJDP awards the majority of its funds to grantees in its five formula/
block grant programs. In fiscal year 2000, the latest year for which awards
data were available, OJJDP awarded (1) about $354 million, or 64 percent of
the total funds awarded, to these formula/ block grant programs and (2) just
over $200 million, or 36 percent of the total funds awarded, to a wide range
of discretionary grant programs. The programs awarded the most funds in
fiscal year 2000 were the Juvenile Accountability Incentive Block Grants
Program ($ 221 million), the Formula Grants Program ($ 70 million), the
Community Prevention Grants Program ($ 36 million), the Child Abuse and
Neglect Program 9 ($ 32 million), the Missing and
7 OJP conducts financial monitoring of grantees through the Monitoring
Division of its Office of the Comptroller. 8 Of OJJDP?s $597 million dollar
appropriation, only $583 million were ?available;? about $12. 7 million were
designated to support OJJDP administrative costs and $1. 3 million (. 0022)
was rescinded.
9 This program encompasses several other programs within OJJDP?s Child
Protection Division including the Court Appointed Special Advocate Program,
Children?s Advocacy Centers, the Model Courts Program, and the Safe Start
Initiative. OJJDP?s Fiscal Year 2000
Awards
Page 6 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Exploited Children Program 10 ($ 32 million), and the Drug- Free Communities
Support Program ($ 30 million).
OJJDP awards funds to a wide range of recipients, with the majority of
awarded funds going to state governments. As shown in figure 1, 67 percent
of the funds OJJDP awarded in fiscal year 2000 went to states, 20 percent to
nonprofit organizations, 6 percent to school districts or educational
institutions, and 5 percent to local governments. However, because many
grantees make subawards to other entities, the awards to these grantees do
not reflect the ultimate recipients of the funds OJJDP awards. For example,
under the Formula Grants Program, states pass through a minimum of two-
thirds of their awarded funds to public and private nonprofit organizations.
(See app. I for data from fiscal years 1996 through 2000 on (1) OJJDP funds
awarded to formula/ block grant versus discretionary grant programs, (2)
OJJDP awards by program area, (3) OJJDP award recipients, and (4) OJJDP
formula/ block grant awards by state.)
10 This program encompasses several other programs within OJJDP?s Child
Protection Division including the National Clearinghouse and Resource Center
for Missing and Exploited Children, the Missing and Exploited Children
Training and Technical Assistance program, and the Internet Crimes Against
Children Task Force Program.
Page 7 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Figure 1: OJJDP Award Recipients, Fiscal Year 2000
Note: Percentages do not total 100 due to rounding. a Local governments
include county governments, cities, townships, and special district
governments.
b Schools include independent school districts and state and private
institutions of higher education. c Other includes for- profit
organizations, individuals, Indian Tribes, and other miscellaneous
recipients.
Source: GAO analysis of OJP database.
To identify programmatic reporting requirements 11 for OJJDP grantees, the
reasons for these requirements, and examples of information grantees have
reported, we reviewed all 5 of OJJDP?s formula/ block grant programs and we
selected 11 of its major discretionary grant programs to review based on
OJJDP officials? input regarding which programs were ?major?
(e. g., number of grantees, program funding, and/ or importance of the
program). To identify grantee reporting requirements, and the purpose of
these requirements, we met with the OJJDP program managers who monitor each
of these 16 programs or with other key officials. For those
11 We did not review requirements associated with OJP?s financial monitoring
of OJJDP grantees. Scope and
Methodology
State Governments
67%
Nonprofit Organizations
20%
Other c Schools
6%
b Local Governments a 5%
3%
Page 8 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
programs in which OJJDP funded an outside evaluation, we also met with
program managers who oversee the evaluations. We reviewed OJP?s Grant
Management Policies and Procedures Manual (January 19, 2001) and Categorical
Assistance Progress Report (progress report) form 12 along with the
instructions for completing the form. (See app. III for a copy of the
progress report form.) We also reviewed OJJDP program documents for each of
the 16 programs, including any special reporting conditions. To supplement
program documents, we reviewed relevant documents from outside evaluators.
We did not assess the adequacy of reporting requirements established by
OJJDP or the outside evaluators.
To identify specific examples of performance data that grantees reported, we
asked OJJDP officials to provide progress reports for each program
demonstrating a range of detail and, in some cases, we asked for reports
from specific grantees. For each program, we then reviewed 3 to 15 progress
reports (or individual performance reports) submitted between 1998 and 2001.
We did not review progress reports from all grantees in every program, nor
did we review grantees? compliance with reporting requirements. For programs
being evaluated by an outside evaluator, we reviewed performance data that
program grantees reported to those evaluators, when available.
To determine whether OJJDP requires grantees to report the number of
juveniles they serve directly and to identify the number of juveniles OJJDP
grantees reported serving in fiscal year 2000, we interviewed OJJDP program
managers for each of the 16 programs we reviewed and examined relevant
program documents, including selected semiannual progress reports. To
determine whether other programs directly serve juveniles, obtain available
data on the numbers of juveniles served, and learn why OJJDP does or does
not require grantees to report such data, we reviewed OJJDP program
literature and met with OJJDP division directors. Nevertheless, the list of
OJJDP programs we identified as directly serving juveniles may not be
comprehensive. We focused our data collection effort on only those programs
we identified in which all grantees reported juveniles- served data.
12 For the Formula Grants Program, we reviewed OJJDP?s instructions for
completing the annual Performance Report, including the Individual Project
Report forms, because Formula Grants grantees are required to complete these
reports in lieu of progress reports.
Page 9 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
To assess the methodological rigor of the impact evaluations OJJDP has
funded since 1995 of its own programs, and to provide information on the
other types of evaluations OJJDP has funded, we asked OJJDP to identify all
program evaluations it had funded since 1995. For each evaluation, we asked
OJJDP to indicate whether it was an impact evaluation, whether the program
being evaluated was funded by OJJDP, and whether the evaluation had been
completed or was ongoing. Overall, OJJDP identified 35 evaluations funded
since 1995.
Eleven of the 35 evaluations were impact evaluations of OJJDP programs, and
all were ongoing. 13 For each of the 10 impact evaluations we assessed, we
asked OJJDP to provide any documentation relevant to the design and
implementation of the evaluation methodologies, such as the initial and
supplemental proposals, peer review documents, progress reports, reports of
interim results, and correspondence between OJJDP and the evaluators. In
addition, we contacted OJJDP officials to resolve any questions that we had
regarding the documentation and to request any missing documents. We did not
contact the program manager responsible for each evaluation.
To assess the methodological rigor of the 10 impact evaluations, we used a
data collection instrument to collect information systematically on each
program being evaluated and the features of the evaluation methodology. We
based our data collection and assessments on generally accepted social
science standards. We examined such factors as whether evaluation data were
collected before and after program implementation, how program effects were
isolated (i. e., the use of nonprogram participant comparison groups or
statistical controls), and the appropriateness of sampling, outcome
measures, statistical analyses, and any reported results. A social scientist
with training and experience in evaluation research and methodology read and
coded the documentation for each evaluation. A second social scientist
reviewed each completed data collection instrument and the relevant
documentation for the impact
13 During the course of our review, OJJDP officials identified 10 of their
35 ongoing evaluations as impact evaluations of OJJDP- funded programs. In
commenting on a draft of this report, the Assistance Attorney General said
that the Teen Courts evaluation has a comparison group and should have also
been classified as an impact evaluation of an OJJDP- funded program, along
with the other 10 evaluations. Because this had not been included as an
evaluation of an OJJDP- funded program when we initially requested the
information, we did not assess the methodological rigor of the Teen Courts
evaluation as we did with the other 10 impact evaluations. On the basis of
comments provided by the Assistance Attorney General, we have reclassified
the Teen Courts evaluation as an impact evaluation of an OJJDP- funded
evaluation.
Page 10 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
evaluation to verify the accuracy of every coded item. We relied on
documents OJJDP provided to us in April 2001 in assessing the evaluation
methodologies and reporting on each evaluation?s status.
For each of the remaining 24 evaluations, which included nonimpact
evaluations of OJJDP- funded programs, as well as evaluations of juvenile
justice programs that OJJDP did not fund, we asked OJJDP to provide general
descriptive information, such as the type and purpose of the evaluation, the
number of sites involved, and whether the evaluation included data on all
participants. We did not assess the methodological rigor of these
evaluations.
We conducted our work at OJJDP Headquarters in Washington, D. C., from
September 2000 to August 2001 in accordance with generally accepted
government auditing standards.
OJP requires virtually all OJJDP grantees 14 to submit semiannual progress
reports, which OJJDP uses to help monitor grantees? project implementation
and achievement of the goals they identified in their grant applications. To
this end, OJP provides grantees standard, general guidance on the types of
program information they are to report, such as narrative information on the
status of each of their project goals and the quantitative results of their
projects. In addition to this standard requirement, grantees for some of
OJJDP?s programs are subject to additional reporting requirements that apply
only to their respective programs. Our review of 16 major programs 15 showed
that grantees in 8 of the programs were required to comply only with the
standard requirement for information, and grantees in the other 8 programs
were required to report additional specified data. The specific reporting
requirements were established primarily to help evaluate the results of
these programs. Table 1 identifies the 16 programs we reviewed and the
reason for the standard or specific reporting requirements for each program.
14 All discretionary and all formula/ block grant program grantees, with the
exception of Formula Grants Program grantees, must submit these reports. 15
We reviewed all 5 of OJJDP?s formula/ block grant programs and 11 of its
discretionary grant programs. All Grantees Receive
Standard Reporting Guidance and Some Are Required to Report Specific Data
Page 11 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 1: Reasons for Reporting Requirements for 16 Selected Major OJJDP
Programs Reason for reporting requirement Program Standard
requirement Specific requirement
State Challenge Activities Monitoring None Community Prevention Grants
Program (Title V) Monitoring None Juvenile Accountability Incentive Block
Grants Program Monitoring None Enforcing the Underage Drinking Laws Program
(block grant component) a Monitoring None Enforcing the Underage Drinking
Laws Program (discretionary grant component) a Monitoring None Tribal Youth
Program Monitoring None Model Courts Program Monitoring None National
Clearinghouse and Resource Center for Missing and Exploited Children
Monitoring None Formula Grants Program b Monitoring Statutory compliance,
OJJDP assessment Comprehensive Gang Initiative Monitoring Outside
evaluation,
grantee self- assessment Rural Gang Initiative Monitoring Outside
evaluation,
grantee self- assessment Juvenile Mentoring Program (JUMP) Monitoring
Outside evaluation SafeFutures Monitoring Outside evaluation Drug- Free
Communities Support Program Monitoring Outside evaluation Internet Crimes
Against Children Task Force Program Monitoring OJJDP assessment Children?s
Advocacy Centers Monitoring OJJDP assessment,
grantee self- assessment Note: Five programs in this table are formula or
block grant programs: State Challenge Activities, the Community Prevention
Grants Program, the Juvenile Accountability Incentive Block Grants Program,
Enforcing the Underage Drinking Laws Program (block grant component), and
the Formula Grants Program. The remaining 11 programs are discretionary
grant programs.
a The Enforcing the Underage Drinking Laws Program has both a block grant
program component and a discretionary grant program component, and OJJDP
administers each from separate organizational units. b Rather than reporting
semiannually through progress reports, grantees under this program report on
their subgrantees? projects annually through Individual Project Reports. The
standard requirements for reporting are similar for both types of reports.
Source: GAO analysis.
Our review of selected progress reports from the 16 programs showed that, in
all but the Formula Grants Program, grantees reported information on the
status of their activities and accomplishments in response to the standard
requirements, although the details they reported varied as did the projects
themselves. Grantees in the eight programs with specific reporting
requirements reported a variety of descriptive information and performance
data to OJJDP and/ or outside evaluators. (See app. IV for a
Page 12 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
program description, summary of reporting requirements, and examples of what
grantees reported in each of the 16 programs we reviewed.) 16
All OJJDP grantees are required to report on their project activities and
accomplishments to OJP twice a year using the Office of Justice Programs?
Categorical Assistance Progress Report (progress report) form. 17 The form
is unstructured and is to be completed in narrative and/ or chart form. 18
The standard instructions to grantees for completing the form state that
grantees should report information on the status of each of their projects?
goals scheduled to be achieved during the reporting period and set forth in
their grant application, including quantitative project results based on
performance measures. Grantees are also instructed to report on actions
planned to resolve any implementation problems and request any technical
assistance they might need. OJJDP program managers are to use reported
information to help monitor grantees? project implementation. OJJDP
officials explained that because the progress report is intended as just one
of their monitoring tools, this standard, general guidance meets their basic
oversight needs. They further explained that guidance needs to be somewhat
general given the variation that can occur among projects as grantees tailor
them to meet local needs and circumstances. OJJDP encourages grantees to
design projects that meet the unique needs of their own communities, and
therefore grantees do not always report on the same performance measures.
Although OJJDP program managers have additional ways of keeping abreast of
grantees? projects, such as phone calls and on- site visits, OJJDP officials
indicated they would prefer to require and obtain more specific,
16 In addition, we aggregated performance data from grantees in OJJDP?s
training and technical assistance programs and research programs because
such data are quite significant in terms of OJJDP?s outputs, and would not
be fully captured if we reported only on the 16 programs we selected for
review. Although training and technical assistance grantees and research
grantees were typically subject only to standard reporting requirements,
they reported similar types of quantitative performance data. Thus, we were
able to report comprehensive output data, rather than examples of output
data, as we did for selected grantees in the 16 programs. See appendix V for
a summary of performance data from training and technical assistance
grantees and research grantees.
17 As previously noted, the Formula Grants Program is the only exception. In
lieu of the semiannual Categorical Assistance Progress Report, Formula
Grants Program grantees are required to report annually on their
subgrantees? activities and accomplishments using Individual Project
Reports.
18 A copy of the progress report form can be found in appendix III. Grantees
Receiving
Standard Guidance Reported a Variety of Information Used for OJJDP
Monitoring
Page 13 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
and even more frequent, information through the progress reports or other
reporting mechanisms. However, according to these officials, they are
reluctant to impose additional reporting requirements on grantees because of
the Paperwork Reduction Act of 1995, which seeks to ensure that federal
agencies balance their need to collect information with the reporting and
paperwork burden they impose. Under the Act, federal agencies have an
obligation to keep the paperwork burden they impose as low as possible, and
agencies must receive prior approval from the Office of Management and
Budget for information collection requests.
We reviewed selected progress reports that grantees from each of the 16
programs submitted to OJJDP and found that, in all but the Formula Grants
Program, grantees reported on the status of their projects. Grantees
reported input, output, or outcome data related to the process,
implementation, and/ or accomplishments of their projects. They included
information such as subgrant awards, specific meetings held, staff hired,
implementation difficulties, number of project participants, and behavioral
change in youths. However, the particular information grantees reported
varied, as did their projects. This variation, coupled with the unstructured
format of the progress report, makes it difficult to aggregate reported
data.
Fourteen of the programs we reviewed had multiple grantees and the
information these grantees reported in response to the standard guidance
varied, even within each program. For example, we found the following:
Under the Tribal Youth Program- a program that recognizes differences
among tribes and encourages diversity in their projects- grantees must
implement projects in keeping with at least one of four broad purpose areas.
19 One tribe reported that it had completed the renovation of a youth
center; another reported that it had collected examples of other tribes?
juvenile law enforcement codes 20 and started drafting model codes
19 Funds are available for projects that support the following four broad
purposes: (1) to reduce, control, and prevent crime by and against tribal
youth; (2) to provide interventions for court- involved tribal youth; (3) to
improve tribal juvenile justice systems; and (4) to provide prevention
programs focusing on alcohol and drugs.
20 To address the severe juvenile crime problem in Indian communities, many
tribal governments have developed juvenile codes that typically include
provisions relating to the disposition of cases involving Indian youth
arrested or detained by the Bureau of Indian Affairs or tribal law
enforcement personnel. Tribal juvenile codes are generally tailored to
Indian youth by incorporating traditionally accepted cultural methods of
dealing with juvenile delinquency issues.
Page 14 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
adapted for each of its villages. A third tribe reported that the
resignation of its community truancy officer had impacted its ability to
reduce instances of misbehavior in school.
Under another program- the Juvenile Accountability Incentive Block Grants
Program- grantees (states) and their subgrantees (communities) have
undertaken a variety of projects and, thus, report different information. In
this program, states and their communities can choose from 12 different
purpose areas 21 under the broad objective of promoting greater
accountability in the juvenile justice system. Thus, one state reported that
one community hired a juvenile court intake officer and included that
officer?s caseload; the same state reported that another community was
unable to start a project because a local agency declined to participate in
its project. Another state reported on the number of youths enrolled in one
community?s drug testing project and reported the number of drug screening
tests performed.
In the Drug- Free Communities Support Program, grantees design projects to
meet the needs of their local communities; thus, the projects and the
information grantees reported varied. For example, one grantee reported that
it helped local students produce a 30- second anti- smoking commercial in
collaboration with the local health department and further reported that
only 9 of 50 invited members attended a strategic planning meeting it had
held. Another grantee reported making presentations on drug abuse to 146
young men at the local juvenile detention facility, and that its pre- and
post- assessments continually showed that the young men gained knowledge in
the harmful effects of alcohol and drugs.
In the Formula Grants Program, not all grantees reported on the objectives
and accomplishments of their subgrantees? projects, as required. OJJDP
requires grantees in this program to complete an Individual Project Report
21 Funding is available to support the following 12 purpose areas: (1)
construction of juvenile detention or correctional facilities, including
training of personnel; (2) accountability- based sanctions programs; (3)
hiring of judges, probation officers, and defenders, and funding pretrial
services; (4) hiring of prosecutors; (5) funding of prosecutor- led drug,
gang, and violence programs; (6) providing technology, equipment, and
training programs for prosecutors; (7) probation programs; (8) gun courts;
(9) drug courts; (10) interagency information- sharing systems; (11)
accountability- based programs for law enforcement referrals or those
designed to protect students and school personnel from youth violence; and
(12) controlled substance testing (including interventions) for juveniles in
the juvenile justice system.
Page 15 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
(IPR) 22 for each subgrantee. The instructions for completing IPRs are
similar to the instructions grantees in other programs receive for
completing semiannual progress reports. Our review of IPRs from selected
states showed that, for one state, none of the IPRs contained any
information on subgrantees? accomplishments, and some did not include
information on subgrantees? program objectives. 23 For another state,
neither OJJDP nor the state was able to provide us with copies of completed
IPRs because OJJDP?s automated reporting system for states was inoperable.
Two of the 16 programs we reviewed had only one grantee each. Although both
received only standard reporting guidance, they reported more detailed,
quantitative output and outcome data than grantees in the other programs
that received only standard guidance. In the first program- the National
Clearinghouse and Resource Center for Missing and Exploited Children- the
grantee has voluntarily reported detailed information in a structured
format. In the second program- the Model Courts Program- OJJDP has
emphasized that the grantee should include quantitative performance data in
its progress reports, but did not prescribe the specific indicators on which
the grantee must report.
OJJDJP has designated the National Center for Missing and Exploited Children
(NCMEC) as the grantee for National Clearinghouse and Resource Center for
Missing and Exploited Children, and NCMEC has developed its own standardized
reporting format that covers 10 categories. This format collects numbers and
other information on each category, such as missing children cases,
exploited children cases, public affairs, and hotline calls. NCMEC reports
to OJJDP quarterly, rather than semiannually, because this timeframe matches
the reporting structure of its data management system. For the first quarter
of 2001, NCMEC reported various output and outcome data that included
receiving 24,983 calls through its hotline; assisting in the recovery of
1,610 missing
22 IPRs are part of annual Performance Reports, which Formula Grants Program
grantees are required to complete. 23 Although assessing the quality of
OJJDP?s monitoring efforts was outside the scope of our review, we inquired
about the lack of information in this particular state?s IPRs. According to
the responsible OJJDP official, OJJDP staff addressed this issue during a
site visit made subsequent to the state?s annual submission, and also in a
follow- up letter to the state.
Page 16 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
children; 24 receiving 5,291 tips on its online child pornography tip line;
and displaying pictures of 1,399 missing children, which resulted in
locating 257 children.
The sole grantee of the Model Courts Program- the National Council of
Juvenile and Family Court Judges 25 -- also reports quantitative information
in its semiannual progress reports. Although OJJDP has not specified the
performance indicators on which the Council must report, it has emphasized
to the Council the need for quantitative performance data in the semiannual
progress reports. As a result, the Council includes specific quantitative
output data in its progress reports. For example, it reported that during
the last half of 2000 it distributed 17,818 technical assistance bulletins,
conducted 96 training presentations, and made 31 site visits to model
courts. In addition, the Council voluntarily publishes an annual report that
provides more detailed information on the accomplishments of the individual
model courts, such as a reduction in the number of children in court
custody. OJJDP officials told us that they do not require the Council to
provide this report, but they have instructed it to report detailed
performance data on the activities of the model courts, when such data
exist. They further explained that if the Council were to stop publishing an
annual report, OJJDP would require it to include model court performance
data in its progress reports.
In eight of the major programs we reviewed, grantees were given additional,
more explicit reporting instructions requiring them to report on the same
specific performance measures as other grantees in the same program. In
these programs, additional requirements were established to meet the
evaluative needs of OJJDP or an outside evaluator. In one of the eight
programs, requirements were also established to ensure grantee compliance
with certain requirements of the Juvenile Justice Act as well as for program
assessment. The specific requirements of each of these
24 ?Missing children? includes nonfamily and family abductions; runaways and
?thrownaways?; and lost, injured, or otherwise missing children. The number
of recovered children includes deceased children.
25 The Council provides training and technical assistance to 23 ?model
courts? nationwide. These designated model courts seek to improve how child
abuse and neglect cases are processed through the court and child protection
systems. Although their initiatives vary, all the model courts address
certain core issues, such as how well court processes work in everyday
practice and how well the child protection system meets the needs of
children and their families. Grantees in Some
Programs Are Required to Report Specific Performance Data, Often for
Evaluative Purposes
Page 17 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
eight programs varied, as they were tailored to each program. However,
grantees in all these programs were still required to routinely report
narrative information on the status of their activities and accomplishments
through semi- annual progress reports.
In five of these programs, OJJDP and/ or an evaluator have established
specific reporting requirements primarily to support an outside program
evaluation. 26 For example, as a condition of receiving a Juvenile Mentoring
Program grant, OJJDP requires all grantees to participate fully in the
evaluation by providing data to the evaluator. This evaluator requires
grantees to report their data through quarterly progress reports that are
similar to semiannual progress reports. The required data include
information on youths participating in each project, participating mentors,
and youth- mentor matches. For instance, grantees are required to report
family structure information for participating youths. The evaluator
aggregates such data from all grantees and has reported, for example, that
56 percent of participating youths lived with their mother only, 20 percent
lived with both parents, 4 percent lived with their father only, and 21
percent were in other living arrangements. 27
In two of the eight programs- the Internet Crimes Against Children Task
Force Program and Children?s Advocacy Centers- specific reporting
requirements were established so that OJJDP could assess program
accomplishments. The governing board of the Internet Crimes Task Force, in
agreement with OJJDP, identified monthly performance measures on which
grantees must report, such as the number of arrests made, search warrants
issued, subpoenas served, and cases opened by the task forces. Under the
Children?s Advocacy Centers program, 28 OJJDP prescribed specific
performance measures on which grantees must report, such as the number of
practitioners trained, training conferences held, and publications
distributed. In this program, specific reporting requirements were
established not only for OJJDP to assess the program?s overall
accomplishments, but also to help grantees assess their own projects.
26 In two of these five programs, a secondary purpose for imposing these
requirements was to allow grantees to assess their own programs. 27 Data
were reported in a September 2000 OJJDP Juvenile Justice Bulletin entitled
Juvenile Mentoring Program: A Progress Review.
28 This program is, in part, a training and technical assistance program.
Page 18 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
The eighth program- the Formula Grants Program- has requirements that are
statutorily based and further spelled out by OJJDP in program regulations.
Program reporting requirements were established to ensure grantees comply
with the four core requirements of the Juvenile Justice Act and as a basis
for assessing the effects of the program. These core requirements are (1)
deinstitutionalization of status offenders, 29 (2) separation of juveniles
from adult offenders, (3) removal of juveniles from adult jail and lockup,
and (4) addressing efforts to reduce disproportionate minority confinement.
OJJDP regulations list in detail the information on which states must
report. 30 For instance, regarding the deinstitutionalization of status
offenders, states must report the total number of accused and adjudicated
status offenders and nonoffenders placed in facilities that are, for
example, not near their home community. (See app. VI for a summary of
states? compliance with the core requirements of the Juvenile Justice Act.)
According to the compliance monitoring coordinator for the Formula Grants
Program, grantees? reports on compliance with the core requirements also
provide the basis for OJJDP to assess the effects of the program.
We identified eight programs that serve juveniles directly and whose
grantees reported such data for fiscal year 2000. However, OJJDP often does
not require grantees to provide this information, in large part because not
all of its programs are intended to provide direct services to juveniles.
29 A status offender is a juvenile who has been charged with or adjudicated
for conduct that would not be a crime if committed by an adult. 30 28 C. F.
R. 31. 303( f)( 5). Some Data Exist on
the Number of Juveniles Served Directly, Although Most Grantees Are Not
Required to Report This Information
Page 19 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
We identified eight programs in which grantees directly serve juveniles and
in which all grantees report the number of juveniles served to either OJJDP
or an outside evaluator. About 400 grantees in these eight programs directly
served close to 142,000 juveniles in one year. For example, in fiscal year
2000, the Juvenile Mentoring Program reported serving about 8,500 juveniles,
and in calendar year 2000, the Court Appointed Special Advocate Program
reported serving 70,348 youths. Table 2 shows the programs we identified as
directly serving juveniles and reporting such data for fiscal year 2000.
Table 2: OJJDP Programs GAO Identified in Which All Grantees Reported Data
on the Number of Juveniles Directly Served, Fiscal Year 2000
Program Description Juveniles directly served Grantees
Arts and At- Risk Youth A program that combines the arts with job training
and conflict resolution skills and provides summer jobs or paid internships
for youths in the program.
381 3 The Arts Program for Juvenile Offenders in Detention and Corrections A
program that works with arts professionals to help youths
develop art skills to use after release from detention. 1,026 6
Comprehensive Gang Initiative A demonstration program that utilizes a
variety of strategies,
including intensive youth and family intervention, community mobilization,
and organizational change, to reduce youth gang crime and violence.
78 a 2 Court Appointed Special Advocate Program A program to help ensure
that abused and neglected children
receive timely and effective representation in dependency hearings. Program
volunteers are appointed by judges to advocate in court for the best
interests of abused or neglected children.
70,348 b 207 Juvenile Mentoring Program (JUMP) A one- on- one mentoring
program for at- risk youths. 8, 509 175 Life Skills Training A training and
technical assistance program that instructs
teachers on how to replicate the Life Skills Training program in a large
number of diverse jurisdictions. Life Skills Training is a tested whole-
school- immersion model for reducing drug use by middle- school students in
selected communities.
53,685 c 1 SafeFutures A program that works to reduce delinquency and youth
violence through the development of a continuum of services. Grantees in
this program use a single grant to draw on a variety of OJJDP categorical
funds.
5,789 d 6 Truancy Reduction Demonstration Program A program to reduce the
number of truant children by
providing services, such as truancy workshops and truancy case workers.
2,085 7
Total 141,901 407
Note: The programs listed are those we identified in which all grantees
reported juveniles- served data. We identified these programs by reviewing
OJJDP literature and interviewing OJJDP division directors. Nevertheless,
the list may not be comprehensive.
In Some Programs, All Grantees Report the Number of Juveniles They Directly
Serve
Page 20 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
a The level of services provided under this program varied widely by
juvenile- i. e., some juveniles received intensive intervention counseling,
while others were only monitored by a case worker. b This number represents
juveniles served in calendar year 2000.
c This number is associated with the academic year 1999- 2000, which roughly
corresponds with fiscal year 2000. Because this is a program to teach drug
prevention skills to all middle- school students in selected communities,
the grantee reports the number of ?students reached? rather than the number
of ?juveniles served.? d This number represents a combination of data
provided by SafeFutures grantees and by the
program?s national evaluator. In addition, the level of services provided to
these juveniles varied widely by individual, with some grantees including
data on juveniles who received a full array of project services, and others
reporting data on juveniles who were assessed but subsequently denied
services or referred elsewhere. Furthermore, one grantee reported data for a
slightly different period- September 1999 through August 2000.
Source: GAO analysis of OJJDP, evaluator, and grantee data.
We also identified a program in which all subgrantees directly serve
juveniles, but not all subgrantees report such data. The national grantee 31
for the Children?s Advocacy Centers program reported that its subgrantees
served over 100,000 juveniles in calendar year 2000, but this number
represents only those juveniles served by subgrantees accredited through a
national membership council. 32
For several reasons, OJJDP does not typically require grantees to report the
number of juveniles their projects directly serve. First and foremost, many
of OJJDP?s programs are not intended to serve juveniles directly. The
Juvenile Justice Act established OJJDP for a variety of purposes, many of
which involve indirect benefits to juveniles, rather than direct services.
Statutorily- established purposes for OJJDP include the following:
To provide technical assistance to and training programs for professionals
who work with delinquents.
To provide for the evaluations of federally- assisted juvenile justice and
delinquency prevention programs.
To establish a centralized research effort on problems of delinquency. 31
The national grantee (one of five program grantees) awards funds to
subgrantees, who in turn provide direct services to juveniles. The other
four grantees, which are regional, provide training and technical assistance
to local Children?s Advocacy Centers, as does the national grantee.
32 Accredited subgrantees received 75 percent of funds available to
subgrantees. Reasons Exist for Not
Requiring Grantees to Report Number of Juveniles Served
Page 21 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
To assist state and local governments in improving the administration of
justice and services for juveniles who enter the system.
Some of OJJDP?s programs, in their entirety, provide indirect benefits,
rather than direct services, to juveniles. OJJDP?s Model Courts Program, for
example, benefits juveniles indirectly by providing training and technical
assistance to court personnel to improve their handling of child abuse and
neglect cases. The Internet Crimes Against Children Task Force Program also
benefits juveniles indirectly by helping to identify and arrest pedophiles
and child pornographers who use the Internet to prey on children.
Furthermore, in commenting on a draft of this report, the Assistant Attorney
General pointed out that although OJJDP?s research projects do not typically
provide services directly, their results can potentially help thousands of
juveniles.
OJJDP officials provided the following additional reasons for not requiring
all grantees to report the number of juveniles their projects directly
serve:
A common interpretation of ?juveniles served? does not exist across, or
even within, programs. For example, grantees in one program might consider
the number of juveniles served as those assessed for services but referred
elsewhere, while grantees in a different program might consider only
juveniles who received at least 10 sessions of therapy. Even within the same
program grantees may not share a common definition of
?juveniles served.? One program grantee might report on the number of
juveniles who received intensive one- on- one drug prevention services over
an extended period of time, while another grantee in that same program might
report on the number who attended a one- time presentation on drug
prevention. Without a common interpretation of ?juveniles served,? the data
grantees report would be inconsistent and would have little value.
For some programs, directly serving juveniles may be only one of a number
of intended program purposes and thus, OJJDP does not typically require all
grantees within these programs to report such data. For example, in the
Formula Grants Program, states and their subgrantees can choose from among
14 different program areas related to preventing and controlling delinquency
and improving juvenile justice systems. Under the program area of ?planning
and administration,? for instance, states can fund planning projects that
benefit juveniles indirectly, such as developing a comprehensive state plan
to identify juvenile service needs and programs that address those needs
over the long term. However, under the area of ?illegal drugs and alcohol,?
a local subgrantee can serve juveniles directly by establishing a drug and
alcohol abuse prevention project.
Page 22 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Juveniles- served data could be used inappropriately to measure the
effectiveness of a program whose primary purpose may not be to provide
direct services to juveniles. For example, the primary purpose of State
Challenge Activities is to stimulate system- wide change, although many of
its 10 activity areas also promote projects intended to directly serve
juveniles. However, grantees in this program are expected to implement
direct service projects within the broader context of promoting systemwide
change. For instance, one State Challenge Activities grantee used funds it
received under the ?deinstitutionalization of status offenders? activity
area to establish two community projects that provide housing for runaway
juveniles, many of whom are girls. The grantee intends to use its
experiences with these two new projects to initiate system- wide change by
developing a comprehensive model program expressly geared to serving runaway
girls. By focusing on the number of girls served by this program, one might
fail to see that its primary purpose was to develop a comprehensive model
program for serving runaway girls.
OJJDP has funded 35 evaluations since 1995, including 11 evaluations
intended to measure the impact of OJJDP- funded programs. We reviewed the
methodological rigor of 10 of the 11 evaluations. 33 Half of these 10
evaluations are in formative stages, while the other five are well into
implementation. 34 None had been completed at the time of our review. Our
in- depth review of these 10 evaluations shows that although several are
well- designed and use, or plan to use, sophisticated data analysis methods,
others raise concerns as to whether the evaluations will produce definitive
results. We recognize that impact evaluations, such as the types that OJJDP
are funding, can encounter difficult design and implementation issues. For
some of the evaluations we reviewed, program flexibility has added to the
complexity of designing evaluations. A lack of comparison groups to aid in
isolating the impacts of some programs, and data collection problems could
compromise some evaluation results.
According to OJJDP officials, the OJJDP weighs a number of factors when
deciding which programs to evaluate and what kind of evaluations to fund.
33 As discussed in the Scope and Methodology section of this report, in
commenting on a draft of this report, the Assistant Attorney General pointed
out that the evaluation of the Teen Courts program is also an impact
evaluation of an OJJDP- funded program. This section addresses only the 10
impact evaluations OJJDP had initially identified.
34 We based our analysis of these 10 ongoing evaluations on documentation
OJJDP provided as of April 2001. Methodological
Concerns Could Adversely Affect Evaluation Studies
Page 23 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Given its budget, it considers how much of its discretionary funds to spend
in support of evaluation activities. In deciding which of its programs to
evaluate, OJJDP gives priority consideration to programs that have been
mandated by the Congress. Other criteria OJJDP uses to determine whether a
program should be evaluated include the program?s level of funding and its
uniqueness, as well as the feasibility and cost of an evaluation and its
potential benefits to the field. Similar criteria are also involved with
decisions to evaluate non- OJJDP funded programs, as well as congressional
interest and other federal agencies? willingness to co- fund an evaluation.
The 10 impact evaluations of OJJDP- funded programs that we assessed vary in
size and scope. The cost to conduct these evaluations ranges from $300,000
to well over $5 million; however, some of these grants involve both impact
and process evaluations and the cost of the impact portion alone cannot be
separated from the total. All 10 evaluations are multi- year, multi- site
projects. The number of evaluation sites ranges from 2 in the Rural Gang
Initiative to 175 in the evaluation of the Juvenile Mentoring Program. As of
April 2001, three evaluations had produced interim findings of some program
impacts. 35 (See app. II for information on OJJDP?s process for
disseminating products with interim findings as well as other products.)
Program evaluation is an inherently difficult task because the objective is
to isolate the impact of a particular program from all other factors that
could have caused a change consistent with the intent of the program, or
mitigated against that change. Given that programs, such as those funded by
OJJDP, operate in an ever- changing environment and involve juveniles and
adults who themselves constantly change, producing definitive evaluation
results can be arduous. For example, the impact of a hypothetical program
intended to improve students? grades could be confounded by the effects of
an outside- of- school mentoring program, the transfer of high- performing
students to a magnet program, changes in school faculty, a new scholarship
program, a severe flu season that results in widespread student absences
from school, and a myriad of other factors.
35 Two other evaluations reported preliminary results of some of the data
collected as part of the impact evaluations; however, these did not comprise
impact results. Evaluations of OJJDP
Programs Are Difficult to Successfully Design and Implement
Page 24 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Our in- depth review of the 10 impact evaluations of OJJDP programs showed
that a number of these evaluations are particularly complex because local
grantees design their own projects to fit their communities? needs. (See
app. VII for a description of the impact evaluations OJJDP has funded of its
own programs since 1995). Although this customization may make sense from a
program perspective, it makes it more difficult to evaluate the program.
Instead of assessing a single, homogeneous program with multiple grantees,
the evaluation must assess the effects of multiple configurations of a
program. Although all of the grantees? projects under each program being
evaluated are intended to achieve the same or similar goals, an aggregate
analysis could mask differences in individual projects? effectiveness and,
thus, not result in information about which configurations of projects work
and which do not.
OJJDP?s evaluation of the Enforcing the Underage Drinking Laws Program
(discretionary grant component) exemplifies this situation. In implementing
their projects, states and local communities have substantial latitude to
employ media campaigns, merchant education, compliance checks, youth
leadership training, or a variety of other activities to deter underage
drinking. Similarly, under the Positive Action Through Holistic Education
program, local educators develop their own ways to prevent student violence
and behavior problems based on their assessments of the causes of these
problems in their schools. Because of the limited number of sites (two
school districts) being evaluated and the likely differences in how each
school has developed its own project, the resulting evaluation may not
provide information that could be generalized to a broader implementation of
the program.
A standard way for evaluators to isolate the impacts of a program from other
potential factors that could have influenced change is to use a comparison
group as a benchmark. In the hypothetical example cited above concerning a
program to improve students? grades, a second set of students who are not in
the program but are matched in academic performance and exposed to all of
the same factors (except the program) could provide a baseline from which to
assess the impact of the program. The grades of students in the two groups
before and after the program would provide the data from which to measure
program impacts. Without the benefit of the comparison group as a baseline,
it is difficult or impossible to isolate changes resulting from the program
from changes due to other factors.
The designs of two of the five evaluations that are well into implementation
lack an appropriate comparison group. The evaluation of Project Variation
Within a
Program Can Complicate Evaluation Design
Lack of Appropriate Comparison Groups Could Compromise Some Results
Page 25 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
the Juvenile Mentoring Program- a one- on- one mentoring program for youths-
compares youths entering the program to those completing it. However, a
variety of other factors, including the fact that youths in the program are
likely to mature and, thus, improve somewhat spontaneously, cannot be ruled
out as a rival cause of change from the beginning to the end of the program.
Although the evaluators are employing multiple and innovative strategies to
determine the effectiveness of the program in achieving its objective, the
lack of a comparison group of nonparticipant youths is an obstacle to
identifying definitive outcomes.
An evaluation of the Partnerships to Reduce Juvenile Gun Violence Program
includes a comparison of before and after crime statistics in project
communities with crime statistics for the same time frames for the cities in
which the projects operate. However, citywide crime statistics would no
doubt include data from communities that are similar to the project
community as well as from those that are not. Thus, the differences between
citywide and project community baselines make it difficult to attribute
potential findings to the program.
Of the five programs for which evaluations are still being developed, two
(the Safe Start Initiative and the Rural Gang Initiative) did not seem to
have plans for comparison groups at the time of our review. Another (Parents
Anonymous) anticipates using a comparison group, but as yet had not
developed specific plans for one.
Regardless of the quality of a program evaluation design, data collection
problems can compromise the validity of findings. Data collection problems
may affect the validity of the findings for three of the five evaluations
that are currently completing or have completed data collection. The
Juvenile Mentoring Program evaluation has experienced problems obtaining
behavioral measures and school performance data with which to gauge program-
driven change. The Comprehensive Gang Initiative evaluation has also
experienced data collection problems such as the lack of fully adequate
comparison youth data at all or most sites, missing police histories, and
missing self- reported data.
The Intensive Aftercare evaluation has experienced survey response rate
shortfalls, in some cases obtaining response rates of less than 30 percent,
which may affect the validity of the findings. In commenting on a draft of
this report, the Assistant Attorney General said that the poor response
rates for some elements at different sites were particularly disappointing
because this evaluation had a strong random assignment design; however, the
strategies for obtaining adequate data turned out to be insufficient. She
Data Collection Problems
Could Compromise Some Results
Page 26 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
added that the program staff who were required to collect data did not give
data collection adequate priority in comparison to their other duties. This
was particularly true of data regarding the comparison groups.
In addition to funding impact evaluations of OJJDP programs, OJJDP has
funded 24 other evaluations since 1995- 11 nonimpact evaluations of OJJDP
programs, 9 impact evaluations of programs that were not supported by OJJDP
funds, and 4 nonimpact evaluations of programs that were not funded by
OJJDP. The nonimpact evaluations are not intended to determine the outcomes
of the various programs, but rather how well the programs have been
implemented. For example, OJJDP has funded a process evaluation of its
SafeFutures program to learn more about the process of community
mobilization and collaboration in building a comprehensive program of
prevention and intervention for at- risk youths and juvenile offenders.
OJJDP has also funded evaluations of programs that are funded by entities
other than OJJDP. For example, although OJJDP does not fund the Act Now
Truancy Program, it has funded a nonimpact evaluation of this program. The
Act Now Truancy Program grew out of a unique Arizona law that allowed
prosecutors to issue citations to parents whose children were chronically
truant. Because there was a great deal of interest in this approach and
OJJDP believed it provided a unique opportunity to learn about the impact of
an unusual approach, it funded an evaluation of the program. Appendix VIII
contains brief descriptions of these 24 other evaluations.
Although there is great interest in assessing results of programs, it is
extremely difficult to design and execute evaluations that will provide
definitive information. Our in- depth review of 10 OJJDP- funded evaluations
of OJJDP's own programs undertaken since 1995 has shown that, in some cases,
the flexibility that can be beneficial to grantees in tailoring programs to
meet their communities' needs has added to the complexities of designing
impact evaluations that will result in valid findings. Furthermore, the lack
of an appropriate comparison group or sites and/ or problems in data
collection may compromise the reliability and validity of some of these
evaluations. Because half of these 10 evaluations are in relatively early
stages, any potential problems with comparison group issues or data
collection shortfalls could still be resolved over the course of the
evaluation. We recognize that not all evaluation issues that can compromise
results are resolvable, including OJJDP Has Funded
Other Types of Evaluations
Conclusions
Page 27 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
many involving comparison groups and data collection. However, to the extent
that appropriate comparison groups can be established and tracked and data
collection issues can be overcome, the validity of the evaluation results
can be enhanced.
Our review of the recent OJJDP program evaluations has shown that, of the
five that are in or near their final stages, some problems with valid
comparison groups and/ or data collection could compromise the usefulness of
some of their results. Five other program evaluations are in a formative
stage where comparison group issues and data collection strategies are not
yet finalized. Accordingly, we recommend that the Attorney General direct
the Administrator of OJJDP to assess the five impact evaluations in the
formative stages to address potential comparison group and data collection
problems and, on the basis of that assessment, initiate any needed
interventions to help ensure that the evaluations produce definitive
results.
We provided a copy of this report to the Attorney General for review and
comment. In an October 15, 2001 letter, the Assistant Attorney General
commented on a draft of this report. Her comments are summarized below and
are presented in their entirety in appendix IX. Her detailed comments have
been addressed in the report as appropriate.
The Assistant Attorney General said that the draft report provides useful
information that highlights areas warranting attention. She added that the
draft report would be an important tool that OJP will use to improve the
quality of its evaluations and to design programs that will achieve greater
impact. Furthermore, OJP will assess the five impact evaluations that are
currently in their formative stages to address potential comparison group
and data collection problems. On the basis of that assessment, OJJDP will
initiate any needed interventions to help ensure that evaluations produce
definitive results.
The Assistant Attorney General said that OJP agrees that it should always
strive for more rigorous and scientifically sound evaluation designs and
that the inclusion of comparison groups would certainly strengthen the
interpretation of evaluation results. However, she disagreed with our
reliance on the use of comparison groups as the only valid evaluation design
for two primary reasons. First, OJJDP seeks to conduct juvenile justice
evaluations in a real- world setting, where laboratory- like comparison
groups may not be possible. Second, sufficient funding is not Recommendation
for
Executive Action Agency Comments and Our Evaluation
Page 28 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
available for including comparison groups in every evaluation. The Assistant
Attorney General also said that given the choice between conducting far
fewer evaluations, all with comparison groups, and conducting a greater
number of evaluations under less- than- ideal conditions, OJJDP?s Research
and Program Development Division works hard to tread a middle ground that
satisfies needs for both quality and quantity. She further pointed out that
a growing number of policy makers and evaluators firmly believe that
community- based initiatives do not lend themselves to the kind of
traditional evaluations that this draft report proposes. Accordingly, some
researchers have strongly urged that new approaches to evaluation be
developed.
In addition, the Assistant Attorney General said that our report suggests
that more evaluations using experimental or quasi- experimental evaluation
designs should be funded. She added that many communities reject
participation in programs that are evaluated in this way (i. e., with
control or comparison groups) because they feel that it requires them to
purposely exclude youths from receiving services.
In her comments, the Assistant Attorney General seemed to be using the terms
?comparison group? and ?control group? interchangeably. However, control
groups are commonly associated with experiments involving random assignment.
We do not intend our statements regarding the need for comparison groups in
impact evaluations to imply that random assignment is necessary for studies
to be valid. Furthermore, we recognize that groups can be compared after
controlling for differences by methods other than random assignment,
including statistical methods and various methods of matching. For impact
evaluations, comparisons should be made, and should involve individuals who
were not subject to the program or treatment being evaluated. However, not
all the evaluations we assessed made such comparisons.
We also recognize that not all evaluation issues that can compromise results
are resolvable, even with the use of comparison groups. We also recognize
that designing evaluations with comparison groups can be expensive and
funding limitations could preclude their use in all evaluations. In
addition, obtaining participants can be troublesome, as the Assistant
Attorney General pointed out. However, the validity of evaluation results
can be enhanced through establishing and tracking comparison groups. If
other ways exist to effectively isolate the impacts of a program, comparison
groups may not be needed. However, we saw no evidence of other methods being
used in the 10 impact evaluations we assessed. While studies that do not
have appropriate comparison groups
Page 29 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
can provide useful information, they should not be considered impact
evaluations. Furthermore, we recognize the fact that communities may not
favor withholding treatments or programs from individuals in control or
comparison groups, however, this problem is commonly handled by phasing in
the treatment or program and offering it to comparison group members
following the evaluation period.
As we agreed with your office, unless you publicly announce the contents of
this report earlier, we plan no further distribution of it until 30 days
from the date of this letter. At that time, we will send copies to the
Senate Judiciary Committee, the Senate Subcommittee on Youth Violence, the
House Committee on Education and the Workforce, the House Subcommittee on
Early Childhood, Youth and Families, the Attorney General, and the Director
of the Office of Management and Budget.
If you or your staff have any questions about this report, please contact
James M. Blume or me at (202) 512- 8777. Key contributors to this report are
acknowledged in appendix X.
Sincerely yours, Laurie E. Ekstrand Director, Justice Issues
Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000
Page 30 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
This appendix provides information on the awards the Office of Juvenile
Justice and Delinquency Prevention (OJJDP) made each year from fiscal years
1996 through 2000. It contains data on OJJDP funds awarded to formula/ block
grant programs versus discretionary grant programs (see fig. 2), OJJDP funds
awarded by more specific program areas (see table 3), types of OJJDP award
recipients (see table 4), and OJJDP formula/ block grant awards by state
(see table 5).
We relied on the Office of Justice Program?s (OJP) awards database to
analyze data on all OJJDP- administered awards made during this 5- year
period. We analyzed awards by the year the award was made- not the year in
which the funds were appropriated. We worked with OJJDP officials to
identify awards by major program or program area, as the database did not
provide sufficiently detailed information. OJP officials advised us that
they perform daily quality control checks on all data entered into their
database, however, we did not verify the accuracy of the database.
Figure 2: OJJDP Formula/ Block Grants Versus Discretionary Grant Awards,
Fiscal Years 1996 Through 2000
Note: Dollar amounts are rounded to the nearest million. Source: GAO
analysis of OJP database.
Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000
Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000
Page 31 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 3: OJJDP Funds Awarded by Program Area, Fiscal Years 1996 Through 2000
Program area FY 1996 FY 1997 FY 1998 FY 1999 FY 2000 Formula/ block grant
programs
Formula Grants Program $68,795,296 $77,469,250 $92,069,600 $74,468,599
$69,699,750 State Challenge Activities 9,352,900 10,363,264 9, 726,800
9,011,429 8,227,500 Community Prevention Grants Program (Title V) 19,762,000
18,368,000 18,707,000 39,921,000 36,158,000 Enforcing the Underage Drinking
Laws Program (block grant component)
a a 18,360,000 17,640,000 19,080,000 Juvenile Accountability Incentive Block
Grants Program
a a 232,250,000 232,250,000 221,094,775
Discretionary grant programs National Institute for Juvenile Justice and
Delinquency Prevention b
Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders
900,730 900,055 1, 835,597 1,842,240 2,640,000 Causes and Correlates of
Delinquency Research 299,999 900,000 599,999 600,000 750,000 Statistics and
Systems Development Program 349,978 249,996 449,998 549,923 649,996 Other
research, statistics, evaluation, information dissemination, and training
and technical assistance grants 12,382,088 15,689,329 21,566,738 40,816,294
42,783,923
Demonstration and replication programs
Comprehensive Gang Initiative 2,349,597 2,848,881 3,224,480 2,249,742
1,149,498 Rural Gang Initiative a a 215,000 1, 112,754 1,474,473 SafeFutures
2, 159,950 5,540,000 4,715,979 9,462,844 10,829,102 Community Assessment
Centers a 1,074,552 1,699,992 125,000 875,000 Juvenile Mentoring Program
(JUMP) a 9,688,488 3,363,281 21,059,220 5, 760,000 Partnerships to Reduce
Juvenile Gun Violence Program
a 551,060 1, 050,230 1,299,975 799,996 Enforcing the Underage Drinking Laws
Program (discretionary grant component)
a a 6,608,019 2,344,986 6,272,003 Drug- Free Communities Support Program a a
8,903,433 19,039,326 29,677,581 Truancy Reduction Demonstration Program a a
a 693,849 711,328 Tribal Youth Program a a aa 8,252,019 Other demonstration
and replication grants 7, 070,330 8,846,421 17,178,085 26,309,408 24,058,149
Missing and Exploited Children Program
National Clearinghouse and Resource Center for Missing and Exploited
Children 3,294,648 4,576,640 7,585,000 8,925,000 19,490,263 Missing and
Exploited Children Training & Technical Assistance program
a 3,448,575 1,749,803 1,848,947 1,949,741 Internet Crimes Against Children
Task Force Program
a a 2,351,829 199,997 8, 486,056 Other missing and exploited children grants
1, 189,639 1,488,935 1,759,887 978,356 2, 074,318
Child Abuse and Neglect Program
Court Appointed Special Advocate Program 6, 040,000 5,904,400 7,062,100
9,034,200 9,796,000 Children?s Advocacy Centers 3, 199,856 3,099,914
5,691,000 4,733,000 5,824,302
Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000
Page 32 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program area FY 1996 FY 1997 FY 1998 FY 1999 FY 2000
Model Courts Program 742,500 984,100 1, 964,900 2,108,097 2,109,200 Safe
Start Initiative a a aa 8,038,573 Other child abuse and neglect grants
1,725,000 3,199,878 4,838,152 6,508,938 6,499,976
Total $139,614,511 $175,191, 738 $475,526, 902 $535,133, 124 $555,211, 522
Note: Award amounts are not adjusted for de- obligated funds, except in
cases in which an entire award was de- obligated and later re- awarded, as
occurred in rare instances of formula/ bock grant awards. a Either the
program did not receive an appropriation or OJJDP did not award funds.
b Awards listed under this subheading comprise a variety of research,
statistics, evaluation, information dissemination, and training and
technical grants. However, the funds awarded under this subheading do not
represent a comprehensive accounting of all such activities funded by OJJDP.
In some cases these activities are accounted for under other subheadings.
For example, the Missing and Exploited Children Training and Technical
Assistance award, which is primarily a training and technical assistance
grant, is listed under the Missing and Exploited Children?s Program. Also,
evaluations for many of the demonstration and replication programs are
accounted for under their respective programs, rather than under the
National Institute for Juvenile Justice and Delinquency Prevention. In
commenting on a draft of this report, the Assistant Attorney General pointed
out that combining training and technical assistance funding under the
National Institute of Justice and Delinquency Prevention does not highlight
the importance of training and technical assistance. Although we recognize
the importance of training and technical assistance, OJP?s awards database
did not readily lend itself to identifying all training and technical
assistance- either awards or portions of awards- because they are part of
many programs across OJJDP divisions.
Source: GAO analysis of OJP database.
Table 4: OJJDP Award Recipients, Fiscal Years 1996 Through 2000 FY 1996 FY
1997 FY 1998 FY 1999 FY 2000 Recipients Amount
awarded % a Amount awarded % a Amount
awarded % Amount awarded % a Amount
Awarded % a
State governments $98,854,634 71 $109,350,562 62 $377,103,675 79
$377,819,401 71 $369,196,062 67 Local governments b 2,584,003 2 7, 230,243 4
13,534,664 3 18,766,350 4 27,078,615 5 Schools c 4,936,373 4 14,425,127 8
18,292,717 4 20,926,891 4 34,975,373 6 Nonprofit organizations 32,289,708 23
41,026,686 23 60,476,935 13 106,044,793 20 109,431,488 20 Other d 949,793 1
3,159,120 2 6, 118,911 1 11,575,689 2 14,529,984 3
Total $139,614,511 100 $175,191,738 100 $475,526,902 100 $535,133,124 100
$555,211,522 100
a Percentages may not total 100 due to rounding. b Local governments include
county governments, cities, townships, and special district governments. c
Schools include independent school districts and state and private
institutions of higher education. d Other includes for- profit
organizations, individuals, Indian tribes, and other miscellaneous
categories.
Source: GAO analysis of OJP database.
Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000
Page 33 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 5: OJJDP Formula/ Block Grant Awards, by State, Fiscal Years 1996
Through 2000 Funds awarded State FY 1996 FY 1997 FY 1998 FY 1999 FY 2000
Alabama $1,424,000 $1,648,900 $5,896,400 $5,855,300 $5,638,100 Alaska
787,500 832,500 2, 822,300 2,839,800 2,594,200 Arizona 1, 502,000 1,821,000
6,197,500 6,714,900 6,327,800 Arkansas 869,000 1, 014,500 4,222,700
4,432,700 4,222,100 California 11,440,000 13,417,100 37,575,000 37,547,300
36,119,800 Colorado 1,279, 000 1,497,000 5,579,400 5,649,200 5,474,219
Connecticut 1,017,000 1,197,000 4,744,200 4,824,300 4,612,400 Delaware
787,500 829,500 2, 133,100 3,469,600 2,726,200 District of Columbia 788,546
187,500 1, 785,400 1,778,000 1,699,100 Florida 4, 302,000 5,144,000
15,440,600 15,631,200 15,089,600 Georgia 2, 467,000 2,907,000 9,432,800
9,534,600 9,188,600 Hawaii 787,500 860,500 2, 984,800 3,402,400 3,096,800
Idaho 787,500 869,500 3, 280,000 3,388,200 3,264,700 Illinois 3, 605,000
1,626,064 17,824,400 14,320,800 12,886,850 Indiana 1, 942,000 2,269,000
7,615,300 7,608,129 7,304,900 Iowa 961,000 1, 106,000 4,446,700 4,641,700
4,438,300 Kansas 909,000 1, 057,000 4,316,400 4,517,200 4,360,300 Kentucky
278,000 0 4,117,800 5,117,100 6,659,600 Louisiana 1, 553,900 1,816,100
6,518,200 6,317,100 6,010,400 Maine 787,500 859,500 2, 965,650 3,027,850
3,076,500 Maryland 1,665,000 1,941,000 6,751,400 6,678,100 6,418,700
Massachusetts 1,537,500 893,000 7, 789,450 7,375,900 7,035,600 Michigan
5,572,000 3,808,000 11,800,200 11,613,500 11,207,319 Minnesota 1, 636,000
1,899,000 6,592,150 6,604,300 6,312,800 Mississippi 997,000 857,000 4,
902,400 4,736,500 4,327,600 Missouri 1, 797,000 2,088,000 7,169,800
7,188,300 6,835,700 Montana 787,500 843,500 2, 957,900 2,619,400 3,223,100
Nebraska 664,500 910,500 3, 558,900 3,700,700 3,563,000 Nevada 795,500
888,500 3, 481,600 3,695,300 3,630,700 New Hampshire 637,500 1, 006,500
3,133,550 3,192,800 3,089,400 New Jersey 2,546,000 2,996,000 7,096,000
7,676,900 9,528,200 New Mexico 830,500 939,500 3, 739,300 3,884,500
3,748,900 New York 5,815,000 6,798,000 19,858,900 19,779,400 18,544,600
North Carolina 2, 260,000 2,693,000 8,924,300 9,016,600 8,746,900 North
Dakota 787,500 827,500 2, 779,400 2,766,400 2,682,900 Ohio 3,763,000
4,363,000 13,101,700 13,060,100 9, 405,700 Oklahoma 1, 160,000 1,340,000
5,102,900 5,170,600 4,946,500 Oregon 1,027,000 1,217,000 4,809,400 4,934,700
3,769,500 Pennsylvania 3, 821,000 4,439,000 13,292,600 13,163,000 12,471,600
Rhode Island 787,500 843,500 2, 965,000 2,992,300 2,905,700
Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000
Page 34 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Funds awarded State FY 1996 FY 1997 FY 1998 FY 1999 FY 2000
South Carolina 1,240,000 1,427,000 5,319,300 5,166,300 4,938,550 South
Dakota 787,500 100,000 2, 013,500 1,993,800 1,921,300 Tennessee 1,710,000
1,999,000 6,897,100 6,905,800 5,332,000 Texas 6, 910,000 8,165,000
23,614,200 24,006,200 22,961,500 Utah 885,000 1, 030,000 4,281,400 4,518,500
4,372,000 Vermont 787,500 822,500 2, 717,300 2,711,200 2,626,619 Virginia
2,113,000 2,460,000 8,155,800 8,205,799 7,794,800 Washington 1,856,000
2,164,000 7,362,500 7,433,800 6,784,350 West Virginia 810,500 899,500 3,
497,100 3,590,800 3,432,100 Wisconsin 1, 755,000 2,045,000 6,538,150
7,368,750 6,621,600 Wyoming 0 0 1, 933,950 1,836,800 1,759,100 American
Samoa 148,000 133,000 617,391 601,962 472,248 Guam 115,000 192,000 842,350
841,700 807,800 Northern Mariana Islands 148,000 151,000 381,959 381,738
370,552 Puerto Rico 1,367,750 1,917,000 5,856,900 5,854,100 5,548,800
Republic of Pulau 15,000 21,350 4, 300 0 0 Virgin Islands 100,000 123,000 1,
374,700 1,407,100 1,331,818
Total $97,910,196 $106,200, 514 $371,113,400 $373,291, 028 $354,260, 025
Source: GAO analysis of OJP database.
Appendix II: OJJDP?s Process for Disseminating Published Products, Including
Interim Results of Impact Evaluations
Page 35 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
The Office of Juvenile Justice and Delinquency Prevention (OJJDP) has a
process for disseminating published interim results of impact evaluations as
well as other publications produced by OJJDP and its grantees. OJJDP
publications are available through the Juvenile Justice Clearinghouse. 1
According to an OJJDP official, OJJDP develops a specific strategy for each
publication that includes the number of copies to be printed, the methods
for announcing availability, and the target audience that will automatically
receive copies. OJJDP promotes products through the
National Criminal Justice Reference Service (NCJRS) Catalog, OJJDP?s
Juvenile Justice journal, the NCJRS and OJJDP Web sites, e- mail lists, the
Office of Justice Programs press announcements, conference displays,
criminal/ juvenile justice newsletters and journals, and flier mailings.
Almost all of OJJDP?s publications are made available to the public through
OJJDP?s Web sites, which is administered by the Juvenile Justice
Clearinghouse. Many publications, depending on their length, are also
available through the Clearinghouse?s fax- on- demand service. Individuals
can also order copies of publications online or by calling the
Clearinghouse?s toll- free number. In addition, the Clearinghouse
automatically sends publications to targeted constituents (e. g., juvenile
justice policymakers, practitioners, researchers, and community- based
organizations) and to individuals who have registered to receive
publications based on their specific areas of interest.
As of May 2001, OJJDP had used this dissemination process to share interim
results from 5 of the 10 ongoing impact evaluations 2 of OJJDP programs that
we assessed. 3 In total, OJJDP had distributed over 400,000 copies of 9
products that contained interim results from the 5 evaluations. Table 6
provides additional information on the distribution of these publications.
1 OJJDP established the Juvenile Justice Clearinghouse to provide
individuals and organizations easy access to information and resources on
juvenile justice topics. The Clearinghouse is a component of NCJRS, a
federally- sponsored information clearinghouse that offers a range of
services including online newsletters, access to publications by partner
agencies, and a calendar of upcoming conferences. Its federal sponsors
include the Office of Justice Programs (including OJJDP) and the Office of
National Drug Control Policy. NCJRS maintains a Web site where publications
by OJJDP and other federal sponsors can be accessed.
2 Although these interim results were based on data collected from impact
evaluations, they did not necessarily comprise impact results. 3 These 10
impact evaluations were ongoing as of April 2001. Appendix II: OJJDP?s
Process for
Disseminating Published Products, Including Interim Results of Impact
Evaluations
Appendix II: OJJDP?s Process for Disseminating Published Products, Including
Interim Results of Impact Evaluations
Page 36 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 6: Dissemination of Published Products With Interim Results of Impact
Evaluations, as of May 2001 Program Product a Automatic
mailing b Other distribution c Other
availability
Juvenile Mentoring Program (JUMP) Juvenile Mentoring Program: 1998 Report to
the
Congress (Report, December 1998). 41,000 7, 100 Online Juvenile Mentoring
Program: A Progress Review (Bulletin, September 2000). 44,000 7, 000 Online
Fax- on- demand FY 2000 OJJDP Discretionary Program Announcement: Juvenile
Mentoring Program (Solicitation, March 2000).
63,700 9, 000 Online Fax- on- demand
Comprehensive Gang Initiative Gang- Free Schools and Communities Initiative:
FY
2000 OJJDP Discretionary Program Announcement (Solicitation, July 2000).
48,700 6, 700 Online Fax- on- demand
Youth Gang Program and Strategies (Summary, August 2000). 45,000 2, 500
Online Intensive Aftercare Implementation of the Intensive Community- Based
Aftercare Program (Bulletin, July 2000). 32,100 2, 500 Online Fax- on-
demand
Reintegrating Juvenile Offenders Into the Community: OJJDP?s Intensive
Community- Based Aftercare Demonstration Program (National Institute of
Justice Research Preview, December 1998).
Not available 664 Online
Fax- on- demand Partnerships to Reduce Juvenile Gun Violence Program
Fighting Juvenile Gun Violence (Bulletin, September 2000). 44,000 2, 300
Online
Fax- on- demand Enforcing the Underage Drinking Laws Program OJJDP Research
2000 (Report, May 2001). 46,000 1, 366 Online
Total 364,500 39,130
a Full citation information for these products is contained in the
bibliography. b ?Automatic mailing? refers to publications disseminated to
targeted and registered individuals and groups. The Juvenile Justice
Clearinghouse maintains 290 targeted mailing lists of key constituents that
include criminal/ juvenile justice policymakers, practitioners, researchers,
and community- based organizations. In addition, individuals can register
with the Clearinghouse to automatically receive information based on their
specific areas of interest. c ?Other distribution? refers to orders for
specific publications. Anyone can order a publication by calling
the toll- free line or using the NCJRS or OJJDP Web site. Source: OJJDP.
Appendix III: Categorical Assistance Progress Report Form Page 37 GAO- 02-
23 OJJDP Grantee Reporting Requirements and Concerns
Appendix III: Categorical Assistance Progress Report Form
Appendix III: Categorical Assistance Progress Report Form Page 38 GAO- 02-
23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 39 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Twice a year, the Office of Juvenile Justice and Delinquency Prevention
(OJJDP) grantees are required to complete a Categorical Assistance Progress
Report- a narrative report that is to include a summary of the status of
their particular projects? goals, quantitative project results based on
performance measures set forth in their grant applications, actions planned
to resolve any implementation problems, and any technical assistance they
might need. In 8 of the 16 major programs we reviewed, grantees received
only this general guidance, and were not subject to any additional reporting
requirements. In the other eight programs we reviewed, grantees were
required to follow this standard guidance and, in addition, report more
specific information. Grantees in all 16 programs reported input, output,
and/ or outcome data related to the process, implementation, and/ or
accomplishments of their projects, such as acquisition of additional funding
for a project evaluation, the number of project participants, or the number
of missing children recovered.
Table 7 provides summary information on the eight programs in which grantees
are not subject to additional reporting requirements and examples from
grantees? progress reports. Table 8 provides similar information regarding
the other eight programs in which grantees are additionally required to
report specified data to OJJDP or outside evaluators, as well as the
specific performance measures on which grantees are required to report.
Unless otherwise noted, the examples of reported information represent
individual grantee or subgrantee data 1 for a 6- month period. Information
provided regarding the specific data on which grantees are required to
report do not necessarily include all performance data required.
1 Examples from OJJDP?s five formula/ block grant programs were typically
reported at the subgrantee or community level. Appendix IV: Descriptions of
OJJDP
Programs, Reporting Requirements, and Examples of Reported Information
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 40 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 7: Eight Programs GAO Reviewed With General Reporting Requirements
Only Program name
Active grantees (FY2000) Program description
Examples of information reported by grantee( s) per general requirements
State Challenge Activities 52 This program, established in 1992 under
Title II, Part E of the Juvenile Justice and Delinquency Prevention Act of
1974, as amended, provides incentive grants to states to improve their
juvenile justice system. States must use funds to develop, adopt, or improve
policies in 10 specified challenge activity areas, including (1) basic
system services, (2) access to counsel, (3) community- based alternatives,
(4) violent juvenile offender facilities, (5) genderspecific policies and
programs, (7) state ombudsman, (8) deinstitutionalization of status
offenders, (9) alternatives to school suspension and expulsion, and (10)
aftercare services/ state agency coordination/ case review system. To
receive these grants, states must also participate in the Title II, Part B
Formula Grants Program. States may subaward State Challenge Activities funds
to communities.
Implemented an aftercare program for state wards returning to their
community that provided community reintegration services including
counseling, employment readiness services, family mediation, and weekly
check- in and monitoring programs.
Hired a disproportionate minority confinement coordinator.
Home visitation research contract delayed due to paperwork complications.
Held weekly meetings with female juveniles identified by the court to
receive counseling and support through a gender- specific services program.
Selected and made subaward to private provider with experience in
developing intake services for youths cited by law enforcement with a focus
on preventing unnecessary placement in attendant care or detention.
Provided quarterly training to all case management staff on more rapidly
and successfully returning youths in state custody from out- of- home
placement into community and in- home placement. Community Prevention Grants
Program (Title V)
54 This program funds collaborative, community- based delinquency prevention
efforts. The program integrates six fundamental principles- comprehensive
and multidisciplinary approaches, research foundation for planning,
community control and decision making, leveraging of resources and systems,
evaluation to monitor program effectiveness, and a longterm perspective-
that combine to form a strategic approach to reducing juvenile delinquency.
Exceeded goal of providing career exploration and training on work
awareness skills to 150 high school students.
Trained 11 students to become mentors.
Provided substance abuse education to six middle school girls.
Experienced difficulties starting a delinquency prevention program to
develop self- esteem and reduce inappropriate school behavior in at- risk
seventh- grade students.
Hired two outreach workers to work with elementary and junior high
schools.
Created an instructional video on quality daycare issues and distributed
it to parents and the community.
Enrolled and ?graduated? 13 children in a 6- week kindergarten preparation
class. Juvenile Accountability Incentive Block Grants Program
54 The Congress established this formula grant program in 1997 to promote
greater accountability among juveniles who are involved in the juveniles
justice system. To qualify for funds, states must demonstrate a commitment
to increasing such accountability. Funds are allocated to states based on
their relative population of youths under age 18, and states must pass on at
Hired a case manager to monitor attendance and grades of juveniles in a
youthful offender program.
A local office declined to participate in the mentoring program and the
subgrantee is considering alternative participants.
Completed a study of achieving a system of graduated sanctions within the
state.
Enrolled 30 youths in a juvenile drug testing program and performed 52
drug screens.
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 41 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program name
Active grantees (FY2000) Program description
Examples of information reported by grantee( s) per general requirements
least 75 percent of their funds to local governments, absent a waiver. Funds
can be used for 12 program purpose areas.
Expanded a drug abuse program for juveniles referred to the court to
include a second group involving both parents and juveniles. To date, the
project has held 3 groups of 6- week classes.
No juveniles completed a 6- week intensive educational program for
substance abuse during the reporting period because of problems starting the
program.
The placement of a full- time police officer at a high school was delayed
due to problems involving the officer?s training.
Contracted with a part- time prosecutor to assist fulltime prosecutor with
juvenile pre- trial and trial services, including compiling case
information, preparing witnesses, and tracking cases. The parttime
prosecutor works an average of 25 cases per week. In total, about 234 cases
have been handled.
Experienced difficulties in recruiting a data analyst due to problems with
applicants meeting the minimum requirements. Enforcing the Underage Drinking
Laws Program (block grant component)
51 The block grant component of the Enforcing the Underage Drinking Laws
Program helps states and the District of Columbia develop comprehensive and
coordinated efforts to enforce state laws that prohibit the sale of
alcoholic beverages to minors and to prevent the purchase or consumption of
alcoholic beverages to those under age 21. Grantees may use funds to support
activities in one or more of three priority areas, including (1) law
enforcement and prosecution efforts that target establishments suspected of
selling alcohol to minors; (2) public advertising campaigns to educate
businesses and youths about the prohibitions against illegal alcohol sales
and purchases; and (3) other innovative programs to prevent and combat
underage drinking.
Formed 15 youth task groups in order to examine the community norms and
messages that young people get regarding alcohol.
Placed billboards to educate the public about alcohol use in 20
communities for 4 weeks.
Produced a 60- second radio spot aimed at preventing underage alcohol use.
Trained 20 community teams about prevention approaches to reduce underage
drinking.
Established a support position to improve coordination and communication
between enforcement, prevention, and treatment systems.
Held a full- day conference for established task force groups.
Experienced implementation problems due to problems with students?
schedules and limitations of the school year schedule.
The local public television station worked with a youth task force group
to design and film a 1- hour documentary about underage drinking. Enforcing
the Underage Drinking Laws Program (discretionary grant component) a
23 The discretionary grant component of the Enforcing the Underage Drinking
Laws Program provides grants to states and territories to support the block
grant component of the same program. Its purpose is to expand the number of
communities taking a comprehensive approach to the problem of underage
drinking, with an emphasis on increasing law enforcement activity regarding
the sale of alcohol to minors. States that receive
Three communities developed a compliance check database form to be used by
law enforcement agencies.
Requested technical assistance for a 1- day training session for local
coalitions on environmentallybased activities.
Published two reports on status of underage drinking in targeted
community.
Held two advisory committee meetings.
Implemented new diversion program for underage alcohol violators and
achieved a 32- percent success rate for youths completing the program.
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 42 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program name
Active grantees (FY2000) Program description
Examples of information reported by grantee( s) per general requirements
these competitive awards select between two and eight diverse jurisdictions
as subgrantees.
Experienced difficulty in bringing one subgrantee on board; the subgrantee
has not yet attended any planning sessions.
Four communities hired part- time program coordinators.
Conducted six monthly conference calls with four communities.
Youth coalition placed 4,000 warning stickers on alcoholic beverages in 13
stores.
Youth coalition designed placemats containing underage drinking facts and
figures for restaurant and grocery store distribution.
Published two newsletters.
Was not able to hold banquet recognizing top 100 businesses that had
discouraged underage alcohol sales due to preparation demands of an
antialcohol youth conference.
Arranged for more than 700 youths to sign a pledge not to drink. Tribal
Youth Program 81 Under this discretionary award program,
federally recognized Tribes can apply for funds to (1) reduce, control, and
prevent crime both by and against tribal youths; (2) provide interventions
for court- involved tribal youths; (3) improve tribal juvenile justice
systems; and (4) provide prevention programs focusing on alcohol and drugs.
The program is intended to provide a flexible funding stream for tribes.
Began keeping juvenile crime data for future analysis of whether targeted
reductions have been achieved.
Drafted court diversion protocol.
Identified tribal judges and court clerks.
Experienced set- backs in establishing four functional tribal courts to
deal with juvenile crime for reasons including the resignation of the
general counsel and competing demands of the commercial and subsistence
fishing season.
Reported conferences and training events attended.
Met with state representatives about possible transfers of cases from
state to tribal court.
Began identifying courtroom equipment needs and researching purchases.
Obtained examples of tribal juvenile codes.
Taught conflict resolution skills to families upon request.
Suffered a set- back in reducing cases of misbehavior and rules violations
in the schools because the community truancy officer resigned and had not
yet been replaced.
Referred five youths for counseling as a result of conflict resolution
curriculum presented in schools. Model Courts Program b 1 This program
awards one earmarked c grant
annually to the National Council of Juvenile and Family Court Judges to
provide training and technical assistance to 23 ?model
courts? around the nation that process child abuse and neglect cases. The
program?s goal is to design and deliver technical
Information reported by grantee through progress reports:
Made 31 on- site staff visits to model courts.
Held 96 national, regional, state, and local training events for nonmodel
court jurisdictions.
Supported the establishment of two model courts.
Trained 9,729 model court and nonmodel court
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 43 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program name
Active grantees (FY2000) Program description
Examples of information reported by grantee( s) per general requirements
assistance and training programs that improve dependency courts? handling of
such cases. Model courts serve as national
?laboratories? for implementing systemswide change in the processing of
abuse and neglect cases. Reforms and initiatives vary across courts. The
Council also supports a mentoring program in which experienced model court
personnel ?mentor? and train personnel from new model courts.
personnel.
Distributed 17,818 technical assistance bulletins.
Conducted 134 telephone consultations.
Distributed 2,341 resource packets. Information reported by grantee
through annual status report: d
Reported one court reduced its caseload of 58, 000 abused and neglected
children in 1995 (prior to program implementation) to 27, 001 children in
1999.
Reported one court reduced the average length of abuse and neglect
complaint cases from 3.5 years in court year 1998 to 2.4 years in court year
1999.
Reported one court established a child protection division with two judges
who hear all child protection cases.
Following this change, the average case length (from petition to
adjudication) for one section of the division dropped from 140 days to
between 29 and 41 days. National Clearinghouse and Resource Center for
Missing and Exploited Children e
1 The National Center for Missing and Exploited Children (NCMEC) is the
national resource center and clearinghouse dedicated to missing and
exploited children?s issues. By law, NCMEC is to (1) establish a toll- free
hotline, (2) provide information on free or low- cost transportation for
missing children and their families, (3) coordinate programs that reunite
missing children with their families, (4) disseminate information that
benefits missing and exploited children, and (5) provide training and
technical assistance to law enforcement and other agencies.
Upgraded toll- free hotline.
Received 24,983 hotline calls.
Assisted in recovering 1,610 missing children.
Described missing children cases in which NCMEC assisted.
Distributed 5,306 pictures of missing children.
Described public awareness campaigns.
Trained 1,708 law enforcement and health care personnel.
Described joint projects with other agencies.
Completed 48 age progressions of photographs.
Received 5,291 child pornography tips online, and 492 tips through the
hotline.
Described technical assistance provided to other agencies. a This program
was established in fiscal year 1998 under the name Combating Underage
Drinking.
b OJJDP has emphasized to the single grantee for this program that it should
include quantitative performance data in its progress reports, but did not
prescribe the specific indicators on which the grantee must report. c
Earmarked grants are those for which the Congress sets aside a specified
amount for an identifiable
grantee and/ or a specific authorized purpose. d The grantee chooses to
provide performance data regarding model court activities and
accomplishments in an annual status report that is available to nonmodel
courts and to the general public. e Although the grantee in this program is
only required to submit progress reports semiannually, it chooses to submit
progress reports on a quarterly basis. Examples presented in this table were
excerpted from a recent quarterly progress report.
Source: GAO analysis of OJJDP and grantee data.
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 44 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 8: Eight Programs GAO Reviewed With Specific Reporting Requirements
and General Reporting Requirements Program name
Active grantees (FY2000) Program description
Specific data required from and reported by program grantee( s)
Examples of information reported by grantee( s) per general requirements
Formula Grants Program 54 OJJDP has administered this
program since 1975. The program provides funds to help states, U. S.
territories, and the District of Columbia implement a comprehensive state
juvenile justice plan based on a detailed study of needs to support state
and local delinquency prevention and intervention efforts and juvenile
justice system improvements. To be eligible for a formula grant, a state
must designate an agency to prepare a 3- year plan, establish a State
Advisory Group, and commit to complying with the four core requirements of
the Juvenile Justice and Delinquency Prevention Act of 1974, as amended.
Grantees (states) must report data including but not limited to the
following:
Number of accused status offenders and nonoffenders held in any secure
detention or correctional facility for more than 24 hours.
Number of accused status offenders and nonoffenders securely detained in
any adult jail, lockup, or nonapproved collocated facility for any period of
time.
Number of accused status offenders and nonoffenders placed in facilities
not near their home, not the least restrictive appropriate alternative, and
not community- based.
Number of juvenile offenders and nonoffenders not separated from adult
criminal offenders in facilities used for secure detention and confinement
of both juveniles and adults.
Number of juveniles detained in collocated facilities not approved by the
state that were not sight and sound separated from adult criminal offenders.
For 21 youths involved in a truancy abatement program, school attendance
increased by an average of 89 percent.
A runaway youth program provided counseling to 129 runaway youths and
their families; of 57 cases closed, 86 percent of youths returned to home, a
friend, or a relative.
Discontinued a violence prevention curriculum for middle and high school
students and a conflict resolution training program for eighth- grade
students due to the programs? lack of effectiveness.
Twenty- six status offenders and their families participated in family
counseling activities that focused on parenting skills, communication, and
prosocial behaviors.
A mentoring program has continued to have difficulty in recruiting
appropriate mentors.
Enrolled 40 youths pending adjudication and/ or disposition in home
detention programs in lieu of incarceration, shelter, or detention
placement.
Despite efforts, was unable to gather and analyze data on youths
participating in a tutoring program due to a lack of school system
cooperation. Comprehensive Gang Initiative 2 This program assists
community coalitions in implementing and testing the comprehensive gang
model- a collection of strategies aimed at the prevention, intervention, and
suppression of youth gang crime and violence. The model requires communities
to assess local gang problems, identify key risk factors among youths,
develop program plans, and
Number of juveniles served.
Age, sex, and income level of participating youths.
School participation and academic performance for participating youths.
Criminal activity, gang activity, drug use, and alcohol use by
participating youths.
Project staff?s perceptions of the seriousness of gang and nongang
problems in
Received university grant to fund local project evaluation.
Developed action team to address problem of female gang members.
Project youths were not participating in tattoo removal program; project
staff need to better promote benefits of tattoo removal.
Recruited five new community leaders to join project.
Held biweekly staff meetings to discuss progress of participating
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 45 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program name
Active grantees (FY2000) Program description
Specific data required from and reported by program grantee( s)
Examples of information reported by grantee( s) per general requirements
evaluate program outcomes. Coalitions focus on five core strategies:
community mobilization, academic and economic opportunities, social
intervention, gang suppression, and organizational change and development.
the community.
Nature and frequency of services provided to youths and families.
Youth workers? (probation officers, case workers) perceptions of the
effectiveness of services provided to youths and families.
youths.
Inadequate staffing levels caused delays in processing higher than
expected numbers of youth referrals and prevented project expansion to
additional neighborhoods.
Revised staff procedures for delivering and coordinating warrants.
Recruited 25 girls for selected projects. Rural Gang Initiative 4 A spin-
off of the
Comprehensive Gang Initiative that adapts the comprehensive gang model to
rural communities. Two sites have incorporated the full version of the model
and two sites follow a modified version of the Comprehensive Gang
Initiative.
Aggregated demographic data for community (unemployment rates, education
attainment, race).
Community attitude data (community?s perception of gang problem).
Youth attitude data (perceptions about ganginvolved youths from at- risk
youths).
Law enforcement crime data (data on nature of criminal gang activity in
community- number of assaults, property crimes, etc.)
Community resource data (inventory of services available to gang and
community members- e. g., conflict resolution, individual counseling, etc.)
Experienced delay in hiring a project coordinator and outreach staff due
to a lack of qualified applicants; project is preparing for second
recruitment effort.
Developed steering committee composed of representatives from community
organizations involved with youths.
Attended training on implementation of the comprehensive gang model.
Located and rented a building to house project staff.
Insufficient staff to follow up on court orders against targeted youths
who were at- risk for joining a gang.
In the process of forming an intervention team composed of law
enforcement, school, and social services personnel.
Hired a gang crime specialist.
Computer failure prevented police department from providing crime data.
Juvenile Mentoring Program (JUMP)
175 JUMP provides one- to- one mentoring for youths at risk of delinquency,
gang involvement, educational failure, or dropping out of school. Awards are
made to local educational agencies or private or nonprofit agencies that
have clearly defined relationships with local schools. Grantees must conduct
thorough background checks on volunteer mentors to ensure a safe environment
Youth information:
Demographic information, such as age, gender, and ethnicity.
Living situation.
Medical or mental health problems.
Total number of school absences. Mentor information:
Demographic information such as age, gender, and ethnicity.
Reason for becoming
Exceeded goal of matching 50 atrisk girls with mentors.
Mentors and mentees participated in classes highlighting other cultures.
Retained 50 percent of mentors from the previous year.
Experienced problems with parents who were unwilling to complete project
requirements, such as interviews and home visits.
Continued to recruit mentors from local businesses.
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 46 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program name
Active grantees (FY2000) Program description
Specific data required from and reported by program grantee( s)
Examples of information reported by grantee( s) per general requirements
for each child. JUMP also emphasizes projects that are established in
communities where 60 percent or more of the youths qualify to receive a free
or reduced- price lunch.
involved in the mentoring program.
Highest education completed.
Previous mentoring experience.
Twenty out of 26 mentors who responded to a survey said the program met
their expectations and 23 said they would participate again.
Twenty- five youths dropped out of a project. SafeFutures 6 This program,
based on
OJJDP?s comprehensive strategy model, targets communities that have already
begun to reform their juvenile care systems. Program goals include
developing a full range of graduated sanctions, ensuring community safety,
and providing treatment services for juveniles. Communities use one grant to
tap into nine OJJDP program components (e. g., juvenile mentoring,
aftercare, mental health, delinquency prevention, gang programs, etc.)
Demographic data (age, race, sex) for youth participants.
Family/ household characteristics (employment, number of family members,
number of juveniles/ adults living at home, school history, family rules)
for youth participants.
Self- reported data by youth participants (neighborhood characteristics,
gang affiliation, number of delinquent acts, characterization of family
relationships, drug and alcohol abuse, emotional problems, participation in
project activities, drug and alcohol abuse by family members).
Developed and submitted applications for additional funding.
Experienced problems providing substance abuse treatment services due to
the lack of a fulltime, trained substance abuse counselor.
Hired a new local evaluator.
Referred 249 juveniles for direct services and care coordination.
Completed 13 community service projects involving 394 students and
families.
Experienced problems implementing a mentoring program- i. e., recruiting
mentors, retaining matches, and motivating mentors to attend sessions.
Continued coordination between law enforcement agencies on gang
intelligence resulted in the arrest of two arson suspects.
Installed a computer system to track gang members and gangrelated
activities in the community. Drug- Free Communities Support Program
307 This program is intended to increase citizen participation and
strengthen community anti- drug coalition efforts to reduce substance abuse
among youths in communities throughout the United States and, over time, to
reduce substance abuse among adults.
Number of organizations represented on the grantee?s coalition.
Types of organizations represented on the coalition.
Number of individuals who are coalition members.
Number of youths, parents, and community leaders who are coalition
members.
Number of recruitment activities held.
Steps taken to engage members in drug prevention planning.
Activities sponsored by the coalition.
Number of participants in activities.
Exceeded goal of 100 community organizations participating in workshops.
Produced anti- smoking commercial.
Invited health professionals to a strategic planning meeting.
Staff worked with drug- free clubs.
Hired first full- time staff.
Printed quarterly newspaper.
Distributed 20,000 copies of a family resource directory.
Experienced low attendance at a strategic planning meeting.
Provided drug prevention education to 78 after- school detention students.
Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and
Examples of Reported Information
Page 47 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program name
Active grantees (FY2000) Program description
Specific data required from and reported by program grantee( s)
Examples of information reported by grantee( s) per general requirements
Target population for activities. Internet Crimes Against Children Task
Force Program
50 This program encourages communities to develop regional, multi-
jurisdictional, and multi- agency responses to online enticement and child
pornography cases. These task forces provide forensic, prevention and
investigative assistance to parents, law enforcement, prosecutors,
educators, and other professionals working on child victimization issues.
Grantees in this program must complete a monthly performance measures report
that collects information including, but not limited to:
Number of arrests.
Number of child pornography producer cases.
Number of child pornography seller cases.
Number of search warrants.
Number of subpoenas.
Number of referrals from other agencies.
Number of law enforcement personnel trained.
Number of presentations given to students, parents, teachers, citizens or
the media.
Provided summaries of 24 cases during the reporting period.
Hired a child victim service specialist.
Achieved goal of implementing training seminars for local, states and
federal law enforcement officers.
Acquired necessary equipment for investigating online offenders such as
desktop and laptop computers, cameras, and investigative software.
Made presentations on Internet safety to teachers, parents, community
groups, and schoolage children.
Children?s Advocacy Centers
5 This program provides training and technical assistance to local
children?s advocacy centers and promotes their expansion nationwide.
Children?s advocacy centers coordinate law enforcement and protective
services responses to victims of child abuse through multidisciplinary
teams. Under this program four regional centers and one national association
provide training and technical assistance to local centers; the national
association also awards subgrants to local centers for training, program
development, and other purposes.
Training and technical assistance grantees are required to report on:
Number of training sessions conducted.
Number of practitioners trained.
Number of technical assistance requests filled.
Number of consultations (on- site, electronic, telephone) provided.
Number of publications developed and distributed. National grantee is
required to report subgrantee data on:
Number of new and developing local centers.
Number of on- site monitoring visits.
Number of grant applications approved.
Type of grants approved (program development, program support, training,
and state chapter).
Training and technical assistance grantees:
Finalized a diversity plan to ensure that children receive services in a
culturally appropriate setting.
Experienced difficulties agreeing on research methods to evaluate the
children?s advocacy centers? impact on case outcomes.
Developed a ?multidisciplinary
team development? curriculum.
Identified and provided assistance to underserved rural communities.
National grantee:
Reviewed subgrantees? biannual reports on activities held during the
reporting period.
Unexpected resignation of grants coordinator hampered data collection
efforts and the awarding and processing of grants.
Approved 409 applications for subgrants and notified recipients.
Surveyed local accredited centers and found that 49,350 juveniles had been
served during the reporting period.
Source: GAO analysis of OJJDP, grantee, and evaluator data.
Appendix V: Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees
Page 48 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
The Office of Juvenile Justice and Delinquency Prevention?s (OJJDP) training
and technical assistance programs and research programs are unique in that
they cut across many of OJJDP?s other programs. Also, grantees 1 in each of
these two areas typically report the same types of quantitative performance
data as other grantees in their area, even though OJJDP does not usually
prescribe the specific performance measures on which the grantees should
report. Training and technical assistance grantees maintain the same types
of data due to the common support services they provide, and research
grantees do the same because they share a common goal of producing research
products.
OJJDP awards grants to training and technical assistance providers to
support grantees in many of OJJDP?s grant programs. OJJDP administers the
vast majority of its training and technical assistance grants through three
of the Office?s divisions: (1) the Training and Technical Assistance
Division (TTAD), (2) the State and Tribal Assistance Division (STAD), and
(3) the Child Protection Division (CPD). 2 Many of the training and
technical assistance providers are required to report information on their
projects? activities and accomplishments semiannually using OJP?s
Categorical Assistance Progress Report form, as do all OJJDP grantees. 3 OJP
provides standard guidance on information to be reported, such as
information on the status of each of the grantees? project goals and
quantitative results of their projects. STAD has not imposed additional or
more specific reporting requirements on its training and technical
assistance providers and, for the most part, neither have TTAD 4 nor CPD. 5
1 OJJDP awards grants, cooperative agreements, and contracts to training and
technical assistance providers, and award grants and cooperative agreements
to research providers. Officials commonly refer to all of these as ?grants?
and to all award recipients as
?grantees.?
2 In commenting on a draft of this report, the Assistant Attorney General
noted that these divisions do not always provide the same types of services.
3 All training and technical assistance providers that receive grants are
required to complete progress reports. Providers that are under contract or
have cooperative agreements with OJJDP must also report on their
accomplishments, but not necessarily through a progress report. Often, they
are required to report their activities on a monthly basis.
4 In commenting on a draft of this report, the Assistant Attorney General
noted that TTAD requires Juvenile Accountability Incentive Block Grants
Program grantees to submit monthly reports. These grantees comprise
approximately one- fifth of all TTAD grantees and account for approximately
$5 million annually. Appendix V: Training and Technical
Assistance and Research Performance Data Reported by OJJDP Grantees
Performance Data for Training and Technical Assistance Grantees
Appendix V: Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees
Page 49 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Officials explained that OJJDP does not require all grantees to routinely
report prescribed data because it is reluctant to place additional reporting
requirements on grantees due to the Paperwork Reduction Act of 1995, 6 which
set goals to reduce the federal government?s reporting and paperwork burden.
Although most of these providers are not subject to additional reporting
requirements for prescribed data, it is not unusual for them to report on
the same or similar quantitative performance measures. Because of the nature
of the services they provide, training and technical assistance providers
tend to maintain like data that can readily be counted, such as numbers of
training events held, practitioners who attended those events (?
practitioners trained?), and technical assistance requests filled. Some of
these providers also produce publications or materials, such as bulletins,
surveys, curricula, brochures, and other support materials, and report such
information to OJJDP. Table 9 summarizes performance data we obtained
regarding training and technical assistance grants. 7
5 CPD has prescribed additional, specific reporting requirements for
providers in two of its programs- the Children?s Advocacy Centers and the
Missing and Exploited Children Training and Technical Assistance program-
such as reporting the number of training events held.
6 The Paperwork Reduction Act applies to identical reporting requirements
imposed on 10 or more parties. CPD?s training and technical assistance
providers that have specific reporting requirements are not affected by the
Act since they involve less than 10 providers.
7 We attempted to obtain fiscal year 2000 data from all grantees; however,
in some cases OJJDP provided calendar year 2000 or fiscal year 2001 data due
to a lack of available or representative data. Because we obtained
publications data that pertained only to some of the grantees, we did not
include such information in the table. For example, State Challenge
Activities training and technical assistance providers reported producing 10
bulletins in fiscal year 2001and Juvenile Accountability Incentive Block
Grant providers reported that in fiscal year 2000 they developed or were
developing 628 supportive materials, such as reports, curricula, and
newsletters.
Appendix V: Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees
Page 50 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 9: Training and Technical Assistance Performance Data Reported by
OJJDP Grantees
Division/ program supported Training events Practitioners trained
Technical assistance
requests filled a Training and Technical Assistance Division (FY2000)
Juvenile Accountability Incentive Block Grants Program 227 13,816 1, 374
Life Skills Training b 64 858 16 National Training and Technical Assistance
Center c d d 329 Other programs e f 36,000 1, 633 g
State and Tribal Assistance Division (FY2001) h
Community Prevention Grants Program (Title V) 55 2,171 4 Formula Grants
Program 10 1, 148 265 State Challenge Activities 2 69 71 Comprehensive
Strategy for Serious, Violent, and Chronic Juvenile Offenders
76 2,252 232
Child Protection Division (Calendar Year 2000)
Children?s Advocacy Centers 183 10,939 6, 501 i Court Appointed Special
Advocate Program 194 5,312 21,323 Model Courts Program 161 42,537 401
Permanency Planning 124 38,888 4, 264 Prosecutor Training f 9,300 1,800
Parents Anonymous 3 149 f Missing and Exploited Children Training and
Technical Assistance program j
68 2,950 f a Data can include the number of in- depth, on- site visits and/
or the number of responses to telephone and e- mail requests. b Data for
this program were from the academic year 2000- 2001, which roughly
corresponds with fiscal
year 2001. c The Center receives requests from the field for technical
assistance, which it then fills in- house or refers to other technical
assistance providers. In fiscal year 2000, it either filled or referred 329
requests for technical assistance. The Center does not provide direct
training to the field. d Not applicable.
e Data reported from 19 other training and technical assistance grantees. f
Data not available. g Reflects the number of communities that received
technical assistance.
Appendix V: Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees
Page 51 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
h At the suggestion of OJJDP officials responsible for the following
programs, data for the Community Prevention Grants Program are prorated from
the first 9 months of the fiscal year, training and technical assistance
data for State Challenge Activities are prorated from the first 8 months of
the fiscal year, and data for the Comprehensive Strategy for Serious,
Violent, and chronic Juvenile Offenders are prorated from the first 6 months
of the fiscal year. i Includes data from the national grantee and three of
the four regional grantees.
j Data are for fiscal year 2000. Source: GAO analysis of OJJDP and grantee
data.
OJJDP officials cautioned that not all providers share common definitions of
?training? and ?technical assistance.? 8 For one thing, the difference
between the two is not always clear and, therefore, it is sometimes
difficult to definitively categorize a provided service as training versus
technical assistance. Furthermore, not all training events are equal. For
example, some providers might characterize both a 1- day training conference
and a 10- day training workshop as a training event; others might
differentiate between the two. Furthermore, one provider might consider a
telephone request from a grantee as merely a query, while another might
consider it a request for technical assistance.
OJJDP administers its research grants out of its Research and Program
Development Division (RPDD). 9 RPDD sponsors empirical studies on an array
of topics related to juveniles and delinquency, from the causes of violence
to the impact of victimization. The overall goal of these research grants is
to generate credible and useful information to help prevent and reduce
juvenile delinquency and victimization. Research grantees are not only
expected to collect data but to analyze and disseminate their analyses to
the public. RPDD requires all research grantees to produce publishable
products and, in some instances, RPDD specifies the type of products to be
published depending on the results of the research. Thus, according to OJJDP
officials, one measure of a research grantee?s performance is the number of
products the grantee has published.
Like most OJJDP grantees, research grantees must report information on their
projects? activities and accomplishments semiannually through
8 In commenting on a draft of this report, the Assistant Attorney General
pointed out that TTAD published the Training, Technical Assistance, and
Evaluation Protocols booklet 2 years ago, which defined both training and
technical assistance.
9 Some grants administered through other OJJDP divisions have a research
component, but they are not considered ?research grants.? Performance Data
for
Research Grantees
Appendix V: Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees
Page 52 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
progress reports. RPDD does not impose additional, specific reporting
requirements on grantees, but it does encourage them to report on products
produced through private publishers (as opposed to those published through
OJJDP). 10 The division director told us that it is not necessary to impose
specific requirements on grantees in addition to the semiannual progress
report requirements because officials work closely with grantees throughout
the life of the grants.
OJJDP research grantees produce products based on their OJJDP- funded
research. Some of these products are approved and published by OJJDP, who in
turn disseminates the products through its own distribution process (see
app. II for a description of OJJDP?s product dissemination process).
Grantees also publish many products that are based on their OJJDP- funded
research through private publishers. OJJDP officials told us they give their
research grantees latitude to privately publish products because the
majority of their research grantees are academics whose funding depends on
the number of products they publish, and because grantees often have funding
sources in addition to OJJDP.
Tables 10 and 11 summarize the products that active OJJDP research grantees
published through OJJDP and private publishers as a direct result of OJJDP-
funded research. Table 10 describes the number of research products
published by OJJDP from 1993 through September 2000, by topic. Table 11
shows the number of products, by topic, that grantees with active research
grants privately published between 1986 and June 2001, or were in the
process of publishing in June 2001.
10 Officials told us that they ask grantees to submit products for review
prior to external publication, but often grantees do not do this. In the
past, grant applicants have turned down funding when it was contingent on
OJJDP?s review of privately published products, citing the importance of
academic freedom. Also, grantees oppose submitting external products to
OJJDP prior to publication because many academic journals will not publish
research findings that have been reviewed or disseminated prior to
submission.
Appendix V: Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees
Page 53 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 10: Active OJJDP Grantees? Research Products, Published by OJJDP Topic
Summaries Reports Fact sheets Bulletins
Aftercare (post- detention) 2 1 1 3 Causes and Correlates of Delinquency a 1
2 46 Crimes against juveniles 0 0 0 7 Delinquency prevention 1 5 1 6
Detention/ corrections 3 2 7 2 Gangs 5 0 4 8 Juvenile courts 1 3 9 4
Juvenile health 0 1 1 2 Juvenile justice system reform 1 2 0 2 Juvenile
offenders 3 3 5 10 School safety 0 2 0 0 Other 0 1 1 0
Total 17 22 33 50
Note: Table includes some products published as a result of OJJDP evaluation
and statistics grants. a This topic is comprised of products from Causes and
Correlates of Delinquency grants. In commenting on a draft of this report,
the Assistant Attorney General noted that these products can be categorized
under a number of the topics listed.
Source: GAO analysis of OJJDP data.
Table 11: Active OJJDP Grantees? Research Products, Published Externally
Topic Journal
articles Book chapters Other publications a
Causes and Correlates of Delinquency b 57 43 17 Crimes against juveniles 12
c 6 2 Delinquency prevention 0 0 2 Detention/ corrections 3 0 0 Gangs 13 d 9
2 Juvenile courts 5 1 1 Juvenile health 51 e 5 7 Juvenile offenders 3 0 2
School safety 16 0 4 Other 0 0 5
Total 160 64 42
Note: Table includes some products published as a result of OJJDP evaluation
and statistics grants. a Includes books, reports, and other publications.
b This topic is comprised of products from Causes and Correlates of
Delinquency grants. In commenting on a draft of this report, the Assistant
Attorney General noted that these products can be categorized under a number
of the topics listed.
Appendix V: Training and Technical Assistance and Research Performance Data
Reported by OJJDP Grantees
Page 54 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
c Includes one paper submitted but not yet accepted for publication. d
Includes one paper presented at an academic conference. e Includes three
papers submitted but not yet accepted for publication.
Source: GAO analysis of OJJDP and grantee data.
Appendix VI: States? Compliance With Core Requirements of the Juvenile
Justice and Delinquency Prevention Act of 1974
Page 55 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Table 12: State- Reported Compliance with Core Requirements, Calendar Year
1998 Core requirements Number of
jurisdictions Jurisdictions a Deinstitutionalization of status offenders
Full compliance - zero violations 9 ME, MT, NY, VT, AS, GU, MP, PR, VI Full
compliance - de minimis exceptions b 39 AL, AK, AR, CA, CO, CT, DE, DC, FL,
GA, HI, ID, IL, IN,
IA, KS, KY, LA, MD, MA, MN, MO, NE, NV, NH, NM, NC, ND, OK, OR, PA, RI, TX,
UT, VA, WV, WI Not in compliance 4 MS, SC, TN, WA Funds withheld pending
additional compliance data 2 NJ, OH Not participating 2 SD, WY
Separation of juvenile and adult offenders
Full compliance - zero violations 41 AL, AZ, AR, CA, CO, CT, DE, DC, FL, IL,
IN, KS, KY, ME, MD, MA, MI, MN, MO, MT, NE, NV, NH, NM, NC, ND, OK, OR, PN,
RI, SC, UT, VT, VA, WA, WI, AS, GU, MP, PR, VI Full compliance - exception
provision 11 AL, GA, HI, ID, IA, LA, MS, NY, TN, TX, WV Funds withheld
pending additional compliance data 2 NJ, OH Not participating 2 SD, WY
Jail and lockup removal
Full compliance - zero violations 11 AL, DC, NC, OR, SC, VT, AS, GU, MP, PR,
VI Full compliance - de minimis exceptions 38 AZ, AR, CA, CO, CT, DE, FL,
GA, HI, ID, IN, IA, KS, KY,
LA, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NM, NY, ND, OK, PA, RI, TN, TX,
UT, VA, WA, WV, WI Not in compliance 3 AK, IL, ME Funds withheld pending
additional compliance data 2 NJ, OH Not participating 2 SD, WY
Disproportionate minority confinement c
Completed identification and assessment, implementing intervention and
monitoring 21 AR, CO, CT, FL, GA, ID, IN, IA, LA, MO, NV, NC, ND, OK,
PA, SC, TN, UT, VA, WA, WV Completed identification and assessment,
implementing intervention 15 AK, AZ, HI, IL, KS, MD, MA, MI, MN, NE, NM, NY,
RI, TX,
WI Completed identification and assessment/ updating data and revising
intervention plan 2 CA, D E Completed identification/ implementing
intervention/ conducting formal assessment 3 AL, MS, MT Conducting
identification phase 1 NH Exempt from requirement 7 ME, VT, AS, GU, MP, PR,
VI Status under review 5 DC, KY, NJ, OH, OR Not participating 2 SD, WY
Note: OJJDP based its initial determination of states? eligibility to
receive fiscal year 2000 Formula Grants on 1998 monitoring reports- the most
recent data available as of March 2001. a AS = American Samoa; GU = Guam; MP
= Northern Mariana Islands; VI = Virgin Islands.
b Fewer than 29.4 violations per 10,000 persons under age 18 in the state. c
States reported data on disproportionate minority confinement in their
fiscal year 2000 comprehensive plans for compliance.
Appendix VI: States? Compliance With Core Requirements of the Juvenile
Justice and Delinquency Prevention Act of 1974
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 56 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
This appendix contains information on the 10 impact evaluations that the
Office of Juvenile Justice and Delinquency Prevention (OJJDP) has funded of
its own programs since 1995 and for which we have assessed the
methodological rigor, as well as information on one impact evaluation- Teen
Courts- that we did not assess. Five of the 10 evaluations are in their
formative stages, and five are well into their implementation. For each of
the 10, we have included a description of the program being assessed, the
evaluating organization, a description of the evaluation and its findings,
and our assessment of the evaluation. 1 As discussed in the Scope and
Methodology section of this report, we did not assess the methodological
rigor of the Teen Courts evaluation. However, we have included a summary of
this evaluation at the end of this appendix. 2
Program Description: Parents Anonymous is a national child abuse prevention
program that began in 1970. It consists of 32 state and local organizations
and over 1,000 weekly mutual support groups. The principal participants are
at- risk parents, though complementary projects exist for children. The
cornerstones of the program are mutual support and shared leadership.
Evaluator: National Council on Crime and Delinquency.
Evaluation description: This evaluation, which is in the beginning stages,
is based on a proposal to conduct a process evaluation in year 1, and an
outcome evaluation in years 2 and 3. The researchers will determine how
Parents Anonymous is staffed and operated in different settings, how it
attempts to change the behavior and attitudes of parents, and what factors
are related to its effectiveness. While the specifics of an outcome
evaluation design are yet to be determined, the researchers
1 We relied on documents OJJDP provided to us in April 2001 in assessing the
evaluation methodologies and reporting on each evaluation?s status. 2 OJJDP
provided us with this evaluation description. Appendix VII: Impact
Evaluations OJJDP Has
Funded of Its Own Programs Impact Evaluations of OJJDP Programs in Their
Formative Stages
Parents Anonymous
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 57 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
indicate that they will most likely compare the Parents Anonymous
participants with a control group and with Parents Anonymous dropouts. The
process evaluation received $300, 000 for a 3- year period.
Evaluation findings: It is too early in this evaluation to have reported
results.
GAO assessment: No assessment of the impact evaluation is possible because
it has not yet been planned. More fully developed proposals will need to be
made to OJJDP to obtain funding for the impact evaluation portion of the
study.
Program Description: This program replicates and evaluates Project PATHE,
which was first implemented in the Charleston County School District in
South Carolina between 1980 and 1983. Project PATHE is a comprehensive
school- based program that combines services to students who are at elevated
risk for developing problem behaviors with schoolwide organizational changes
intended to improve both school climate and students? behavior. Local
educators are encouraged to develop their own (1) explanations for the
causes of their schools? violence and behavior problems and (2) specific
local objectives and ways to prevent these problems by empowering teachers?
decision- making and fostering collaborative and nonhierarchical efforts.
For various reasons, the grantee has had difficulty in selecting a school
district for the replication. Funding for this effort began in October 1999
with funds provided by the Centers for Disease Control and Prevention.
In commenting on a draft of this report, the Assistant Attorney General
pointed out that all funds for this effort come from the Centers for Disease
Control and Prevention through an interagency agreement. OJJDP awarded this
grant, which the two agencies jointly manage. OJJDP originally identified
Project PATHE as an OJJDP- funded program. However, on the basis of the
Assistant Attorney General?s comments, this program appears to be a non-
OJJDP- funded impact evaluation. Since we assessed its methodological rigor,
we have included it with the other OJJDP- funded evaluations.
Evaluator: Institute of Behavioral Science, University of Colorado.
Evaluation description: Although the impact evaluation was expected to be
completed in July 2001, it has yet to begin. Original plans called for one
school district to be selected for the replication and evaluation. In a
Positive Action Through
Holistic Education (Project PATHE)
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 58 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
school district, one high school and one middle school would be selected to
receive the program, once school principals had been informed and staff
surveys had been conducted to determine interest in participating.
Comparison schools (one high school and one middle school) in the same
district would be selected with similar demographic characteristics of
students, levels of problem behaviors, and other unspecified organizational
characteristics, as well as a low probability of mounting other school- wide
efforts to reduce problem behavior during the study period. Plans are to
collect data before and after project implementation. All students and staff
in all schools would be surveyed in September and May for 3 consecutive
school years. In addition, 200 students from each high school- 100 seniors
and 100 sophomores- and 200 students from each middle school- 100 eighth
graders and 100 sixth graders- would be sampled in the first year of the
study for followup for 3 years. Each sample is to include 25 students
identified as ?high risk.? It is not clear, however, how these samples will
be selected. Schools are to be visited three times yearly during the study,
and school records and teacher ratings of student behavior in each school
will be used in establishing differences between the program and comparison
schools. Program outcomes are to be selected after determining goals and
objectives collaboratively with local school officials. Multivariate
statistical analyses, such as logistic regression, are planned. Because of
the difficulty in selecting a school district the latest progress report
indicates some changes in this design. Agreements have been signed with two
school districts (Charleston, SC and Baltimore, MD), instead of one, and
plans are to conduct the replication and evaluation in four middle schools
(two program and two comparison schools) in each district- high schools have
been excluded. The amount of the grant is $875,000. However, additional
funds have been requested.
Evaluation findings: It is too early in this evaluation to have reported
results.
GAO assessment: The evaluation, as designed, is basically sound. The
variation in program structure and implementation between schools may limit
generalizability. While the researchers do suggest awareness of potential
problems due to students switching schools, they do not clearly indicate, at
this early design stage, how possible contamination will be handled.
Program Description: The Rural Gang Initiative is a comprehensive strategy
to ameliorate gang problems in rural areas. The program was Rural Gang
Initiative
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 59 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
adapted from the comprehensive gang model, developed at the University of
Chicago, and implementation began in two rural areas in fall 2000. The
program consists of five elements: community mobilization, opportunities
provision, social intervention, suppression, and organizational change and
development.
Evaluator: National Council on Crime and Delinquency.
Evaluation description: The impact evaluation of this program began in
January 2001 and is expected to be completed in December 2003. It focuses on
two of the four sites, Mount Vernon (IL) and Glenn County (CA), that were
part of a year- long feasibility study that began in April 1999. The other
two sites are not part of the evaluation because they did not fully
implement the model. 3 Information will be collected from these two sites on
gang- involved youths and youths at- risk of gang involvement, all of whom
have participated in the program. However, it is unclear how the youths will
be sampled or whether all participants will be included. Data will be
obtained from a variety of sources, including interviews, organizational
surveys, and administrative data from schools and the justice system. No
comparison groups are planned, though the researchers indicate that attempts
will be made to collect data that will permit an assessment of alternative
explanations for program effects. The researchers plan to collect data
before and after program implementation; then they will measure any changes
and follow- up with program participants for at least 12 months after their
participation in the program. Individual sites are to identify specific
program outcomes. Although the researchers have described prospective
outcome measures, such as the reduction of gang- related crime and the
prevention or reduction of gang involvement, they (1) have not chosen the
outcome measures that they will use and (2) have not provided information on
the types of statistical analyses planned. However, the evaluation is still
in the formative stages. At the time of our review, this evaluation had
received $525,000 in funding.
Evaluation findings: It is too early in this evaluation to have reported
results.
3 In commenting on a draft of this report, the Assistant Attorney General
pointed out that these two sites did not implement the comprehensive gang
model because their gang crime problem did not warrant the model?s
prescribed intervention approach.
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 60 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
GAO assessment: It is too early to tell from this evaluation how effective
it will be. However, the absence of plans for comparison groups and the lack
of specificity regarding any other control mechanisms, make it unclear how
program effects will be distinguished from alternative explanations at this
stage of the evaluation.
Program Description: The Safe Schools/ Healthy Students program has funded
77 school districts nationwide, with grants ranging up to $3 million, to
develop services and activities to promote healthy childhood development and
prevent violence and drug abuse. The program also aims to develop greater
collaboration and cooperation between communities and schools to enhance
their effectiveness in responding to and reducing violence. Each project
model is intended to evolve over time.
Evaluator: Research Triangle Institute.
Evaluation description: This impact evaluation began in October 1999 and
data will be collected through the Spring of 2005. School and community-
based archival records, surveys of key coalition personnel, teachers,
superintendents, principals and other school staff, and teacher behavioral
checklists for students in selected grades will be gathered in all 77 school
districts. The evaluation will compare data from participating sites with
national norms and with similar information from matched nonparticipating
(comparison) sites that the researchers surveyed in each of two large,
nationally representative studies of school districts. The matching will be
based on unspecified socio- demographic characteristics and responses to
policy- related questions in the baseline survey. Archival data will be
collected yearly over a 5- year period. Survey data will generally be
collected at three points in time, about 2 years apart. The survey items
will be drawn from established instruments and will provide, in conjunction
with the archival or administrative data, information on behavioral
outcomes, risk factors and inhibiting factors, and indicators of positive
development and mental health. At the time of our review, this evaluation
was funded at approximately $5.6 million.
Evaluation findings: It is too early in this evaluation to have reported
results.
GAO assessment: This evaluation, as designed, is basically sound. Safe
Schools/ Healthy
Students
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 61 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program Description: This demonstration program seeks to prevent and reduce
the impact of family and community violence on young children, primarily
aged 0 to 6, in up to 12 communities. The program plans to create a
comprehensive service delivery system that integrates service providers (in
the fields of early childhood education/ development, health, mental health,
and all manner of prevention, intervention, and treatment programs), law
enforcement, legal services, and the courts. It also seeks to improve the
access, delivery, and quality of services to children exposed to, and at
high risk of, violence. Project sites are to be selected through a
competitive grant process. Funding for the program began in October 1999.
Evaluator: Caliber Associates.
Evaluation description: This evaluation began in May 2000, and is expected
to end in September of 2005. The effect of the Safe Start Initiative will be
measured within and across all participating communities at both the
community and individual levels. Multiple data collection methods, including
focus groups, service agency usage logs and documents, and random- digit-
dialing telephone surveys, will be used. Plans are to collect data before
program implementation and for 4 years after the program begins in each
site. Although specific outcome measures have not yet been identified, they
are expected to address such areas as agency referral levels and quality of
service, increased interagency collaboration, knowledge and perceptions of
police and child protective services, rates of child maltreatment, physical
injuries, and mental health problems. No comparison communities are to be
studied. Analyses are to include regression and time series models. At the
time of our review, this initiative had received $1 million.
Evaluation findings: It is too early in this evaluation to have reported
results.
GAO assessment: The absence of any appropriate comparison communities and
the variability in program implementation and components across the 12 study
sites will make it difficult to find compelling evidence of program effects.
Safe Start Initiative
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 62 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program Description: This program aims, in the five sites in which it is
being implemented and evaluated (Mesa and Tucson, AZ; Bloomington, IL; San
Antonio, TX; and Riverside, CA) to reduce gang- related crime through five
interrelated strategies: community mobilization; provision of social,
educational, and economic opportunities; suppression of gang violence;
social intervention; and organizational innovation. It involves the
collaborative efforts of the police, probation officers, prosecutors,
judges, schools, youth agencies, churches, housing authorities, and
governmental agencies. It targets youths at strong risk of gang membership
and crime, and youths already involved in serious gang crime. The evaluation
began in 1995.
Evaluator: University of Chicago.
Evaluation description: The impact evaluation of the program began in May
1995, and is expected to be completed in April 2002. Each project site is to
be matched with a comparison site. In four of the five sites, the program
participants and comparison groups were selected from similar gang problem
areas within the same city; in the fifth site, a separate comparison
community was selected. Between 100 and 115 youths, ages 12 to 21, who were
involved in gangs or at risk of involvement, were selected to participate in
the program in each site, and between 77 and 134 similar youths in each site
were selected for comparison purposes. Neither program nor comparison group
youths were selected randomly. A large, complex, communitywide data
collection effort is being employed in each site, through a variety of
methods and sources, including organizational surveys, youth surveys,
reports from service workers, police and school records, local newspaper
reports, and census data. Data were to be collected at baseline and after
the first and third years of the program. The principal outcomes to be
measured are gang crime patterns at the individual, gang, and community
levels. The evaluation will also consider changes in opportunities, as well
as integration in and alienation from conventional individuals and
institutions. A variety of analyses of the data are planned, including time-
series analyses and hierarchical linear models. At the time of our review,
the evaluation had received approximately $3 million. Impact Evaluations of
OJJDP Programs Well Into Their Implementation
Comprehensive Communitywide Approach to Gang Prevention, Intervention, and
Suppression Program (Comprehensive Gang Initiative)
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 63 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Evaluation findings: Preliminary results have been reported for program
participants, but not comparison groups. Because of missing data and other
problems, reporting deadlines may not be met, and two of the five project
sites and associated comparison sites have been deferred from the current
analyses.
GAO assessment: The evaluation, as designed, is basically sound. However,
numerous difficulties in obtaining data threaten parts of the evaluation.
Also, the way in which subjects were selected for the study may be
problematic, and it is unclear how program and comparison youths were
matched within sites.
Program Description: Since 1998, 85 communities and 4 colleges in at least
10 states have been awarded subgrants under the discretionary grant
component of this program to enforce underage drinking laws. In most states,
a diverse group of stakeholders are involved in planning a variety of
projects under this program that can include media campaigns, merchant
education, compliance checks and other enforcement, youth leadership
training, school- based education, and the development of local coalitions
and interventions aimed at reducing underage drinking. States and
communities are given substantial latitude in planning their projects;
interventions are not standardized across communities.
Evaluator: Wake Forest University School of Medicine.
Evaluation description: The effort to evaluate the discretionary grant
component of this program began October 1, 1998, and is expected to be
completed December 31, 2001. Data are to be collected- from telephone
surveys of police chiefs, sheriffs, and youths in participating communities
and matched comparison communities in at least 9 states- before or early on
in project implementation and at least 1 year after project initiation.
Project sites to be evaluated were initially chosen from all states
participating in the program. It is unclear whether these sites are
representative of all participating project sites. In the first year,
surveys were conducted in 52 participating communities and a similar number
of comparison communities. In the second and third years, surveys will be
conducted in those same communities and 34 others- 17 in each group. The
participant and comparison communities were matched on median income, liquor
law violations, percentage attending college, and population size. The
surveys of the top one or two law enforcement officials in each community
will provide information on local law enforcement efforts, including the
number of compliance checks Enforcing the Underage
Drinking Laws Program
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 64 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
conducted in each year. A small number of youths from each site are to be
selected at random for the surveys each year. The youth surveys will obtain
data on perceptions of alcohol availability, peer and personal alcohol use,
and alcohol- related problem behaviors including binge drinking and drunk
driving. At the time of our review, this evaluation had received
approximately $945,000.
Evaluation findings: While some demographic data have been reported from the
baseline survey, no results have been reported involving program effects.
GAO assessment: The researchers suggest aggregating all program communities
together and all comparison communities together to diminish community
sample size problems, which may mask program effects. In addition, the wide
variation allowed in program implementation may compromise the
interpretation and generalizability of any findings.
Program Description: The Intensive Aftercare program provides intensive
supervision and services to serious juvenile offenders for 6 months
following their release from secure confinement. The goal is to facilitate
reintegration and reduce recidivism. The program was implemented, beginning
in June of 1993, by various youth service offices and departments of
corrections in four states: Colorado, Nevada, New Jersey, and Virginia. New
Jersey was eventually dropped because of implementation problems, so the
evaluation of the program is being completed in the other three states.
Evaluator: National Council on Crime and Delinquency.
Evaluation description: Beginning in 1995, youths entering correctional
facilities in the three states (four counties in Colorado including
Metropolitan Denver; Clark County, NV; and Norfolk County, VA) were screened
for eligibility and randomly assigned, within each site, to the treatment
group (whose members participated in the Intensive Aftercare program upon
release) or control group. Between 1995 and 1999, 82 youths were assigned to
the program and 68 to the control group in Colorado, 120 youths were
assigned to the program and 127 to the control group in Nevada, and 75
youths were assigned to the program and 45 to the control group in Virginia.
Information was collected for study participants at baseline (that is, upon
entry into the institution), before release from the institution (9 to 12
months after baseline, or entry), immediately after completing the program
(6 months after release), and 6 Intensive Aftercare
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 65 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
months after completing the program (12 months after release). The data
collected, using survey instruments, standardized tests, monthly case
management forms, and administrative (police and court) databases, included
social and criminal history and demographic data, information on the extent
of supervision and services, and the extent of criminal activity following
institutional release. Many of the measures being employed in the study,
according to the researchers, are standard and have been validated. At the
time of our review, the evaluation had received approximately $932,000 in
funding.
Evaluation findings: The preliminary findings offered from this evaluation
suggest that the Intensive Aftercare participants did receive greater
supervision and more services after release than the control group, which
suggests some success in implementing the program. Outcome results related
to reintegration and recidivism are not complete, and the interim results
are mixed as to whether the program is associated with positive outcomes.
GAO assessment: This is a well- designed study, though serious missing data
problems, if not corrected, may make it difficult to determine the outcome
of this program.
Program Description: JUMP was established by Part G of the Juvenile Justice
and Delinquency Prevention Act of 1974, as amended in 1992. Through that
legislation, the Congress authorized OJJDP to award 3- year grants to
community- based not- for- profit organizations or to local educational
agencies. The grantees are to support one- on- one mentoring projects that
match volunteer adult mentors with youths at risk of delinquency, gang
involvement, educational failure, and dropping out of school. The
legislation also provided funding for a national, cross- site evaluation of
JUMP. OJJDP guidelines emphasize the need for projects to recruit, train,
supervise, and do thorough background checks for all volunteer mentors;
develop procedures for appropriately matching youths and mentors; define the
population of at- risk youths to be served; develop guidelines for the type,
frequency, and duration of youth and mentor project activities; and
establish procedures for gathering and reporting data to support the
evaluation process. As of November 2000, 175 JUMP projects had been funded,
in amounts ranging from $180,000 to $210,000 over a 3- year period.
Evaluators: Information Technology International and Pacific Institute for
Research and Evaluation. Juvenile Mentoring
Program (JUMP)
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 66 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Evaluation description: This evaluation began in May 1997 and is expected to
conclude on September 30, 2002. 4 Three approaches are being taken to
determine how well JUMP is accomplishing its objectives. The first is a
modified pre- post design that involves a within- subject comparison of the
characteristics of youths at the time they enter and exit the program and
between- subject comparisons of youths entering and exiting the program at
the same time. The second approach is a best practices approach that will
use structural equation models to estimate what program features or
activities, including success in matching mentors and youths, are most
likely to contribute to program success in reducing the risk of school and
family problems, delinquency, and drug use among youths. The third approach
relies on combined youth outcome data and community data to determine
community cost offsets. The evaluation was funded at $3. 3 million.
Evaluation findings: In their November 2000 JUMP annual report, the
evaluators provided considerable descriptive information about the various
JUMP projects, the characteristics of the youths and mentors, and
information on youth- mentor matching. The only ?outcome? information thus
far provided, however, is information on how satisfied youths and mentors
were with the mentoring experience and how much benefit each perceived was
derived from the experience. None of the three analytic approaches described
above has been successfully applied to study outcomes because of a variety
of pitfalls experienced by the national evaluation team, most notably
insufficient data on school performance and behavioral measures (e. g.,
delinquent behavior and arrests).
GAO assessment: The researchers are employing multiple and innovative
strategies to determine the effectiveness of JUMP in achieving its
objectives. It is not clear, however, whether definitive evaluation results
can be reached in the absence of outcome data on youths who, in the same
project areas at the same points in time, do not receive the program. In
addition, data limitations, if not corrected, may be serious enough to
compromise findings.
4 In commenting on a draft of this report, the Assistant Attorney General
pointed out that the latest award sets project and budget end- dates of
September 30, 2002. However, she expects the evaluation will not be
completed before September 30, 2004. OJP does not move the project period
forward beyond the budget period as this may imply an assurance of future
funding.
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 67 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Program Description: The Partnerships to Reduce Juvenile Gun Violence
Program is a multi- year demonstration program planned for four sites (Baton
Rouge and Shreveport, LA; Syracuse, NY; and Oakland, CA). It began in 1997
and is expected to conclude in 2001. However, one site, Shreveport, was
dropped from the program early. The program aims to reduce youth gun
violence by enhancing, in specific target areas of these cities, prevention
and intervention strategies and strengthening partnerships among community
residents, law enforcement agencies, and the juvenile justice system. The
program involves mobilizing the community, establishing agency linkages, and
planning case management for juveniles with gun charges in year 1, linking
at- risk youths to services in year 2, and expanding opportunities for
youths in year 3.
Evaluator: COSMOS Corporation.
Evaluation description: The strategy for evaluating the impact of this
program has evolved as the program has unfolded. An impact evaluation was
planned for three sites and was to include (1) a comparison of changes in
crime rates in target areas of these cities before and after the
implementation of the program, (2) a comparison of responses from highrisk
youths in targeted areas surveyed before and after the program was
implemented and services were provided, and (3) information on changes in
policies and caseloads revealed through focus group meetings and interviews
with agency officials. Crime rate information has thus far been reported
only for Oakland and Baton Rouge, and surveys have been conducted only in
Baton Rouge. In Baton Rouge, surveys were given to 92 high- risk youths in
the criminal justice system identified through a variety of processes. The
sampling strategies for surveying these high- risk youths were unlikely to
yield generalizable results. In addition, fifth-, seventh-, and ninth- grade
students in six schools in the target area were surveyed in March of 1999.
It is unclear why these students and schools were sampled and what response
rates were. In 2000, a small sample of 50 youths in Baton Rouge was
identified as a possible matched comparison group for arrest rate
comparisons. At the time of our review, this evaluation had received $1.2
million in funding, although a process evaluation is also being conducted
with these funds.
Evaluation findings: The researchers report decreases in gun- related
homicides and arrests in Oakland that were larger in the target area than
for the city as a whole. They also report decreases in gun- related
homicides in Baton Rouge. No analyses of results from the survey data have
been reported to date. Partnerships to Reduce
Juvenile Gun Violence Program
Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs
Page 68 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
GAO assessment: Comparisons between crime rates in the target community and
the city as a whole may not be appropriate. Student and school selection
criteria are unclear, making it difficult to assess their appropriateness
for obtaining definitive results. In addition, if supporting survey and
administrative data are only gathered in one site, it will be very difficult
to generalize findings whether they appear positive or not.
The purpose of the Teen Courts evaluation is to measure the effect of
handling young, relatively nonserious violators of the law in teen courts,
rather than in traditional juvenile family courts. Although teen courts
often include many of the same steps used by formal juvenile courts (for
example, intake, preliminary review of charges, court hearing, and
sentencing), they differ from formal courts in that young people are able to
assist in the community decision- making process for dealing with juvenile
offenders. Youths may act as prosecutors, defense counsel, jurors, court
clerks, bailiffs, and judge (or as a panel of judges). To evaluate teen
courts, both a process and impact evaluation are used, with case studies and
comparison groups as part of the research design. In each of the four case
study sites (Anchorage, AK; Independence, MO; Maricopa County, AZ; and
Rockville, MD), data are collected on about 100 youths handled in teen
courts (experimental group) and 100 youths handled in the traditional
juvenile justice system (comparison group). Data are also collected on
several dimensions of program outcomes, including post- program changes in
teens? perceptions of justice and their ability to make more mature
judgements as a result of the program. A process evaluation of the projects-
exploring legal, administrative, and case- processing factors that hinder
the achieving of project goals- is also being conducted. Impact Evaluation
of
OJJDP Program- Summary of Evaluation We Did Not Assess Teen Courts
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 69 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
This appendix contains summaries of the 24 evaluations the Office of
Juvenile Justice and Delinquency Prevention (OJJDP) has funded since 1995
(excluding the 11 impact evaluations it has funded of its own programs
discussed in app. VII). For 22 summaries, we used descriptions of
evaluations that were provided to us by OJJDP; for 2 summaries, 1 we wrote
the descriptions based on OJJDP documents. OJJDP categorized the 24
evaluations into the following three groups:
OJJDP- funded programs: nonimpact evaluations (11).
Non- OJJDP- funded programs: impact evaluations 2 (9).
Non- OJJDP- funded programs: nonimpact evaluations (4). The purpose of
this evaluation is to test the feasibility and effectiveness of the OJJDP
community assessment center concept in different environments. Community
assessment centers seek to facilitate earlier and more efficient delivery of
prevention and intervention services at the front end of the juvenile
justice system. The evaluation uses a two- phase process to (1) measure some
outcomes at the two enhancement sites, with quasi- experimental design, 3
and (2) achieve more and better outcome measures. But, according to OJJDP,
implementation and data problems will limit the effectiveness of the
quantitative methods employed. OJJDP also believes that the attempt to
implement a random assignment study at one project site will probably need
to be abandoned. The first phase covers the four project sites that
comprised the Community Assessment Centers program- two of these sites
funded enhancements to existing
1 We used OJJDP documents to develop summaries for SafeFutures and Coping
With Life Course. 2 Impact evaluations assess the extent to which a program
causes changes in the desired direction in the target population. 3 In a
quasi- experimental design, methods other than random assignment are used to
create a comparison group. When randomly assigning individuals to a control
group is not feasible, quasi- experimental impact evaluations can be used to
compare the performance of program participants on various measures to
individuals not in the program. Appendix VIII: Twenty- Four Other
Evaluations Funded by OJJDP OJJDP- Funded Programs: 11 Nonimpact Evaluations
Community Assessment Centers
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 70 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
programs and the other two funded the planning and implementation of new
programs. The second phase covers the two project sites- one enhancement and
one planning- in which the program is being continued after the end of the
first funding cycle. 4 Many of the evaluation measures are at the project or
community level rather than at the participant level.
The purpose of this evaluation is to examine the viability and effectiveness
of the community- based delinquency prevention model used by grantees in the
Community Prevention Grants Program. The Community Prevention Grants Program
encourages communities to develop comprehensive, collaborative plans to
prevent delinquency. The evaluation focuses on two main questions: (1) What
is the impact of the program on community planning, service delivery, risk
factors, protective factors, and juvenile problem behaviors? (2) What
factors and activities lead to the effective implementation of the Community
Prevention Grants Program model and to positive program outcomes? This
evaluation employs a case study approach supplemented by a basic profile of
communities that are participating in the program. Case studies are to be
implemented in 11 communities in 6 states. Evaluation measures are to be
applied at the project, community, and program levels.
The purpose of this process evaluation 5 is to address the following
questions about the Comprehensive Strategy program: (1) What are the factors
associated with successful Comprehensive Strategy planning and
implementation? (2) To what extent do project sites adhere to the prescribed
Comprehensive Strategy framework? (3) What are the major implementation
challenges program grantees face in implementing the Comprehensive Strategy?
(4) To what extent does the training and technical assistance provided to
project sites help them acquire the knowledge, skill, and tools necessary to
develop the Comprehensive Strategy? (5) What role should OJJDP play in the
future implementation of the Comprehensive Strategy? The Comprehensive
Strategy is OJJDP?s
4 In commenting on a draft of this report, the Assistant Attorney General
explained that the other two project sites were not included in the second
phase of the evaluation because they did not receive additional funding. One
site is no longer a separate project and the other project- a planning site-
continues to move toward operational status.
5 A process evaluation uses empirical data to assess delivery of a program
and verifies whether it was delivered as intended to the targeted
recipients. Community Prevention
Grants Program (Title V) Comprehensive Strategy for Serious, Violent, and
Chronic Juvenile Offenders (Comprehensive Strategy)
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 71 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
approach for addressing juvenile violence and delinquency at the community,
state, and national levels through a systematic plan. It advocates the use
of local planning teams to assess the factors and influences that put youths
at risk of delinquency, determine available resources, and establish
prevention programs to either reduce risk factors or provide protective
factors that buffer juveniles from the impact of risk factors. This
evaluation uses a multilevel design to assess how project sites implement
the Comprehensive Strategy. The evaluation began with telephone interviews
with site coordinators from all 48 project sites; 25 of the 48 project sites
were randomly selected for a stakeholder survey. One year later, 10 of the
25 project sites are being given a second stakeholder survey. Subsequently,
five sites are to be selected for visits, and intensive case studies are
being done in three cities.
The purpose of this evaluation is to examine (1) community coalitions?
developmental processes from the early planning and adoption stages through
implementation and later stages and (2) the impact of coalitions? prevention
efforts concerning risk and resiliency factors and, to the extent feasible,
alcohol, tobacco, and other drug use. The Drug- Free Communities Support
Program provides grants to community coalitions to strengthen their efforts
to prevent and reduce young people?s illegal use of drugs, alcohol, and
tobacco. The evaluation is studying two cohorts of program grantees- those
that received grants in 1998 or 1999 (cohort 1) and those that received
grants in 2000 (cohort 2). The national evaluation sample is comprised of a
total of 213 grantees. The sample is divided by years of operation: 1- 5
years, 6- 9 years, and more than 9 years. Semiannually, cohort 1 grantees
are required to submit progress reports to OJJDP and the evaluator that
include a special section (Part II), which provides information about the
compositions of the coalitions and outcome data collection. Cohort 2
grantees do not have a Part II reporting requirement and submit progress
reports semiannually only to OJJDP. In addition, 21 grantees (15 from cohort
1 and 6 from cohort 2) serve as intensive study sites, where interviews with
staff and stakeholders provide greater detail about coalition development
and local program evaluation.
The purpose of this process evaluation is to provide feedback to OJJDP on
the implementation of the Juvenile Accountability Incentive Block Grants
Program. The program encourages states and local jurisdictions to implement
accountability- based programs and services in 54 states and U. S.
territories. The evaluator is surveying state and local practitioners,
policy makers, and grant program administrators about their perceptions
Drug- Free Communities
Support Program Juvenile Accountability Incentive Block Grants Program
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 72 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
and attitudes about the program and its administration. Specifically, the
evaluation focuses on (1) understanding how states and local units of
government plan for and administer program funds and (2) examining the
perceptions of states and local units of government about how well the
program is achieving congressional intent. In addition, in- depth case
studies are conducted at a limited number of sites.
The purpose of this evaluation is to (1) provide feedback to the
Performance- based Standards Project team on improving design and
implementation support to the sites, (2) assist the project team in refining
the Performance- based Standards Project model and in maximizing
responsiveness to the needs of the participants, that is, those who are
implementing the project model, and (3) chronicle the development of the
project and summarize lessons learned. OJJDP established the Performance-
based Standards Project to improve the quality and conditions of juvenile
corrections facilities. Specifically, the project develops and implements
outcome standards and an assessment tool. Corrections facilities can use
both to monitor progress towards meetings goals in areas of operations, such
as health and safety. The evaluation uses a case study approach. This
approach consists of the collection of both quantitative and qualitative
data describing the processes used to implement the project model in 80
juvenile detention and correctional facilities across the country. Site
visits are made and in- depth case studies are planned. An all- site survey
is distributed to key participants to determine satisfaction with the
supports provided to them in the implementation of the project model. In
addition, the survey seeks the participants? assessment of (1) the impact
the project has made on conditions of confinement and management of the
facilities and (2) the overall utility of the project model.
The purpose of this evaluation is to (1) determine the extent to which
replication project sites have been able to conform to the original program
model, and (2) assess the ?prosocial? (that is, positive, socially- oriented
behavior) outcomes for mothers and their babies. The Prenatal and Early
Childhood Nurse Home Visitation Program consists of intensive and
comprehensive home visitation by nurses during a woman?s pregnancy and the
first 2 years following the birth of her first child. The evaluation
involves six project sites and employs a quasi- experimental design with
matched comparison groups. Performance- based
Standards Project Prenatal and Early Childhood Nurse Home Visitation Program
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 73 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
This purpose of this process evaluation is to document and understand the
process of community mobilization and collaboration. SafeFutures is designed
to build a comprehensive program of prevention and intervention strategies
for at- risk youths and juvenile offenders. The program comprises six
project sites that represent urban, rural, and Native American communities.
The evaluation is examining all six project sites. Project sites collect and
record performance data on program operations and client outcomes using the
Client Indicator Data Base. The project sites are required to collect
extensive information from selected SafeFutures program components on
individual participants? risk and protective factor profiles, youths?
service utilization, and agencies? coordination of services for youth during
the course of their involvement in the SafeFutures program. In addition,
they collect information on outcome measures regarding youths? educational
commitment (that is, school attendance, achievement, and behavior), youths?
involvement in delinquency and crime, and any changes in youths? risk
profiles. Analysis of these data will provide a picture on program
performance in three key areas: reaching the intended high- risk youth
clientele, coordinating services for youths with multiple problems, and
monitoring subsequent school performance problems and involvement in the
juvenile justice system.
The purpose of this evaluation is to document the lessons learned and
factors associated with the successful development and implementation of the
Safe Kids/ Safe Streets program. The Safe Kids/ Safe Streets program is
designed to (1) help communities break the cycle of early childhood
victimization and later criminality and (2) reduce child abuse and neglect,
as well as the child fatalities that often result. The evaluation is
surveying five project sites through five data collection strategies: agency
administrative data, case tracking, key informant interviews, surveys of
agency professionals, and surveys of stakeholders.
The purpose of this evaluation is to (1) support culturally appropriate
process and outcome evaluations of activities funded under Tribal Youth
Program grants and (2) build the capacity of tribes to better evaluate their
own juvenile justice programs and activities. The Tribal Youth Program
assists grantees in developing projects, within tribal communities, for the
prevention and control of youth violence and substance abuse. The evaluation
is participatory in nature, that is, project personnel and stakeholders will
be involved in developing the evaluation designs, with the assistance and
guidance of an evaluation facilitator. The five project sites are
implementing different projects and have not yet completed their
Partnerships to Reduce
Youth Violence and Delinquency Program (SafeFutures)
Safe Kids/ Safe Streets Tribal Youth Program
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 74 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
evaluation designs. According to OJJDP, it is too early in the evaluation to
tell exactly what designs are to be used. OJJDP has required that all
evaluations be designed to examine both program implementation and program
outcomes.
The purpose of this process evaluation 6 is to (1) determine how community
collaboration can affect truancy reduction and lead to systemic reform and
(2) assist OJJDP in the development of a model for a truancy reduction
program, including identifying the essential elements of that model. The
Truancy Reduction Demonstration Program encourages communities to develop
comprehensive approaches- involving schools, parents, the justice system,
law enforcement, and social service agencies- in identifying and tracking
truant youths. The evaluation is employing site visits, interviews with key
personnel, and case studies of individual sites. Process data are gathered
from all seven project sites participating in the evaluation and, from some
sites, limited outcome data are gathered.
The purpose of this evaluation is to evaluate the efficacy of three
Adolescent Female Offenders programs in Wayne County, Michigan. The three
programs are (1) a program incorporating gender- specific programming, home-
based intervention, and community involvement, including pregnant and
parenting adolescents; (2) an intensive probation program with limited
gender- specific programming; and (3) a traditional, female- only
residential program that provides limited gender- specific training. The
evaluation is using a quasi- experimental design. Using random assignment,
the home- based intervention model is to be compared with the established
intensive probation model; the outcomes of these models are then to be
compared with outcomes of the traditional,
6 The Truancy Reduction Demonstration Program and its evaluation are jointly
funded by OJJDP, the Office of Justice Program?s Executive Office of Weed
and Seed, and the U. S. Department of Education. Truancy Reduction
Demonstration Program Non- OJJDP- Funded Programs: Nine Impact Evaluations
Adolescent Female Offenders (Three Programs)
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 75 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
female- only residential program. The comparison analysis involves at least
50 young women in each of the 3 programs. A wide range of outcomes-
including recidivism, substance use, depression, community integration,
academic performance and career aspirations, parenting readiness, and
responsible sexual behavior- is to be examined. The evaluator is also
exploring the relationship of specific program components to these outcomes.
The purpose of this impact evaluation is to evaluate the effectiveness of a
cognitive- behavioral group intervention. The Coping With Life Course is
aimed at enhancing prosocial coping and problem solving for adolescents
incarcerated in youth correctional facilities. To evaluate the program, a
minimum of 120 adolescents in one youth correctional facility are randomly
assigned to either the Coping with Life Course intervention group or a
standard- care control group. Six Coping with Life Course cohort groups of
10 each are followed. The evaluation is allowing for attrition (from the
initial 60 participants down to 48) in each of the intervention and control
groups. Participant functioning is assessed before and after intervention
through a battery of questionnaires. Recidivism, return to close custody,
and service utilization are tracked through databases and statewide records.
The purpose of the evaluation is to (1) document the implementation of a new
?family index? case management system (through a process evaluation) and (2)
examine the impact of the family index on juvenile court case processing
(through an impact evaluation). The family index system allows cross-
referencing to identify all family members involved in family law; juvenile
dependency; juvenile delinquency; and criminal, civil, and probate matters.
For the process evaluation, a case study approach is used to describe the
implementation of the family index at one project site, the Riverside,
California, Court. For the impact evaluation, a pre- post design is used to
examine how the family index has affected juvenile court matters (for
example, court processing time, coordination between courts, and content of
hearings).
The purpose of this evaluation is to evaluate the effects of Flashpoint on
the antisocial patterns of juvenile offenders? thoughts and actions and high
school students? thoughts and actions. Specifically, changes are assessed
for (1) media use and literacy, (2) violence- supporting beliefs and
behavior, and (3) substance use and abuse. The Flashpoint program is Coping
with Life Course
Creation and Implementation of a Family Index in Riverside County,
California, Courts
Flashpoint
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 76 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
designed to build critical thinking skills young people need to (1) see
through false media portrayals that glorify violence and drug use and (2)
apply decision- making in their own lives. The evaluation is using
comparison groups, pre- post interventions, and case studies. Participants
include 264 juveniles, ages 14 to 17. Treatment groups are compared with no-
treatment control groups for baseline- to- posttest changes. Three groups
and project sites are involved: (1) repeat and serious offenders in a
correctional program, (2) first- time offenders in a diversion program, and
(3) students in a public high school.
The purpose of this evaluation is to determine if Free to Grow can reduce
substance abuse (alcohol use, smoking, and illegal drug use). The Free to
Grow program builds on existing Head Start programs, adding
communitystrengthening and family- intervention components to address the
problem of substance abuse. The evaluation is attempting to determine the
independent effects of these two components on substance abuse prevention.
It involves 16 project sites and 16 comparison sites and employs a
multistage experimental research design.
The purpose of this evaluation is to provide a process and outcome
evaluation of the Gaining Insight into Relationships for Lifelong Success
Project. The project involves two primary levels of intervention: (1) a
psycho- educational counseling group, dealing with relationships and
involving girls in four relational domains (relation to self, family, peers,
and teachers), and (2) a focus on individual consultations, educational
workshops and the policies and procedures of the local juvenile justice
system, and the involvement of court service workers from the system.
Specifically, the evaluation will (1) investigate the applicability of a
relational approach to the treatment of female juvenile offenders; (2)
examine the components of the relational approach that deal with
relationships to self, family, peers and teachers; (3) evaluate the impact
of increasing the knowledgebase of professionals involved in the local
juvenile justice system; and (4) provide an empirically based, alternative
treatment model that can be replicated in other settings. The evaluation of
the first level of intervention- the counseling group- focuses on each of
the four relational domains through the use of multimethod data collection;
this collection includes self- reports and other reports, school records,
and recidivism data. The evaluation of the second level of intervention
focuses on the court services that workers use, specifically gender-
sensitive treatment recommendations and referrals; qualitative observational
data, gathered from monthly meetings, will be used. There is Free to Grow:
Head Start
Partnerships to Promote Substance- Free Communities
Gaining Insight Into Relationships for Lifelong Success Project
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 77 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
random assignment between girls referred to either the project intervention
or to the standard intervention currently being used by the Clark County
Court in Athens, Georgia. Approximately 180 girls- 90 referred to the
project intervention and 90 referred to the standard court intervention- are
to be evaluated.
The purpose of this evaluation is to assess the implementation and the
impact of Quantum Opportunities. The Quantum Opportunities program is
designed to reduce the incidence of delinquency, criminal behavior, and
subsequent involvement in the criminal and juvenile justice systems amongst
educationally at- risk inner city youths. The evaluation is using an
experimental design with random assignment. Ninth- grade students at six
sites are randomly assigned to treatment and control groups, with the
treatment group enrolled in the Quantum Opportunities program. The students
are followed through their high school careers and 2 years beyond.
Information is collected from academic achievement tests, administered each
year, and from two questionnaires.
The purpose of this evaluation is to test the impact- on recidivism, program
completion, and victim satisfaction- of the Restorative Justice Conferences
for a population of youthful offenders (aged 14 and under) in an urban
setting (Indianapolis, IN). Restorative Justice Conferences bring together
the offender, victim, and supporters of each so as to provide an opportunity
for fuller discussion of the offense; the effect of the offense on the
victim, the offender?s family, and greater community; and steps the offender
can take to make amends. The evaluation is using a single- site evaluation
with an experimental design. As part of the design, youths are randomly
assigned to a treatment group (Restorative Justice Conferences) or a matched
control group.
The purpose of this evaluation is to determine if the program reduces the
amount of delinquency in a city. The Risk- Focused Community Policing
program increases protection by the community police, potentially reducing
delinquency. The evaluation is using an experimental research design. The
project site (city) is divided into approximately 40 census blocks, with 20
blocks randomly selected as program blocks and the other 20 designated as
control blocks. Quantum Opportunities
Restorative Justice Conferences
Risk- Focused Community Policing
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 78 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
The purpose of this evaluation is to study the Act Now Truancy Program. The
program is a prosecutor- led truancy reduction program. The evaluation is
using a pre- post intervention design involving one project site.
Information is collected and aggregated (for example, truancy rates rather
than individual truancy behavior) for all participants.
The purpose of this evaluation, conducted in two schools, is to assess the
impact of the Childhood Violence Prevention Program. The program is designed
to prevent the legitimization of aggression among pre- adolescent,
elementary, and middle school children, with special focus on victims of
child maltreatment. The evaluation is using a pre- post intervention design,
with comparison groups. The study involves having elementary school students
participate in a class activity using a workbook designed to encourage
problem solving action rather than aggressive behavior in interactions with
peers.
This project is not an evaluation, per se, but rather a synthesis of
existing evidence on community- level interventions and service programs.
Its purpose is to identify the strengths and weaknesses of community- level
evaluations and to provide recommendations to the field about how to
structure and carry out such evaluations. Community- level programs for
youths are designed to promote positive youth development. To evaluate the
programs, a committee- experts from several disciplines (child and
adolescent development, child health, sociology, psychology, evaluation
research, youth services, and community development)- is assessing the
strengths and limitations of measurements and methodologies that have been
used to evaluate these interventions. Non- OJJDP- Funded
Programs: Four Nonimpact Evaluations
Act Now Truancy Program Childhood Violence Prevention Program (Let?s Talk
About...)
Community- Level Programs for Youth
Appendix VIII: Twenty- Four Other Evaluations Funded by OJJDP
Page 79 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Conce
The purpose of this evaluation is to assess the activities undertaken by
project sites, determine whether they can be evaluated, and ultimately
assess the impact of these activities on the youthful offenders
participating in the program. The program is intended to (1) enhance school-
to- work education and training in juvenile correctional facilities and (2)
improve youthful offenders? transition into the community. The evaluation
design has not been completed, but random assignment study is strongly
preferred, if feasible. At the time of our review, only one of the three
potential sites could be evaluated. One more project site is to be awarded
and, if it can be evaluated, it will be added as a second evaluation site.
Department of Labor?s
Education and Training for Youthful Offenders Initiative
Appendix IX: Comments from the Department of Justice
Page 80 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IX: Comments from the Department of Justice
Note: GAO comments supplementing those in the report text appear at the end
of this appendix.
Appendix IX: Comments from the Department of Justice
Page 81 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now on pp. 23- 24. Now on pp. 3, 8- 10, and
22- 25.
Appendix IX: Comments from the Department of Justice
Page 82 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IX: Comments from the Department of Justice
Page 83 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now on p. 10. Now on p. 9.
Now on p. 6. Now on p. 4. Now on p. 21.
Appendix IX: Comments from the Department of Justice
Page 84 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
See comment 1. Now on p. 22. Now on pp. 10- 18.
Appendix IX: Comments from the Department of Justice
Page 85 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IX: Comments from the Department of Justice
Page 86 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now on p. 25.
Appendix IX: Comments from the Department of Justice
Page 87 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now on p. 31. Now on p. 26. Now on p. 26. Now on pp. 25- 26.
See comment 2. Now on p. 25.
Appendix IX: Comments from the Department of Justice
Page 88 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IX: Comments from the Department of Justice
Page 89 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now on pp. 48- 51 Now on p. 48.
Now on p. 36, table 6. Now on p. 35. Now on p. 35.
Appendix IX: Comments from the Department of Justice
Page 90 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now on p. 66. Now on p. 57- 58.
Now on p. 59. Now on p. 53, tables 10
and 11. See comment 3.
Now on p. 51. Now on p. 51.
Appendix IX: Comments from the Department of Justice
Page 91 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now on p. 73. Now on p. 71.
Now on p. 70. Now on p. 64.
See comment 4.
Appendix IX: Comments from the Department of Justice
Page 92 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Now in app. VII, p. 68.
Appendix IX: Comments from the Department of Justice
Page 93 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
The following are GAO?s comments on the Department of Justice?s October 15,
2001, letter.
1. As we indicated in our report, impact evaluations, such as the types that
OJJDP is funding, can encounter difficult design and implementation
challenges. (See section titled, Evaluations of OJJDP Programs are Difficult
to Successfully Design and Implement.) Also, we are aware that virtually all
impact evaluations have limitations. However, where possible, impact
evaluations should be designed to mitigate as many rival explanations of
program effects as feasible, and potential limitations of the chosen
research design should be acknowledged.
2. Our statement that the Juvenile Mentoring Program evaluation ?has
experienced problems obtaining behavioral measures and school performance
data? was not intended to criticize the evaluators? level of effort, but
rather to indicate that their inability to obtain data from school and law
enforcement officials in many of the study sites makes it more difficult to
evaluate how well the program is achieving its objectives of diminishing
delinquency, gang involvement, and school failure. While enhanced analysis
of sites with the best data may be warranted, it does not overcome the
problem of having a large number of sites with little or no reliable data
from school and law enforcement officials. This problem was explicitly
recognized by OJJDP in its November 2000 report.
3. During the course of our review, OJJDP officials told us that one measure
of a research grantee?s performance is the number of products the grantee
has published. These officials provided us a listing of all products
published by active research grantees through OJJDP and private publishers
as a direct result of OJJDP- funded research. We summarized these voluminous
data by topic to facilitate the presentation.
4. Our report points out that the Enforcing the Underage Drinking Laws
Program evaluation documents OJJDP provided to us were not clear on whether
the sites chosen were representative. Our report does not suggest ?there is
no way of achieving a legitimate representative sample.? However, we agree
with OJJDP that the evaluation may not be evaluating the Enforcing the
Underage Drinking Laws Program because there may be no program components
common to all project cities. The Assistant Attorney General states that the
evaluation will be GAO Comments
Appendix IX: Comments from the Department of Justice
Page 94 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
able to measure impacts on several program areas across each site. However,
our point is that the evaluator?s plan to aggregate data across sites may be
inappropriate because wide variation allowed by the program means that
program activities are not common across all sites. Therefore, interpreting
and generalizing results may be problematic.
Appendix X: GAO Contacts and Staff Acknowledgments
Page 95 GAO- 02- 23 OJJDP Grantee Reporting Requirements and Concerns
Laurie E. Ekstrand, (202) 512- 8777 James M. Blume, (202) 512- 8777
In addition to the above, Lori A. Weiss, Barbara A. Guffy, Michele J. Tong,
Leslie C. Bharadwaja, David P. Alexander, Douglas M. Sloane, Shana B.
Wallace, Michele C. Fejfar, Charity J. Goodman, and Jerome T. Sandau made
key contributions to this report. Appendix X: GAO Contacts and Staff
Acknowledgments GAO Contacts Acknowledgments
Bibliography Page 96 GAO- 02- 23 OJJDP Grantee Reporting Requirements and
Concerns
Howell, J. C. Youth Gang Programs and Strategies. U. S. Department of
Justice, Office of Justice Programs, Office of Juvenile Justice and
Delinquency Prevention. Washington, D. C.: Aug. 2000.
Novotney, L. C., E. Mertinko, J. Lange, and T. K. Baker. ?Juvenile
Mentoring Program: A Progress Review.? Juvenile Justice Bulletin, Sept.
2000.
Sheppard, D., H. Grant, W. Rowe, and N. Jacobs. ?Fighting Juvenile Gun
Violence.? Juvenile Justice Bulletin, Sept. 2000.
U. S. Department of Justice, Office of Justice Programs, National Institute
of Justice. ?Reintegrating Juvenile Offenders Into the Community: OJJDP?s
Intensive Community- Based Aftercare Demonstration Program.? Research
Preview, Dec. 1998.
U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
Justice and Delinquency Prevention. FY 2000 OJJDP Discretionary Program
Announcement: Juvenile Mentoring Program. Mar. 2000.
U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
Justice and Delinquency Prevention. Gang- Free Schools and Communities
Initiative: FY 2000 OJJDP Discretionary Program Announcement. July 2000.
U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
Justice and Delinquency Prevention. Juvenile Mentoring Program: 1998 Report
to the Congress. Washington, D. C.: Dec. 1998.
U. S. Department of Justice, Office of Justice Programs, Office of Juvenile
Justice and Delinquency Prevention. OJJDP Research 2000. Washington, D. C.:
May 2001.
Wiebush, R. G., B. McNulty, and T. Le. ?Implementation of the Intensive
Community- Based Aftercare Program.? Juvenile Justice Bulletin, July 2000.
Bibliography
(185012)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***