No Child Left Behind Act: Education Could Do More to Help States
Better Define Graduation Rates and Improve Knowledge about
Intervention Strategies (20-SEP-05, GAO-05-879).
About one third of students entering high school do not graduate
and face limited job prospects. The No Child Left Behind Act
(NCLBA) requires states to use graduation rates to measure how
well students are being educated. To assess the accuracy of
states' graduation rates and to review programs that may increase
these rates, GAO was asked to examine (1) the graduation rate
definitions states use and how the Department of Education
(Education) helped states meet legal requirements,(2) the factors
that affect the accuracy of graduation rates and Education's role
in ensuring accurate data, and (3) interventions with the
potential to increase graduation rates and how Education enhanced
and disseminated knowledge of intervention research.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-879
ACCNO: A37441
TITLE: No Child Left Behind Act: Education Could Do More to Help
States Better Define Graduation Rates and Improve Knowledge about
Intervention Strategies
DATE: 09/20/2005
SUBJECT: Data integrity
Educational standards
School dropouts
Statistical data
Statistical methods
Surveys
Federal/state relations
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-879
United States Government Accountability Office
GAO
Report to Congressional Requesters
September 2005
NO CHILD LEFT BEHIND ACT
Education Could Do More to Help States Better Define Graduation Rates and
Improve Knowledge about Intervention Strategies
GAO-05-879
Contents
Letter
Results in Brief
Background
Many States Moving toward Using A Definition That Follows
Students over Time; Education's Guidance Regarding NCLBA Requirements Is
Limited Several Factors Affected the Accuracy of Graduation Rates, and
Data Quality Remains a Key Challenge Few Interventions Have Been
Rigorously Evaluated, and Education
Has Done Little to Evaluate and Disseminate Existing Research Conclusions
Recommendations for Executive Action Agency Comments and Our Evaluation
1
3 6
11
24
31 45 47 47
Appendix I Objectives, Scope, and Methodology
Appendix II Comments from the Department of Education
Appendix III GAO Contact and Staff Acknowledgments
Related GAO Products
Bibliography
Tables
Table 1: Number of States That Allow Schools to Maintain Previous
Year's Rate or Show Progress toward Graduation Rate
Targets to Make AYP, as of July 2005 22 Table 2: Number of Interventions
Visited by School Level and Type 32 Table 3: Key Features of the Check and
Connect Model 34 Table 4: States Selected for Site Visits and Phone
Interviews by
Purpose 50
Figures
Figure 1: Student Mobility and Graduation Outcome for a
Hypothetical High School Class 13 Figure 2: Cohort Formula Definition 14
Figure 3: Departure Classification Definition 15 Figure 4: Definitions by
State, as of April 2005, and Planned to Use
by State, School Year 2007-08 18 Figure 5: State Graduation Rate Targets,
as of July 2005 20 Figure 6: Estimated School Graduation Rates under
Varying
Assumptions of Errors in Counting Dropouts 28 Figure 7: Project GRAD
Structural Model 35 Figure 8: Aviation High School Presentation by the
Blue Angels 41
Abbreviations
AYP Adequate Yearly Progress
GED General Education Development
ESOL English for Speakers of Other Languages
HOSTS Help One Student to Succeed
IASA Improving America's Schools Act of 1994
NCES National Center for Education Statistics
NCLBA No Child Left Behind Act of 2001
Project GRAD Project Graduation Really Achieves Dreams
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office Washington, DC 20548
September 20, 2005
The Honorable Edward M. Kennedy
Ranking Minority Member
Committee on Health, Education, Labor, and Pensions United States Senate
The Honorable Lamar Alexander
Chairman
Subcommittee on Education and Early Childhood Development
Committee on Health, Education, Labor, and Pensions United States Senate
The Honorable Jeff Bingaman United States Senate
The Honorable Patty Murray United States Senate
The Honorable Olympia J. Snowe United States Senate
About one third of students who enter high school do not graduate and face
limited job opportunities. The No Child Left Behind Act of 2001 (NCLBA)
was passed in part to increase the likelihood that all of the 48 million
students in our nation's public school systems will graduate and requires
states to use high school graduation rates, along with test scores, to
assess how much progress high schools are making in educating their
students. Graduation rates-used in conjunction with test scores-provide a
more complete picture of school performance than test scores alone,
because a school's test proficiency rate will be higher if low-performing
students drop out and do not have their scores included with their peers.
Graduation rates are used as part of the determination about whether
schools meet federal requirements for school progress. If schools do not
meet such requirements, their students may be eligible to transfer to
another school or receive tutoring. Currently, the Department of Education
(Education), National Governors Association, and several national
education organizations and foundations are working on high school reform
initiatives to address issues, such as school structure and
curriculum, which may help low-performing students and increase the
likelihood of graduation. In addition, our 2002 report on high school
dropouts identified the need for better information on the success of
interventions designed to increase the likelihood of students staying in
school until they graduate.1
NCLBA defines graduation rates as the percentage of students who graduate
from high school with a regular diploma in the standard number of years.
Education's regulations do not permit states to count an alternative
degree that is not fully aligned with the state's academic standards, such
as a certificate of attendance or a General Educational Development
certificate (GED). Each state has flexibility, however, in determining how
its graduation rate will be specifically calculated as long as the rate
is, as the law requires, "valid and reliable."
In response to congressional requests, we are providing information on:
(1) the definitions states have developed for graduation rates and how
Education supports states in meeting the law's requirements for defining
and measuring graduation rates; (2) the factors, such as student mobility,
that affect the accuracy of the data used to calculate graduation rates
for all students and those in designated groups, and what Education does
to ensure accuracy of rates reported by states; and (3) what is known
about the success of interventions with the potential to increase
graduation rates and how Education has enhanced and disseminated knowledge
about these practices.
To address these objectives, we used a variety of methodological
approaches. We analyzed the plans states were required to submit to
Education to identify the graduation rate definitions states used and
graduation rate goals set by states, reviewed updates to plans through
July 2005, and letters from Education to states regarding its decisions
about state plans and updates. We also surveyed officials in 50 states,
the District of Columbia, and Puerto Rico2 to obtain information about the
extent to which states verify school and district data used to calculate
high school graduation rates and use unique student identifiers. We
selected and
1GAO, School Dropouts: Education Could Play A Stronger Role in Identifying
and Disseminating Promising Prevention Strategies. GAO-02-240 (Washington,
D.C.: Feb. 1, 2002).
2Hereinafter, the term states will refer collectively to the 50 states
plus the District of Columbia and Puerto Rico.
contacted 20 states for further analysis. States were selected to capture
variation in high school graduation rate definitions, geographic location,
and types of interventions with the potential to increase graduation
rates. We conducted a case study in 1 state to calculate graduation rates;
site visits in 3 states to review data accuracy; site visits in 6 states
to observe interventions and interview staff at 16 schools; and phone
interviews in all 20 states to obtain information on definitions used,
implementation status, and guidance provided. To identify which
interventions have the potential to increase graduation rates, we reviewed
the research on interventions and interviewed Education officials and
dropout prevention experts. We also reviewed available evaluations of the
types of interventions we observed to assess their findings and
methodological approaches. To determine how Education assists states, we
reviewed Education regulations, guidance, and other documents and
interviewed Education and state agency officials. We also interviewed
Education and state officials to determine the degree to which Education
has enhanced and disseminated knowledge about interventions. To determine
the extent to which reported dropout rates may be understated, we
interviewed experts in this area and reviewed research on the topic.
Finally, we interviewed officials from the National Governors Association,
national education organizations, and other experts in the area of high
school graduation rates and reviewed related research to obtain an
understanding of the issues surrounding these rates and high school reform
efforts to address them. For a more detailed explanation of our
methodology, see appendix I. We conducted our work between September 2004
and July 2005 in accordance with generally accepted government auditing
standards.
A majority of states used or planned to use a graduation rate definition,
referred to as the "cohort" definition, which follows a group of students
over time from when they entered high school until they left. Education
has assisted states; however, it has not provided guidance on ways to
account for certain students. The cohort definition, used by 12 states as
of spring 2005, compares the number of 12th grade graduates with the
number of students enrolled as 9th graders 4 years earlier, while also
taking into account the number of students who left the school, such as
those who transferred in and out. Thirty-two states used a definition of
high school graduation rate based primarily on the number of dropouts over
a 4-year period and graduates, referred to as the "departure
classification definition." The remaining eight states used a variety of
other definitions. Many states using the departure or other definitions
are planning to move to the cohort definition by school year 2007-08 or
when their data systems can accommodate its use. This definition may help
Results in Brief
schools provide more precise graduation rates; however, it requires data
systems that can track students or groups of students over time. Most
states used these definitions to set graduation rate targets (for example,
80 percent a year). Although states generally set numerical targets, many
considered a school as meeting state graduation rate requirements if the
school showed progress toward these targets. The progress states allowed
generally ranged from any progress up to 1 percent, with two states
allowing schools to maintain the graduation rate of the previous year.
Education has supported states' efforts to develop definitions that are
intended to produce more precise results, developed some guidance, and
provided support such as on-site peer reviews, conferences, and
information on its Web site. Education also commissioned a task force that
published a report identifying the advantages and disadvantages of
different definitions. States also encountered challenges in resolving
common issues, such as how to account for students with disabilities who
graduate with a regular diploma in more than the standard number of years
based on their Individualized Education Plans. Education has not provided
guidance to all states on how to account for students in such programs;
instead, Education's approach has been to provide such information to
states on a case-by-case basis. As a result, some states were not aware of
the modifications available to count such students in their graduation
calculation, and there is less consistency among states, even those using
similar definitions, in how their rates are calculated.
Difficulty tracking mobile student populations was the primary factor
affecting the accuracy of graduation rates; while Education has taken some
steps to help states address this challenge, concerns about data accuracy
still exist. According to state, school district, and school officials and
experts we interviewed, the more that a school's students come and go, the
more challenging it is for a school to maintain accurate records on
whether students leave school by transferring or dropping out. Other
factors-such as the degree to which states verify school and district
data-also affect the accuracy of graduation rates. For example, fewer than
half of the states reported conducting audits that verify these data. Data
inaccuracies, such as miscounting the number of dropouts, can
significantly raise or lower a school's reported graduation rate. Because
most states were in the process of adopting a different graduation rate
definition, Education officials told us that they could not examine the
reliability of the data used to calculate such rates until after the new
definitions had been in place for multiple years. Such time would allow
them to determine if the rates produced consistent results. Also,
Education enhanced its state monitoring by adding a review component to
examine data states used for graduation rates, among other aspects of
states' participation in the Title I program. Furthermore, in response to
recommendations from GAO and Education's Inspector General, the agency
contracted with a firm to develop a guide by the end of 2005 to help
states improve data collection processes. In July 2005, Education
announced that it planned to calculate and report interim graduation rate
estimates for each state to provide a nationwide perspective. However, in
our review we found that data problems exist, and it is unclear whether
the department's monitoring efforts are sufficient for states to provide
accurate data for Education's estimates.
Few of the interventions that states and school districts have implemented
to increase high school graduation rates have been rigorously evaluated,
and Education has done little to evaluate and disseminate existing
knowledge about effective interventions. We identified five interventions
that had been rigorously evaluated and showed potential for improving
graduation rates. In our visits to six states we visited three schools
that were using such interventions. For example, Check and Connect, an
intensive mentoring program, showed increased levels of educational
attainment for students with emotional and behavioral disabilities.
Another program, Project GRAD, a comprehensive kindergarten-to-12 reform
program, demonstrated some promise in improving test scores and graduation
rates. In addition to the programs we visited, recently completed rigorous
evaluations of two other programs, the Talent Development High School
Model and First Things First, suggest that these interventions may also
increase graduation rates. Most other programs we visited fell into one of
three categories-restructuring schools, providing supplemental services,
such as tutoring, and creating alternative learning environments-similar
to findings in our 2002 report on high school dropouts. While these had
not been rigorously evaluated, research and program officials noted some
promising results that may lead to improving student outcomes including
high school graduation. With the NCLBA requirement that interventions be
research-based, there is a need in the education community for additional
scientifically based research. However, Education's efforts to evaluate
and disseminate existing knowledge on interventions have been minimal.
We are recommending that the Secretary of Education develop approaches to
provide information on how to account for different types of students to
all states rather than providing this information on a stateby-state basis
and assess the reliability of data submitted by states that Education
plans to use to develop interim graduation rates. We are also recommending
that the Secretary establish a timetable to carry out the recommendation
in our 2002 report regarding evaluating research on
Background
dropout interventions, including those that focus on increasing graduation
rates, and that the Secretary disseminate research on programs shown to be
effective in increasing graduation rates. In comments on a draft of this
report, Education concurred with our recommendations about accounting for
different types of students and the need for evaluating and disseminating
research on dropout interventions. On our recommendation to assess the
reliability of data submitted by states, Education noted that it was
taking steps to assess data reliability; however, it is not clear that
these steps apply to data that Education plans to use to calculate interim
rates.
Despite the increasing importance of a high school education, only an
estimated two thirds of students graduate from high schools nationwide.
Students in certain subgroups, such as the economically disadvantaged and
certain racial and ethnic groups, have historically graduated from high
school at substantially lower rates than their peers. Students who do not
graduate from high school are at a serious disadvantage compared to their
peers who do. They are much less likely to obtain good jobs or attend
college. The NCLBA includes several requirements for states to improve
school and student performance, including measuring high school graduation
rates.
NCLBA Requirements
NCLBA expanded the requirements of the Improving America's Schools Act of
1994 (IASA) for states, school districts, and schools to demonstrate that
their students are making adequate progress toward their state's academic
goals. IASA required testing in each of three grade spans to determine
whether a school made adequate yearly progress (AYP). NCLBA requires, by
the 2005-06 school year, that annual tests in math and reading be
administered to students in grades 3 through 8 and once in high school; by
2007-08, students must also be tested in science. In order to make AYP,
schools are to show that increasing numbers of students reach the
proficient level on state tests and that every student is proficient by
2014. NCLBA also designated specific groups of students for particular
focus. These four groups are students who (1) are economically
disadvantaged, (2) represent major racial and ethnic groups, (3) have
disabilities, and (4) are limited in English proficiency.3 For a school to
make AYP, its student
3Students with disabilities refers to students covered under the
Individuals with Disabilities Education Improvement Act of 2004, the
primary law that addresses the unique educational needs of children with
disabilities.
body as a whole and each of the student groups must, at a minimum, meet
the state targets for testing proficiency.
Under NCLBA, schools must also use at least one other academic indicator,
in addition to annual tests, to measure AYP. High schools must use
graduation rate as one of their other academic indicators. The law defines
graduation rate as the percentage of students who graduate from secondary
school with a regular diploma in the standard number of years. Education
officials told us that standard number of years is determined by a state
and is generally based on the structure of the school. For example, a high
school with grades 9 through 12 would have 4 as its standard number of
years while a school with grades 10 through 12 would have 3 as its
standard number of years.
NCLBA regulations specifically require a high school, in order to make
AYP, to meet or exceed its other academic indicators, including what the
state has set as the graduation rate for public high schools. NCLBA does
not specify a minimum graduation rate that states must set. States have
used a variety of methods to measure AYP on their graduation rate
indicator. For example, states have set graduation rate targets or goals
or have allowed schools to show progress toward a target or goal as a way
for schools to meet the graduation rate indicator requirement. The law
does not require states to increase their graduation rate over time.
The law requires states to demonstrate that their definitions produce
graduation rates that are valid and reliable. A valid rate would be one
that measures what it intends to measure. A reliable rate is one which,
with repeated data collections and calculations, produces the same result
each time such collections and calculations are performed. A key aspect of
the reliability of graduation rates is the quality of the data used to
calculate them. The National Center for Education Statistics (NCES),
Education's chief statistical agency, has funded a document that describes
the following dimensions for ensuring that data are of high quality:
o Accuracy. The information must be correct and complete. Data entry
procedures must be reliable to ensure that a report will have the same
information regardless of who fills it out.
o Security. The confidentiality of student and staff records must be
ensured and data must be safe.
o Utility. The data must provide the right information to answer the
question asked.
o Timeliness. Deadlines are discussed, and data are entered in a timely
manner.4
This document suggests that school staff members are responsible for
entering data accurately and completely and maintaining data security. It
provides ideas for assisting staff to accomplish these tasks, such as
sharing best practices with a peer and implementing school-district
policies on data security, such as changing passwords frequently.
If schools receiving funding under Title I, Part A of the act do not make
AYP-including meeting the state's requirements for graduation rates-for 2
consecutive years or more, they are "identified for improvement." They
must take certain actions such as offering parents an opportunity to
transfer students to a school that had made AYP (school choice). If these
schools continue not to make AYP, they must take additional actions, such
as providing supplemental services to students-including transportation,
tutoring, and training.5 States and school districts are required to
provide funding for such actions up to a maximum specified in law.
However, according to Education officials, most high schools do not
receive Title I funding, and therefore, if these schools do not make AYP,
they are not required to take improvement actions, such as offering school
choice or supplemental services. However, NCLBA requires each school
district receiving Title I funds to prepare a report card that must
contain graduation rates for high school students and is available to the
public.
Education's Responsibilities
Education has responsibility for general oversight of Title I of NCLBA. As
part of its oversight effort, Education has implemented the Student
Achievement and School Accountability Program for monitoring each states'
administration of Title I programs. This monitoring effort was designed to
provide regular and systematic reviews and evaluations of how states
provide assistance in terms of funding, resources, and guidance to school
districts to ensure that they administer and implement programs in
accordance with the law. Monitoring is conducted on a 3-year cycle and
4U.S. Department of Education. National Forum on Education Statistics,
Forum Guide to Building a Culture of Quality Data: A School and District
Resource, NFES 2005-801 (Washington, D.C.: 2004).
5Schools designated as in need of improvement under the IASA had their
designation carry over after NCLBA took effect. Also, schools receiving
students through the school choice option must not be identified for
improvement.
addresses high school graduation rates among other requirements. Teams of
federal officials visit state offices, interview state officials, and
review documentation on how states comply with federal law and
regulations. NCLBA also requires the Secretary of Education to report to
the Congress annually regarding state progress in implementing various
requirements, including the number of schools identified for improvement.
Education has required states to report their graduation rates for the
state as a whole and for designated student groups.
All states submitted plans to Education as required under NCLBA, which
were to include their definitions of graduation rates. By June 2003,
Education reviewed and approved all state plans, including their
definitions of graduation rates and their statements regarding how such
rates were valid and reliable. Education provided many states with
approval to use a definition of their choosing until they are able to
develop ones that better meet the law's requirements for defining and
measuring graduation rates. Education has also reviewed and approved many
amendments to plans submitted by states, including those that make changes
to the state's definition of its graduation rate.
Additionally, NCES commissioned a task force to review issues about
definitions, data, and implementation. In its report, the Task Force
discussed the data challenges faced by states in calculating their
graduation rates.6 Regarding data used to measure student performance
generally, GAO and Education's Inspector General have commented on the
importance of data accuracy.7
Dropout Prevention To attempt to improve graduation rates in high schools
or keep students from dropping out of school, Education, state
governments, school districts, schools, and foundations have funded or
implemented various interventions to address the educational needs of
students. Such
6U.S. Department of Education, National Center for Education Statistics.
National Institute of Statistical Sciences / Education Statistics Services
Institute Task Force on Graduation, Completion, and Dropout Indicators,
NCES 2005-105 (Washington, D.C.: 2004).
7GAO, Title I: Education Needs to Monitor States' Scoring of Assessments,
GAO-02-393, (Washington, D.C.: Apr. 1, 2002) and Title I Program: Stronger
Accountability Needed for Performance of Disadvantaged Students,
GAO/HEHS-00-89, (Washington, D.C.: June 1, 2000). U. S. Department of
Education, Office of Inspector General, Department of Education Management
Challenges (November 2004).
interventions are based on the idea that many factors influence a
student's decision to drop out of school, such as low grades,
socio-economic challenges, and disciplinary problems. These factors may be
evident as early as elementary school, and therefore some interventions
are designed for these students.
During the late 1980s and through the mid-1990s, Education supported
dropout prevention programs across the country. In an attempt to determine
which programs effectively reduced the drop out rate, Education conducted
several evaluations of these programs. The largest of these was the
evaluation of the second phase (1991 to 1996) of the School Dropout
Demonstration Assistance Program. This evaluation looked at more than 20
dropout prevention programs including school within a school, alternative
middle and high schools, restructuring initiatives, tutoring programs, and
GED programs. While two of these programs showed promise in reducing
dropout rates-alternative high schools and middle schools-the major
finding was that most programs did not reduce dropping out.8
In our 2002 report, we identified three intervention approaches to prevent
students from dropping out of school:9
o Restructuring schools. This approach modifies a school or all schools
in a district through such initiatives as curriculum reform or dividing
schools into smaller, more individualized learning communities.
o Providing supplemental services. This approach provides additional
services such as tutoring or mentoring in language and math; interventions
attempt to raise student academic achievement and self esteem.10
8Dynarski, Mark and Philip Gleason, How Can We Help? What We Have Learned
from Evaluations of Federal Dropout Prevention Programs? A Research Report
from the School Dropout Demonstration Assistance Program Evaluation
(Princeton, New Jersey: Mathematica Policy Research, Inc., 1998).
9The following two approaches we identified-restructuring and supplemental
services- do not refer to the specific restructuring and supplemental
services provisions in the NCLBA. Instead, these approaches include those
that are more diverse and include a variety of different intervention
practices that states and districts are attempting.
10These included literacy programs, which, although not specifically
discussed in our 2002 report, are also examples of how these approaches
can be implemented.
o Creating alternative learning environments. These interventions target
at-risk students and attempt to create personalized learning environments,
such as career academies that focus the entire school around a specific
career theme.
However, our 2002 report found that additional research was needed to
document which interventions were particularly successful for certain
groups of students. Education agreed that additional rigorous evidence is
needed and that it would consider commissioning a systematic review of the
literature.
Many States Moving A majority of states used or planned to use a
graduation rate definition
based on the group of students entering high school who graduate on
time,toward Using A referred to as the cohort definition. Education has
assisted states, Definition That approved their graduation rate
definitions, and given some states more
time to develop planned definitions intended to produce more precise
Follows Students over results. However, states faced challenges in
resolving common data issues Time; Education's and in providing
information on how to modify definitions to better
account for certain students, such as for those with disabilities.
Guidance Regarding NCLBA Requirements Is Limited
A Majority of States Used or Planned to Use a Definition That Follows
Students over Time
According to state plans, 12 states used a definition that followed a
group of students over time from when they entered high school until they
left- referred to as the cohort definition. An additional 18 states using
other definitions planned to adopt the cohort definition no later than the
2007-08 school year.11 The cohort definition compares the number of 12th
grade graduates with a standard diploma, with the number of students
enrolled as 9th graders 4 years earlier, while also taking into account
those who left the cohort, such as those who transferred in and out.12 A
study
11In July 2005 governors of 47 states signed a compact agreeing to adopt
the National Governors Association's recommended cohort-based graduation
rate formula in order to develop a comparable graduation rate definition.
However, our analysis was based on the state plans rather than on this
agreement.
12States may either track individual students from a 9th grade cohort or
approximate a cohort, such as by estimating the number of students who
enter the 9th grade and who transfer in and out.
commissioned by NCES found that a cohort definition designed to track
individual students over time-from when they enter high school until they
leave-could result in a more precise high school graduation rate than one
calculated with other definitions.13 The data in figure 1 show a
hypothetical high school class from the time students enrolled in 9th
grade until they graduated with a standard diploma, including those who
dropped out, transferred, received alternative degrees, continued in
school, or took 5 years to graduate.
13U.S. Department of Education, National Center for Education Statistics,
National Institute of Statistical Sciences / Education Statistics Services
Institute Task Force on Graduation, Completion, and Dropout Indicators,
NCES 2005-105 (Washington, D.C.: 2004).
Figure 1: Student Mobility and Graduation Outcome for a Hypothetical High
School Class
Source: GAO example.
If the school was in a state that used the cohort definition and
considered 4 years to be on-time graduation, its graduation rate would be
60 percent. The 60 percent figure comes from using the number of students
who started (100), the net number of transfers over the 4 years, and the
number who graduate in 4 years (60).14 Figure 2 shows the formula of the
cohort definition. The year students in the cohort graduate is denoted by
"y,"
14For the purposes of simplifying this example, we set the number of net
transfers over the 4-year period at zero. We recognize that cohorts likely
would have some number of net transfers.
while "T" signifies the net number of students who transfer in and out in
any given year. The cohort definitions actually used by states may vary
somewhat from the basic definition. For example, Kansas used dropout and
transfer data in its definition. Additionally, some states track
individual students, while others track groups of students based on the
entering 9th grade cohort.
Figure 2: Cohort Formula Definition
Source: National Center for Education Statistics (modified by GAO).
According to state plans, 32 states used a definition of high school
graduation rate, referred to as the departure classification definition,
based primarily on the number of dropouts over a 4-year period and
graduates. Essentially, this definition looks back from a 12th grade class
at those who (1) graduated (regardless of when they started high school),
(2) dropped out in 9th, 10th, 11th, and 12th grades (including those who
enrolled in GED programs) and (3) did not graduate, but received some form
of alternative completion certificate.15 So, using this definition, the
data from the high school shown in figure 1 would result in a graduation
rate of 65 percent. The 65 percent figure comes from using the number of
15Ten of these states consider students receiving alternative certificates
separately from dropouts, while the remaining 22 states count them as
dropouts in their definitions. NCES calculates a high school graduation
rate using only diploma recipients as graduates and excluding other high
school completers, such as those who earned a certificate of attendance
and GED certificates. It also calculates a "high school completer rate"
using diploma recipients and other high school completers, except GED
recipients, as completers.
students who graduated (65), the number who received an alternative
certificate (5), and the number who dropped out (30), as shown in Figure
3. Unlike the cohort definition, this definition does not take into
consideration the number of students entering high school 4 years earlier.
As noted earlier, some of these states (13) planned to adopt the cohort
definition by school year 2007-08.
Figure 3: Departure Classification Definition
Source: National Center for Education Statistics (modified by GAO).
The departure classification definition includes students who drop out.
Each of the "D" designations refers to the number of dropouts during one
year. For example "D y-2g10" stands for the number of students who dropped
out in the 10th grade.
Prior to NCLBA, many states had been using a similar version of this
formula, which NCES developed in collaboration with several states.
However, earlier definitions used by states may have also included as
graduates those who receive GED certificates. Under NCLBA, Education
required states to modify the formula so that GED recipients were not
counted as graduates.
Different data systems accommodated the use of different definitions. The
departure classification definition allowed many states to continue using
existing data systems, according to Education officials. Such systems
generally collect aggregate data, rather than data at the student level.
The cohort definition generally requires states to implement a state-level
student tracking system, often with a mechanism that can uniquely
identify each student. Such a system identifies students in the 9th grade
and tracks them throughout high school, indicating whether they graduate,
transfer or drop out. This system also allows for students who transfer
into a school to be placed in the proper cohort.
The more specific information required by the cohort definition may result
in the calculation of more precise graduation rates than those produced by
the departure classification definition. Since the cohort definition
follows students entering high school, either by individual students or
groups of students, it can better be used to include only on-time
graduates. However, how it is implemented may affect the level of
precision of the rate calculated. Tracking individual students may result
in a more precise rate than tracking groups of students.
In our analysis of one state's school year 2002-03 data, we found that the
variations in data collection and calculations between the two types of
definitions, produced different graduation rates. Our analysis showed that
the departure classification definition produced a graduation rate that
was 12 percent greater than when we used the cohort definition.16 Because
the departure classification definition does not track the entering
cohort, it does not account for students who were held back, and therefore
differences may result. Our findings are consistent with observations made
by other researchers that show differences in graduation rates based on
the definition used.17 In addition, NCES plans to complete a study this
year that examines high school graduation rate definitions and how rates
differ depending on the definition used.
According to state plans, the remaining eight states that did not use
either a departure classification or cohort definition used a variety of
other
16We followed the state's version of the cohort definition, which used
dropout rates and not transfers. The basic cohort definition (fig. 2)
accounts for the original number of students in the cohort plus transfers,
while the state's version accounts for dropouts.
17See for example, Greene, J. P. Public School Graduation rates in the
United States (New York: Manhattan Institute for Policy Research, 2002),
http://www.manhattaninstitute.org/pdf/cr_31.pdf (downloaded June 21,
2005); and Swanson, Christopher B.
Keeping Count and Losing Count: Calculating Graduation Rates for All
Students under NCLB Accountability (Washington, D.C.: Urban Institute,
2003), http://www.urban.org/url.cfm?ID=410843 (downloaded June 21, 2005).
definitions. Five of these states plan to adopt cohort definitions no
later than 2007-08.18
Figure 4 shows the definitions each state used as April 2005 and planned
to use by school year 2007-08.
18For example, one state's graduation rate definition divides the number
of graduates by the number of 12th graders at the beginning of the school
year. This definition does not take into consideration the number of
students who dropped out in earlier years, resulting in a higher
graduation rate than would have been produced using a definition that
considered such students. In contrast, 2 states used a dropout rate
definition that divides the number of dropouts in grades 9 through 12 by
the number of students enrolled in those grades for the current year.
Figure 5: State Graduation Rate Targets, as of July 2005
Graduation (in percent)
.Ne
.N.Y
Colo. Alaska
.Va
Ga. Maine
.CD.
h.
sWa
Ill.
.Ore
Okla. Conn.
.xTe
Ariz. Miss. Kans. Vt. R.I.
.Ky
Del.
.hMic
ii
waHa
State
Source: State plans on Education's Web site as of July 7, 2005, with
exceptions (see notes).
Minn.
Mont.
N.H.
.Pa
a.
VW.
.WyoWisc.
.Calif
.NebrMo. Utah
N.Dak. Ala. Idaho
waIo
x.N.Me
N.C. Ohio S.Dak. S.C.
. Md.
.
RP.
nnTe
Notes: These state graduation rate targets were drawn from state plans on
Education's Web site
(http://www.ed.gov/admins/lead/account/stateplans03/index.html) as of July
7, 2005, for all states except Arizona, Colorado, District of Columbia,
Louisiana, Maine, Mississippi, Missouri, New Hampshire, New Jersey, New
York, Oregon, and Puerto Rico. Education provided information on these
states.
This figure includes only those states that were using graduation rates at
the time of our review. States that used other rates, such as dropout
rates, were not included. These states are Arkansas, Indiana, Louisiana,
Massachusetts, and New Jersey. Florida is also not included in this chart
because its requirement is that schools show a one percent annual increase
in their graduation rates.
Valid comparisons of graduation rate targets across states cannot be made,
in part, because of differences in rates used. For example, Alabama and
North Carolina both had targets of 90 percent graduation rates. However,
Alabama arrived at its target by using a departure classification
definition that accounted for dropouts, while North Carolina used a
definition that did not account for dropouts.
According to state plans, 36 states considered their schools as meeting
their graduation rate requirements if the schools increased their
graduation rates from the previous year, known as "showing progress." In
addition, two states allowed their schools to meet such requirements if
they maintained the previous year's rates. A majority of states that
allowed progress as a way for schools to demonstrate they met state
graduation rate requirements had set no minimum rate of progress. We found
instances in which very little progress, less than 1 percent, enabled a
school to meet such requirements. Table 1 shows the number of states that
allow schools to show progress toward the state goals as a means of
meeting state graduation rate requirements, for all states as of the time
we completed our review.
Table 1: Number of States That Allow Schools to Maintain Previous Year's
Rate or Show Progress toward Graduation Rate Targets to Make AYP, as of
July 2005
Progress must be of a specific amount
Maintain previous Any progress year's rate allowed 0.1 percent 1 percent
Othera Total
b
Number of states 2 28 3 4 1
Source: State plans on Education's Web site as of July 7, 2005, with
exceptions (see note).
Note: This information was drawn from state plans found on Education's Web
site. (http://www.ed.gov/admins/lead/account/stateplans03/index.html) as
of July 7, 2005, for all states except Arizona, Colorado, District of
Columbia, Indiana, Louisiana, Maine, Mississippi, Missouri, New Hampshire,
New Jersey, New York, Oregon and Puerto Rico. Education provided
information for these states.
aReduce the difference between the actual and target rate by 10 percent
over a 2-year period.
bThis table does not include 14 states-the 5 states that did not use a
graduation rate (but instead used a different rate, such as a dropout
rate), and the 9 states that did not allow schools to show progress toward
the state graduation rate target to make AYP, but instead required the
schools to meet the target.
By showing progress toward state graduation rate targets, schools can
still make AYP even though they do not meet target rates.19 For example,
our analysis of one state's data from the 2002-03 school year showed that
46 out of 444 high schools made AYP by increasing their graduation rates
toward the state graduation rate target of 66 percent rather than by
meeting or exceeding this target. Specifically, these schools met or
exceeded the state's requirement for 1 percentage point progress in
increasing the graduation rate, even though the schools were below the 66
percent target. Another 232 schools made AYP for the year by meeting or
exceeding the target of 66 percent.
In addition, allowing schools to use progress as the NCLBA graduation rate
indicator could result in schools making AYP annually, while not
19These schools would make AYP, assuming they also met the testing
requirements.
meeting state graduation rate targets for decades, if at all. For example,
a hypothetical school with a graduation rate of 56 percent can meet the
state high school graduation indicator by increasing its graduation rate
by 0.1 percent each year. At this rate, the school would not make the
state graduation rate target of 66 percent for 100 years.
Education's Guidance Did Not Specify Modifications Available to Account for
Certain Students
Education provided states with assistance with their graduation rate
definitions; however, Education's guidance did not specify modifications
available to account for certain types of students. To help states with
their definitions, Education developed some guidance and provided support
such as on-site peer reviews, conferences, and information posted on its
Web site. Education also commissioned a task force that published a report
identifying the advantages and disadvantages of different definitions. In
addition, Education officials told us they granted states time to develop
definitions that met the law's requirements better for defining and
measuring graduation rates.
Education has provided information on how to account for students in
special programs and students with disabilities to states that have
requested it. Education's approach has been to provide such information on
a case-by-case basis rather than to all states. Education officials stated
that they preferred to work with each state's specific circumstances.
However, we found that issues raised, such as students enrolled in 5-year
programs, were common to many states.20 States varied in how they included
students enrolled in these programs in their graduation rate definitions.
For example, one state counted students in 5 year programs who graduated
as dropouts until it received approval to count them as graduates. Another
state planned to count such students as graduates without requesting
approval to do so. Officials in that state said that since it was unclear
what the actual requirements for counting graduates were, they were doing
what they believed was allowable under the law. Without guidance on how to
account for students in special programs and students with disabilities,
there is less consistency among states in how students in these programs
are included in graduation rates.
20This issue is relevant because the number of states that had such a
college component is growing. For example, 19 states had Early College
High Schools as of September 2004 and 25 were projected to as of 2005.
These high schools are designed so that students can receive 2 years of
college credit at the same time as they earn a high school diploma-up to 5
years after starting 9th grade.
Education also has not provided information to all states on how their
definitions can be modified to better accommodate students with
disabilities. State plans in 16 of the 52 states indicated that Education
approved these states to allow students with disabilities more than the
standard number of years to graduate based on the number of years in their
Individualized Education Plans.21 In the 20 states we contacted, we found
that they varied in whether they sought approval from Education on how to
include students with disabilities in their graduation rate definitions.
For example, six of the states we contacted had sought approval from
Education to include students with disabilities who need more than the
standard number of years to graduate in their graduation rate definitions.
In contrast, officials in seven other states contacted told us they did
not seek approval for the same issue. Officials in the remaining seven
states provided no information on this topic or said it did not apply to
them.
State, school district, and school officials and experts we interviewed
reported several factors that affect the accuracy of data used to
calculate graduation rates, especially student mobility. While Education
has taken steps to assist states and districts in improving the quality of
their data, the Department has not reviewed the accuracy of all states'
data, because, at the time of our review, many states were in the process
of implementing new definitions, data collection strategies, or both.
Several Factors Affected the Accuracy of Graduation Rates, and Data Quality
Remains a Key Challenge
Several Factors, Especially Student Mobility, Compromise the Accuracy of
Data Used to Calculate High School Graduation Rates
Officials in six schools, three school districts, and three states we
visited and several experts we interviewed cited challenges in tracking
student mobility, the key factor in calculating accurate high school
graduation rates. Some inaccuracies may lead to the reporting of lower
graduation rates, such as recording all students with "unknown" status as
dropouts or counting students who drop out, return to school, and then
drop out again as a dropout each time, as may happen in schools in states
that use the departure classification definition. Other inaccuracies may
lead to the
21As of July 2005, Education stated that it had received requests from 5
additional states to consider those students with disabilities who receive
a regular diploma as graduates, but take additional years. Education also
received requests from 4 states for similar consideration for Limited
English Proficient students. The remaining plans did not include or did
not address this topic.
reporting of higher graduation rates, such as schools' recording students
who drop out as transfers. This may occur when school staff record such
students as transfers before they receive documentation that the student
actually enrolled in a different school.22 Since the number of dropouts
counts against a school in calculating its graduation rate in many states,
schools that record such students as transfers-because they were unaware
that the students had actually dropped out-may be reporting inflated
graduation rates.23
A second factor that affects data accuracy is how staff members understand
and follow policies and procedures for recording students as transfers to
other schools. For example, staff members in schools in two states
reported that they electronically record a student as having transferred
to another school on the day that student withdraws from their schools.
However, the policy in these states is that a student is to be recorded as
having transferred only upon receiving a request for records from the
school to which the student transfers. In one of these schools, staff
assigned to record student data reported contradictory practices and
beliefs about state policy regarding when to record a student as a
transfer. One staff member stated that the policy and her practice was to
record the student as a transfer upon receiving the records request while
another staff member said that no such policy existed and that she
recorded the student as a transfer on the day of withdrawal. Therefore,
how a student transferring out the school was counted depended on which
staff member recorded the student's data.
The accuracy of data may be further compromised when schools have large
numbers of students who transfer in a given year because the more students
come and go, the more difficult it is for schools to accurately account
for them. Some schools are in areas where families tend to move more
frequently. For example, officials in one school we visited near an Army
base reported that their school had an enrollment of about 1,200 students
and that 187 students had left the school by December of the
22For example, research has shown that this is particularly true for
students with disabilities. See Wagner, Mary, Dropouts with Disabilities:
What Do We Know? What Can We Do? (Menlo Park, Calif.: SRI International,
1991), a report based on the National Longitudinal Transition Study of
Special Education Students. According to the author, a second phase of the
study is under way, and data collected as of June 2005 have shown that
this continues to be the case.
23States were required to provide an assurance that students who drop out
would not be counted as transfers.
academic year. The status of 19 of those 187 students was recorded as
"unknown" because of difficulty in maintaining contact with these
families. The policy in that state was for students whose status is
"unknown" (because they could not be contacted) to be counted as dropouts,
even if, in fact, the student had transferred to another school. Staff in
another school reported the presence of several children from another
country. Their experience has been that these particular students report
plans to return to their country of origin, but they often do not know the
status of these students once they leave the school. The school's
procedure is to record such students as having an "unknown" status, and
these are eventually counted as dropouts, unless another school requests
their records. Research has shown higher mobility rates among certain
subgroups of students compared to all other students, including those who
are African-American, Hispanic, Native American, and those classified as
having limited English proficiency and as children from migrant
families.24 Consequently, schools with higher concentrations of these
subgroups would likely report less accurate graduation rates.
Another factor affecting the accuracy of graduation rate data is the
absence of state audits or verification checks. For example, in our survey
of state officials, over half (27) reported that their states did not
audit the data received from local officials that the state used to
calculate high school graduation rates. The lack of such auditing or
verification implies that states were likely to be unaware of the extent
of certain errors in data-such as students' indicating they were
transferring to another school but not actually doing so-and consequently
were unable to ensure that data they received from schools and districts
were accurate. Officials in only one of the six schools we visited
reported that their data on student transfers had been audited or verified
by an outside party, leaving the accuracy of transfer data in the other
schools uncertain.
A fourth factor that contributes to challenges in assuring accurate data
is the lack of a unique identifier for each student. In our survey,
officials in 22 states reported that their state did not have a unique
identifier for each of their students. Concerns about using student
identifiers include the cost
24Rumberger, Russell, and Scott Thomas, "The Distribution of Dropout and
Turnover Rates among Urban and Suburban High Schools," Sociology of
Education, vol. 73, no. 1 (2000): 39-67. Kerbow, David. "Patterns of Urban
Student Mobility and Local School Reform." Journal of Education for
Students Placed at Risk, vol. 1, no. 2 (1996): 147-169. GAO. Elementary
School Children: Many Change Schools Frequently, Harming Their Education,.
GAO/HEHS-94-45 (Washington, D.C.: Feb. 4, 1994).
of implementing data systems that support such identifiers and privacy
issues. The lack of a unique identifier for students made it difficult to
obtain accurate data. Officials in one state that did not use unique
identifiers stated that they had to compute graduation rates based on
aggregating student data and as a result, they could not track on-time
graduates. Officials in another state estimated that they were only 90
percent accurate in identifying students, because, without a unique
identifier for each student, they had to use other information. Using this
information, such as the student's name or birth date, can lead to
identifying more than one student with the same characteristics, resulting
in inaccurate data used in calculating graduation rates.
A fifth factor we found that may affect data accuracy is variation in
security and accountability practices. For example, we found that while
some schools restricted the ability to change student enrollment
information (such as transfers) to one or two people in the building
(e.g., a registrar), others allowed many staff members to do so. Further,
while some schools' data systems kept a record of each person who accessed
a student's record and the changes made, other systems did not maintain
such information. Without sufficient security and record monitoring, there
is a greater risk of inaccurate data being entered and used to calculate
graduation rates.
Data Inaccuracies May Affect Schools' Meeting State Graduation Rate Goals
We analyzed data from one state to estimate the effect of errors of
various sizes in reporting dropouts on school graduation rates and found
that such errors could raise or lower a school's graduation rate
substantially. This state used a high school graduation definition that
incorporated the number of graduates and dropouts in calculating its
graduation rate. For example, its median high school in school year
2002-03, with 924 students, reported 41 dropouts and had a graduation rate
of 75 percent.25 We reestimated its graduation rate after assuming that
the school had more dropouts, up to twice as many more than reported.26 In
this case, if the school had 82 dropouts, its graduation rate fell to 64
percent. We also reestimated its graduation rate after assuming that it
had fewer dropouts, as few as half as many dropouts as reported. Thus, if
it had 21 dropouts, its
25The median high school in this example is the school in the middle of
all the state's schools when they were rank-ordered according to their
graduation rates.
26Experts we interviewed said that the hypothetical error rates chosen
were reasonable given the quality of dropout data typically maintained by
schools and school districts.
graduation rate rose to 88 percent. Figure 6 shows how the estimates of
graduation rates were affected by assumed errors in counting dropouts for
this school.
Figure 6: Estimated School Graduation Rates under Varying Assumptions of
Errors in Counting Dropouts
Our analysis was performed for all high schools in the state. As expected,
when we assumed the number of dropouts was higher than what schools
reported, their estimated graduation rates decreased.
Our analysis also found the extent to which schools miscount their
dropouts affects their likelihood of reaching the state's graduation rate
target. We estimated that an additional 70 of 444 high schools in the
state in school year 2002-03 would not have reached the state target if
they were in fact reporting only half of their dropouts. On the other
hand, an additional 77 high schools would have reached the state target if
in fact their dropout counts were overreported at twice the actual level.
According to the NCLBA, high schools that do not meet the state's
requirements for its graduation rate are designated as not making AYP.
Such designations if made for 2 or more consecutive years would result in
the district's providing technical assistance to schools that receive
Title I funding. Thus, schools that undercount their dropouts may be
precluded from receiving the attention and assistance from the state they
need to
improve students' school retention and graduation while those with over
counts may receive such services unnecessarily.
Education Has Taken Some Steps to Help States with Data Issues, but Data
Accuracy Remains a Key Challenge
Education has taken steps to help states address data collection issues.
First, Education helped states prepare information to address how their
graduation rate definitions were valid and reliable. Education gave
instructions in its regulations and in a template given to each state to
help states prepare the accountability plans they were to submit to
Education for approval in 2003. Education also worked with states on an
as-needed basis when state officials had questions about what information
the Department needed to review. Education officials indicated that they
reviewed information in each state's plan when they conducted site visits
to states as part of the state plan approval process. According to
Education, most states were in some stage of transition in calculating
their graduation rates: some were implementing plans to transition from
their current definition to a cohort indicator; others were improving
their data systems; and some were collecting information on designated
student groups for the first time. For these states, Education reported
that it was unable to meaningfully examine the reliability of data used to
calculate the graduation rate because such definitions of such rates had
not been in place for a sufficient number of years necessary to determine
whether the rate would produce consistent results.
Second, Education, as part of its state monitoring, introduced a data
review component to examine data states used for graduation rates, among
other aspects of their participation in the Title I program. As of August
2005, Education had monitored and reported on 29 states, and expected to
monitor the remaining states by the end of fiscal year 2006 as part of its
3-year monitoring plan. This monitoring consisted of broad questions
intended to collect information about how states corrected or addressed
errors in student data received from districts and schools, including data
used to calculate graduation rates. The monitoring was also designed to
identify written procedures states used to confirm the accuracy of their
data, the extent to which these procedures were communicated to districts,
and how data validity issues related to schools and districts have been
addressed. According to Education officials, their reviews of the nine
states identified no significant problems with data systems these states
used to calculate high school graduation rates.
Third, in response to recommendations from GAO and Education's Inspector
General, Education contracted with a firm to develop a guide to help
states improve data collection processes. According to Education
officials, this guide is to consist of three parts. One part is designed
for state officials and is to focus on the design and implementation of
data systems. A second part, which focuses on data management issues such
as methods for verifying the accuracy of data, is designed for district
and school officials. A third part summarizes the first two parts and is
to be suitable for oral presentation to state, district, and school
officials. According to department officials, this guide will be issued by
the end of 2005.27
Although Education monitors states to determine if they have written
procedures for ensuring data quality and have methods to address data
quality issues, it does not evaluate other methods of ensuring data
accuracy. For example, it does not assess whether states ensure that
districts and schools have effective controls to accurately record student
status, including transfers. Further, Education's monitoring approach does
not capture whether states ensure that schools have computer controls that
allow only authorized staff to make changes to student data. Department
officials said that the guide it is developing is planned to address these
issues.
However, departmental efforts have not resolved immediate data accuracy
problems. In July 2005, Education announced that it planned to calculate
and report interim graduation rate estimates for each state to provide a
nationwide, comprehensive perspective. Education stated that the interim
rate that it developed, based on data NCES collects from states, will
provide more accurate on-time graduation rates. Some states' graduation
rates rely on the same data reported to NCES, while other states rely on
different data. However, these states also provide data that are requested
by NCES. The quality of the data states provide to NCES varies across
states depending, in part, on the extensiveness and rigor of their
internal controls and other data verification checks. Because Education
plans to rely on state-reported data to calculate interim graduation
rates, the accuracy of such data is critical.28
27The National Forum on Education Statistics issued a similar guide, Forum
Guide to Building a Culture of Quality Data: A School and District
Resource, NFES 2005-801 (Washington D.C.: 2004).
28Education will calculate the rate based on the number of high school
graduates receiving a regular diploma in a given year divided by the
average number of students enrolled in 8th grade 5 years earlier, 9th
grade 4 years earlier, and 10th grade 3 years earlier.
Few Interventions Have Been Rigorously Evaluated, and Education Has Done
Little to Evaluate and Disseminate Existing Research
While states and school districts have implemented numerous interventions
designed to increase high school graduation rates, few of these programs
have been rigorously evaluated, and Education has done little to evaluate
and disseminate existing research. Several of the interventions that have
been rigorously evaluated have shown potential to increase graduation
rates. In addition to these interventions, schools are trying other
approaches to enhance students' chances of success, though the
effectiveness of these approaches has not been demonstrated.
About one third of students who enter high school do not graduate and are
likely to earn less money, are more frequently unemployed, and are more
likely to receive public assistance compared with those who graduate from
high school. In response, some schools and districts have implemented
programs to address the factors that influence a student's decision not to
complete high school.
Research has shown that a student's decision to leave school may be
affected by experiences that begin as early as elementary school. For
example, studies have shown that students who are not at least moderately
skilled at reading by the end of 3rd grade are less likely to graduate
from high school.29 Besides basic literacy skills, there are a variety of
other academic and family-related factors that contribute to whether a
student graduates. For example, poor grades and attendance, school
disciplinary problems, and failure to advance to the next grade can all
gradually lead to disengagement from school and result in a student not
finishing high school. In addition to these academic factors, students
from low-income backgrounds, students with low levels of self esteem, or
students with a learning or behavioral disability drop out at a much
higher rate than other students.
Schools and districts have implemented a range of interventions to address
these factors and they vary in scope from redesigning the structure of an
entire school to an individual school's mentoring program. While there is
variability among interventions, most generally fall into one of the three
following categories that we identified in our 2002 report30: (1) school
wide restructuring efforts; (2) alternative forms of education for
29See, for example, Snow, Catherine E, Susan M. Burns, and Peg Griffin,
Eds. Preventing Reading Difficulties in Young Children (Washington, D.C.:
National Academy Press, 1998).
30GAO-02-240.
students who do not do well in a regular classroom; and (3) supplemental
services, such as mentoring or tutoring services, for at-risk students.
While most of the schools we visited used interventions from only one of
the three categories identified above, some schools combined aspects of
these categories. (See table 2 for a complete list).
Table 2: Number of Interventions Visited by School Level and Type
School Alternative
restructuring learning Supplemental efforts environment services
Number of schools visited
1 Elementary schools
1 Elementary/middle school
2 Middle schools 1
1 Middle/high school
9 High schoolsa 4 7
2 Elementary/middle/high
a
schools 1 1
Source: GAO analysis of interventions visited.
aOne of these schools/programs used more than one approach.
Few Interventions Have Been Rigorously Evaluated, Though Some Showed
Potential to Increase Graduation Rates
Several of the programs at schools we visited have conducted evaluations
of how they affect high school completion, while others are reporting
positive results on other outcomes such as attendance or academic
performance. We identified and reviewed five intervention evaluations that
used a rigorous research design and have shown potential to increase
graduation rates. We visited schools that had implemented three of these
programs.31 In addition, we visited other schools that were trying other
interventions that experts and Education officials noted were promising
for improving high school graduation rates. While the effectiveness of
these approaches to increase graduation rates had not been demonstrated,
research does point towards the possibility that these interventions may
help increase high school completion.
The three schools we visited whose programs displayed positive results all
used a rigorous research design. However, evaluations of the effectiveness
31Two of these evaluations, the Talent Development Model and First Things
First, were released after we had completed our fieldwork.
Promising Approaches: Check and Connect, Project GRAD, Help One Student to
Succeed (HOSTS), Talent Development, and First Things First
of these interventions are not as strong as they need to be for results to
be conclusive. For example, design limitations or data collection concerns
were evident during our review of these evaluations. It is worth keeping
in mind that research of this nature is limited in the education field due
to a variety of factors, and these studies represent some of the most
promising research on graduation rate interventions available.
In our visits to 16 school programs in 6 states, we observed 3
interventions where research has indicated potential for improving high
school graduation rates. These interventions addressed a variety of
student risk factors and provided services to students in elementary
through high school.
One school we visited in Minneapolis, Minnesota, had implemented the Check
and Connect program which provides mentoring services in an
alternative-learning environment. The program began in 1990 with a model
developed for urban middle school students with learning and behavioral
challenges. It has since been expanded to serve additional at-risk
populations as well. This intervention is designed around a mentor who
acts as both an advocate and service coordinator for students who have
been referred into the program due to excessive absences combined with
poor academic performance and behavioral problems. Program officials noted
that the mentors offer around-the-clock services including monitoring
school performance, regularly checking student data (attendance, grades,
and suspensions), and identifying and addressing out of school issues. The
mentor also regularly communicates with the student's parents or relatives
to ensure that the whole family is engaged in the student's education.
The mentoring is built into a program model that relies on several
interrelated features including relationship building, individualized and
timely intervention, and long-term commitment. A complete listing of
program features can be seen in table 3.
Table 3: Key Features of the Check and Connect Model
Feature Definition
Relationship building Mutual trust and open communication, nurtured
through a long-term commitment that is focused on student's educational
success.
Routine monitoring of alterable indicators
Systemically checking warning signs of withdrawal (attendance, academic
performance, behavior) that are readily available to school personnel and
that can be altered through intervention.
Individualized and timely
intervention Support that is tailored to individual student needs, based
on level of engagement with school, associated influences of home and
school, and the leveraging of local resources.
Long-term commitment Committing to students and families for at least 2
years, including the ability to follow highly mobile youth from school to
school and program to program.
Persistence plus Refers to a persistent source of academic motivation, a
continuity of familiarity with the youth and family, and a consistency in
the message that "education is important for your future."
Problem solving Designed to promote the acquisition of skills to resolve
conflict constructively and to look for solutions rather than a source of
blame.
Affiliation with school and learning Facilitating student's access to and
active participation in school-related activities and events.
Source: Check and Connect Web site, http://ici.umn.edu/checkandconnect/.
The school we visited in Minneapolis had 220 students in the program
during the 2004-05 school year. Program officials noted that students in
the program were divided among four mentors and had two separate
classrooms they could use to meet with their mentor or to study between
classes. The program had no set schedule for the student-it was the
responsibility of the mentor to make sure they followed up with the
students, parents, teachers, courts or counselors on a regular basis. A
student in the program noted that Check and Connect helps because it
"provides someone who cares how you do and keeps after you about coming to
school and doing well academically."
A school official remarked that both attendance and retention rates had
improved significantly since the program was implemented. An evaluation of
program impacts on students with emotional and behavioral disabilities
showed that students participating in Check and Connect were more likely
than students not participating to have either completed high school,
including GED certification, or be enrolled in an educational program.32
While graduation rates are not available yet for the first Check and
Connect cohort at the school we visited, a teacher at the school
32Sinclair, M.F., S. L. Christenson and M. L. Thurlow, "Promoting School
Completion of Urban Secondary Youth with Emotional or Behavioral
Disabilities." Exceptional Children (in press).
commented that the staff knows that the program is working "because the
students are coming to class everyday." School officials noted that the
program is funded through a renewable grant from a private foundation.
Another program we visited, Project GRAD (Graduation Really Achieves
Dreams), seeks to ensure a quality public education for students in
economically disadvantaged communities through school restructuring,
curriculum reform, and social services. The goal of the program is to
increase high school graduation rates in Project GRAD schools to at least
80 percent, with 50 percent of those students entering and completing
college.
Originally established in 1989 as a scholarship program, it has since
developed into a replicable and comprehensive k-12 school reform model.
The reform design relies on two components-a structural model and an
instructional model. Structural components include an independent local
organization to provide implementation oversight, and community
involvement such as mentoring, tutoring, and financial support. Figure 7
shows Project GRAD's structural components.
Figure 7: Project GRAD Structural Model
Source: GAO graphic rendition of Project GRAD structure,
http://www.projectgrad.org.
Local Project GRAD sites-such as one located in Atlanta-also used the
instructional component of the model, which emphasizes specific reading
and math programs for students in kindergarten through 8th grade. Program
officials commented that this component also incorporates campus based
social services (which focus on dropout prevention as well as family case
management), classroom management techniques, and college scholarships to
all high school students who qualify.
In 2004, the local Atlanta site served 29 schools and approximately 17,000
students in the inner city. Officials at one of Atlanta's schools noted
that the program provided additional outreach staff to advocate on behalf
of students and address other issues that may interfere with a student's
ability to attend school and learn. Students at the school, commenting on
the program's effect on their lives, noted that the program should be
expanded to all of the schools in the district because of the
opportunities it offers students. Project GRAD-Atlanta officials noted
that the effectiveness of the program has been demonstrated through higher
test scores and increased college attendance since implementing Project
GRAD in these schools. Additionally, the results of an independent
evaluation of Project GRAD also suggest an increase in students' test
scores and graduation rates.33 However, aspects of the study's design may
limit the strength of study findings.
The Project GRAD-Atlanta model relies on a mix of public funding and
private local fundraising. As of school year 2003-04, Project GRAD had
also been replicated in feeder systems in Akron, Ohio; Brownsville, Tex.;
Cincinnati, Ohio; Columbus, Ohio; Houston, Tex.; Kenai Peninsula, Alaska;
Knoxville, Tenn.; Lorain, Ohio; Los Angeles, Calif.; Newark, N.J. and
Roosevelt, N.Y.
We also visited a school that had implemented the language arts component
of the HOSTS program, an intervention focused on literacy, an area that
research has linked to students' graduating. This program is a structured
tutoring program in reading and language arts that targets low performing
elementary students whose reading skills are below grade level. School
officials at the elementary school we visited noted that they had been
using the program for 7 years to increase at-risk student's reading scores
as well as raise their self esteem. The 90 students in the
33Opuni, K.A., Project GRAD Newark: 2003-2004 Program Evaluation Report
(Houston, Texas: Center for Research on School Reform, February 2005).
program worked individually with a tutor 4 days a week for 30 minutes each
day. School officials considered the program a success because of the
number of students who successfully transitioned into grade level reading
in the regular classroom. The program, which has been replicated in
schools or districts in 12 states, was cited in the report language of the
NCLBA as a scientifically based intervention that has assisted schools in
improving student achievement. A recent study of the program in nine
Michigan elementary schools suggests reading improvement for students at
schools participating in HOSTS programs.34 While this study displayed some
promising results for elementary literacy, students were not tracked over
time to determine its effect on high school graduation rates.
Two recently completed rigorous program evaluations also displayed
promising results for increasing graduations rates. These two programs,
the Talent Development Model and First Things First, are both
comprehensive school reform initiatives with numerous components.
The Talent Development program in Philadelphia, Pennsylvania, is designed
to improve large urban high schools that face serious problems with
attendance, discipline, achievement scores, and graduation rates. The
program has been implemented in twenty districts nationwide and consists
of several different components including a separate career academy for
all 9th graders, career academies for students in 10th through 12th
grades, block scheduling (4 courses a semester, each 80-90 minutes long)
and an after hours program for students with attendance or behavioral
problems. An evaluation of the first five schools in Philadelphia to
implement the Talent Development program suggest that it may have
contributed to increasing the graduation rate for two high schools
compared with other high schools in the district that did not implement
the program.35
The First Things First program was first launched in Kansas City, Kansas,
and has since been tested in 12 middle schools and high schools in four
additional districts. The program has three central components: small
34Burns, Matthew K., Barbara V. Senesac, and Todd Symington, "The
Effectiveness of the HOSTS Program in Improving the Reading Achievement of
Children At-risk for Reading Failure." Reading Research and Instruction,
vol. 43, no. 2 (2004): 87-104.
35Kemple, James J., Corinne M. Herlihy, and Thomas J Smith. Making
Progress Towards Graduation: Evidence from the Talent Development High
School Model (New York: Manpower Demonstration Research Corporation, May
2005).
learning communities of up to 350 students, a family advocate system that
pairs students with a staff member who monitors their progress, and
instructional improvement that aims to make lessons more rigorous and
better aligned with state and local standards. A recent evaluation in
Kansas City schools suggests that students in the four high schools with
First Things First had increased reading and math scores, improved
attendance, lowered dropout rates, and increased graduation rates compared
with schools that did not participate in the program.36 For middle schools
in Kansas City, the study found increased reading and math scores and
somewhat improved attendance compared with other scores. However, the
research did not show significance differences in the First Things First
schools when compared with other schools in two other school districts.
Approaches Selected Schools Are Trying to Enhance Students Chances for
Success
School-Restructuring Efforts: Making Schools Smaller
In addition to the 3 school programs we visited whose rigorous evaluations
displayed potential for increasing graduation rates, we also visited 13
other school programs which experts, Education officials, and evaluations
noted were promising. While the effectiveness of these approaches has not
been demonstrated, research points toward the possibility that these
interventions may help increase high school completion. These other school
programs generally focused on one specific approach which generally fell
into one of three categories-school restructuring, alternative learning
environment, and supplemental services. Selected programs that illustrate
these approaches are discussed below.
Schools and districts used schoolwide restructuring to change a school or
all schools in the district to provide a more personalized education and
increase graduation rates. Schoolwide restructuring efforts are generally
implemented in schools or districts that have a history of high dropout
rates.
One restructuring approach is to create many small schools from larger low
performing schools. For example, the New Century High Schools Consortium
for New York City is a New York Public School's small schools initiative
that is funded by the Bill and Melinda Gates Foundation,
36Quint, Janet, Howard S. Bloom, Alison Rebeck Black, LaFleur Stephens,
and Theresa M. Akey, The Challenge of Scaling Up Educational Reform:
Findings and Lessons from First Things First (New York: Manpower
Demonstration Research Corporation, July 2005).
the Carnegie Corporation of New York, and the Open Society Institute.
School officials commented that the project began in the Bronx with the
conversion of six low performing high schools that served between 1,500
and 3,000 students each. This intervention began in 2001 and, as of
September 2004, New York City had created 77 small schools.
One of those schools, Morris High School, has been a part of this program
since the small schools program begun in 2001. School officials noted that
the school has been divided into several small schools including the Bronx
International High School and the Bronx Leadership Academy, which serve
300 and 252 students respectively. While housed in the same building, each
school has a different curriculum and student population. For example, the
Bronx International High School provides an intensive English language
program for recent immigrants while the Bronx Leadership Academy offers a
science-based curriculum for college bound students. The core concepts for
both these programs are the small school size, team approach to teaching,
and school-based learning that also has relevance within their community.
A student at the school noted that the small groups they work in allow
students to help and support each other, something that did not happen in
junior high school. School officials commented that teacher investment in
the school is expected and is often displayed by working overtime, serving
as counselors to students, and participating in school governance.
Additionally, the project-based curriculum is developed by teacher teams
who work collaboratively to plan activities for incoming students.
School officials did not indicate a plan for a formal outcome-based
evaluation of the schools; however, they did consider the intervention a
success based on positive improvement in a number of areas including
higher percentages of students meeting state standards, higher attendance
rates, and higher passing grades. The New York City Department of
Education reported similar results for small schools throughout the
district including more students advancing from 9th to 10th grade and
higher attendance rates. While these results provide a snapshot of some
possible benefits of New York's school reform initiative, it is still too
early to look at student outcomes. The Gates Foundation has commissioned
an 8-year evaluation of the small schools program.
Alternative-Learning Environment: Providing Individualized Education
States and school districts are also using alternative learning
environments for students at risk of school failure. These interventions
are designed to foster a supportive school environment through small
enrollments, one-onone interaction, flexible schedules, and structures,
and a curriculum that appeals to students' interests.37 Often, enrollment
is limited and the programs are tailored to individual students' needs to
ensure that they graduate.
One type of alternative learning environment, the career academy, is
focused on keeping students in school by providing an interesting
curriculum focused on a specific career theme. For example, Aviation High
School in Washington State is an aviation-themed public high school housed
at a local community college. School officials noted that the school
addresses a range of student risk factors, including those related to
academics (learning and literacy), social issues (attendance and
behavior), and family (counseling and strategies for living with drug
addicted family members). With a 2004 enrollment of only 103 students,
Aviation High School offers small class sizes, aviation themed curriculum,
and mentoring opportunities. (See figure 8 for an example of a school
event focused on aviation).
37Lehr, Camilla A. and Cheryl M Lange, "Alternative Schools Serving
Students with and without Disabilities: What Are the Current Issues and
Challenges," Preventing School Failure, vol. 47, no. 2 (2003): 59-65.
Figure 8: Aviation High School Presentation by the Blue Angels
Source: Aviation High School, Seattle, Washington.
Additionally, school officials report that each teacher at the high school
serves as a student advisor who assists students with academic, social,
and emotional development. Students noted that while transportation to the
school was challenging due to its distance from their home, they still
selected the program because of the aviation curriculum, the personalized
attention they received, and the highly motivated students at the school.
Aviation High School officials indicated that it is too soon to tell the
impact of the program, but they noted that the school will be included in
a national evaluation to be conducted by the Gates foundation. Research on
career academies has demonstrated positive gains for employment and
earnings for graduates, but also found that high school completion rates
of career and non career academy students were not significantly
different.38
38Kemple, James J., and Judith Scott-Clayton, Career Academies: Impact on
Students' Initial Transitions to Post-Secondary Education and Employment
(New York: Manpower Demonstration Research Corporation, December 2001).
Supplemental Services: Targeting Literacy and Self-Esteem
Alternative learning environments may also allow students to tailor their
learning experience to individual needs that are not being met in
traditional schools. For example, we visited an alternative high school in
Atlanta, Georgia, that uses a computer-based instructional program
designed for students to learn the state-certified curriculum at their own
pace. Students rotate through classrooms, each of which contains a
different computer module for the particular subject being taught.
Students received assistance from teachers as needed. According to
officials, the school is made up of a team of 6 teachers and 75 at-risk
11th and 12th grade students (for the 2004-05 school year). The school's
enrollment is composed of students who were referred to the school either
through other schools, court tribunals, or parents. School officials noted
that the program also includes a motivational component. For example, each
school morning begins with an assembly where students discuss the
obstacles they have had to overcome and the people who have helped make a
difference in the world. After the assembly, students get up and shake
hands with each other and then move to their first hour class. School
personnel stated that this allows students to begin each day with
confidence and prepares them to learn. School officials noted that the
school's graduation rate, which they stated was consistently over 90
percent, indicated that the program was effective.
Research on alternative programs in general has shown some promising
outcomes. For example, an evaluation of 8 middle school dropout prevention
programs showed some positive impacts on dropout rates, grade promotion,
grades, and test scores for students in alternative programs.39 The same
study also looked at five alternative high school programs and found
limited evidence that these programs reduced dropout rates, but did note
that alternative programs oriented toward GED certificates experience were
more effective than those oriented toward high school diplomas.40
Several schools we visited used targeted supplemental services to provide
at-risk students with extra help. These services aim to improve students'
academic performance, acclimate them to a new culture, or increase their
self-esteem. Supplemental service programs are offered at all grade
levels,
39Dynarski, Mark, Philip Gleason, Anu Rangarajan, and Robert Wood, Impacts
of Dropout Prevention Programs, Final Report (Princeton, New Jersey:
Mathematica Policy Research, Inc., 1998).
40Ibid.
with research showing the importance of building academic and social
skills at an early age.
Supplemental services can focus on the needs of a specific group of
students, such as immigrant students or students with limited English
proficiency. One such intervention we visited in Georgia was designed to
provide educational and cultural services to immigrant students with low
level English skills and limited formal schooling. These interventions,
often referred to as "newcomer" models, provide intensive language
development courses and may also offer a cultural orientation component.
Newcomer programs can take place within a school or at a separate site and
vary in the amount of time a student is enrolled. The benefits of the
newcomer program is supported by research on English language learners
that notes one major factor that decreases risk of dropping out of school
is level of understanding and mastery of the English language.41
At the program we visited, international students who were new to the
district were registered, tested, and placed depending on their skill
level. Students with no English language skills were placed in an
intensive 3 - to 6-week English program that helped ease the transition
into school. Students who were 14 years or older and had fewer than 7
years of formal schooling in their native country were placed in the
English for Speakers of Other Languages (ESOL) lab program. School
officials noted that the lab served 132 students in school year 2004-05
and is designed to help students achieve grade level proficiency within 3
years. The ESOL lab focused on listening, speaking, reading, and writing
English in addition to other core high school courses such as math,
science, and social studies. Additionally, several district schools have
added Saturday school tutorials for parents and students. Students can
study language arts while their parents attend citizenship classes,
orientation, and career awareness sessions. School officials noted that
they believe the number of ESOL students graduating has increased, based
on state-reported rates as well as the numbers of students who pass the
ESOL tests and exit the program.
Other supplemental services incorporate cultural elements as a means of
addressing student self-esteem. For example, a k-8 school located on the
Arapahoe Indian reservation in Wyoming offers all students services that
41See for example, Gingras, Rosano, and Rudy Careaga, Limited English
Proficient Students at Risk: Issues and Prevention Strategies (Silver
Spring, Md.: National Clearinghouse for Bilingual Education, 1989).
include after-school academic programs, drug awareness events, and a 2week
summer cultural camp focusing on Native American traditions. School
personnel emphasized that the path to high school graduation begins with
helping students address their self-esteem issues. School officials
mentioned that students already have a mindset that they are not going to
graduate from high school and do not have a future on or off the
reservation. The cultural element of the school's programs is a
significant component of building up the student's self-esteem and
instilling a pride about their Native American identity. Students
commented that they participated in the program because of the Native
American cultural activities offered, including clogging, dancing, and
drumming. Program officials noted that since implementing interventions
designed specifically to address the issues of Native Americans, they have
noticed general improvement in student attitudes and performance. While
studies suggest that self-esteem affects dropout rates,42 a study over
time of the intervention programs used by the Arapahoe school would be
needed to determine its effectiveness.
Education Has Done Little to Evaluate and Disseminate Knowledge about
Interventions
Graduation rates have become increasingly important since the passage of
NCLBA, but Education has done little to evaluate and disseminate knowledge
about interventions that could help increase such rates. The increased
interest in high school reform by the National Governors Association,
combined with concerns about low graduation rates, have set the stage for
designing strategies that encourage more students to graduate. While many
types of interventions are available for school districts, most have not
been rigorously evaluated, and there is little information on which are
successful and for what student subgroups. Most officials from the 20
states we included in our study told us that such information would be
useful. For example, one school official noted that little information
exists on what interventions increase graduation rates among Native
American students and that such information would be helpful in designing
interventions.
Education has made some efforts to address the problem of high school
completion by sponsoring research and disseminating information through
conferences and on its Web site. For example, Education officials noted
42See for example, Shannon, Sue G., and Pete Bylsma, Helping Students
Finish School: Why Students Drop Out and How to Help Them Graduate
(Olympia, Wash.: Office of Superintendent of Public Instruction, 2003).
that Education's Office of Special Education Programs has supported
research papers on dropout interventions for youth with disabilities.
These studies are currently being completed and will be available in late
2005. In terms of dissemination, Education's 2nd Annual High School
Leadership Summit held in December 2004 included sessions on dropout
prevention and recovery as well as strategies for creating
higher-performing schools. Additionally, Education's Office of Vocation
and Adult Education has dedicated a part of its Web site to the High
School Initiative. The pages on the Web site contain information on high
school reform models, adolescent literacy initiatives as well as
information on research based practices that may help high schools.
While Education has made some efforts to help states and districts address
the dropout problem, the agency has not acted on its commitment to
implement the recommendation, contained in our 2002 report on
interventions, that Education evaluate results from research. Agency
officials have commented several times that they plan to evaluate the
research on dropout prevention efforts and then disseminate the results
through the agency's What Works clearinghouse. However, the Web space for
this effort still contains placeholder information.43 Agency officials
indicated that reviews of other topics, such as elementary reading and
math, have come before the reviews necessary for the dropout section of
the Web site.
Conclusions
The nation's public school systems are responsible for educating 48
million students, the majority of our future workforce. Providing them
with the skills needed to succeed is vital to the nation's economic
strength and ability to compete in a global economy. NCLBA was passed to
ensure that all students have access to a high-quality education and to
increase the likelihood that these students will graduate. In particular,
the act seeks to make significant changes in public education by asking
federal, state, and local education officials to reconsider how they
assess the academic achievement of the nation's students. NCLBA specifies
that states must set high school graduation rate indicators as an
additional benchmark, along
43The What Works Clearinghouse, funded by Education, has a Web site that
will summarize evidence on the effectiveness of different programs,
products, practices, and policies intended to improve student outcomes.
The site is planned to include interventions in middle school, junior high
school, or high school designed to increase high school completion
including such techniques as the use of incentives, counseling, or
monitoring as the prevention/intervention of choice.
with test results, for measuring schools' progress. However, increasing
and accurately calculating graduation rates have been formidable
challenges for many states and districts. Many states have used
flexibility to define their indicators as both numerical goals as well as
progress toward those goals, where progress has generally ranged from no
increase to a 1 percent increase from the previous year. Therefore, some
states have set expectations that their schools may not graduate many more
students than previously.
Education has addressed these challenges by developing some guidance and
providing support such as on-site peer reviews, conferences, and
information on its Web site. However, because Education's approach has
been to provide guidance on how to deal with specific student
circumstances on a case-by-case basis, not all states have received such
guidance. Without guidance, state officials may not appropriately include
students in these specific circumstances in their graduation rate
definitions, resulting in graduation rates that may be inaccurate. Such
inconsistent calculations raise questions about the quality of graduation
rates reported by states.
A key challenge for states is to ensure that student data used for
calculating state graduation rates, as well as data provided to NCES, are
accurate and that state systems have the internal controls and data
verification checks to promote data reliability. As some states transition
to new graduation rate definitions, it is important that they ensure that
such controls are part of new student data systems. Student data accuracy
is particularly important because Education plans to use those state data
reported to NCES to develop interim graduation rate estimates, which are
intended to promote consistency across states and provide a nationwide
perspective.
Finally, little is known about the success of interventions that are
designed to increase high school graduation rates. While some programs
have shown potential to increase such rates, few have been rigorously
evaluated. Some interventions have conducted limited evaluations of a
variety of different outcomes (attendance, test scores, job attainment),
but more comprehensive evaluations are necessary to understand programs'
effects on graduation rates. As a result, schools and districts may not be
using the most effective approaches to help their students stay in school
and graduate. Education could play an important role in evaluating
existing research, which was a recommendation we made in our 2002 dropout
report. Although Education agreed with this recommendation, the agency has
not established a clear plan or timetable for carrying it out.
Recommendations for Executive Action
Agency Comments and Our Evaluation
Additionally, Education should disseminate the results of research, since
such information will be critical as high school reform moves forward.
To assist states in improving their definitions of high school graduation
rates and enhancing the consistency of these rates, we recommend that the
Secretary of Education make information available to all states on
modifications available to account for students in special programs and
students with disabilities in their graduation rate calculations. This
information could include fuller explanations or examples of available
flexibilities.
We recommend that the Secretary of Education, before developing interim
graduation rate estimates, assess the reliability of data submitted by
states used for this purpose. This assessment could include specific
criteria that demonstrate that states' data systems can produce accurate
data.
We recommend that the Secretary establish a timetable for carrying out the
recommendation in our 2002 report that Education evaluate research on
dropout interventions, including those interventions that focus on
increasing graduation rates. In addition, we recommend that the Secretary
disseminate research on programs shown to be effective in increasing
graduation rates.
We provided a draft of this report to Education for review and comment. In
its letter, Education concurred with two of our three recommendations: (1)
about making information available to all states on modifications
available to account for students in special programs and students with
disabilities in their graduation rate calculations and (2) about
evaluating research on dropout interventions and disseminating such
research on those programs shown to be effective in increasing graduation
rates. Regarding our recommendation that that the department assess the
reliability of data submitted by states that it plans to use to develop
interim graduation rate estimates, Education noted that it has taken a
number of steps to conduct such reliability assessments. However, it is
not clear whether these efforts include those data that Education will be
using to develop interim graduation rate estimates. Although data
submitted to Education are publicly available and have been reported by
states for years, their reliability has not been determined. We believe
that Education should take additional steps to ensure the reliability of
these data before they are used in calculating such estimates. Education
officials also provided technical comments that we incorporated into the
report where appropriate. Education's written comments are reproduced in
appendix II.
We are sending copies of this report to the Secretary of Education,
relevant congressional committees, and other interested parties. We will
also make copies available to others upon request. In addition, the report
will be made available at no charge on GAO's Web site at
http://www.gao.gov.
Please contact me at (202) 512-7215 if you or your staff have any
questions about this report. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. Other contacts and major contributors are listed in
appendix III.
Marnie S. Shaul, Director Education, Workforce, and Income Security Issues
Appendix I: Objectives, Scope, and Methodology
To address the objectives of this study, we used a variety of
methodological approaches. We analyzed the plans states were required to
submit to Education to identify the graduation rate definitions states
used and graduation rate indicators set by states, reviewed updates to
plans submitted through July 2005 and reviewed letters from Education to
states regarding its decisions about state plans and updates.
As part of another GAO review, we surveyed officials in 50 states, the
District of Columbia, and Puerto Rico to obtain information about two
issues-the extent to which (1) states verify school and district data used
to calculate high school graduation rates and (2) have unique student
identifiers. The surveys were conducted using self-administered electronic
questionnaires posted on the World Wide Web. We sent e-mail notifications
to all 52 state Performance Based Data Management Initiative coordinators
(50 U.S. states, the District of Columbia, and Puerto Rico) beginning on
November 15, 2004. We closed the survey on January 13, 2005, after the
50th respondent had replied. Washington state and the District of Columbia
did not complete the survey in time to be included in our analysis.
We selected 20 states for further analysis. States were selected to
capture variation in high school graduation rate definitions, geographic
location, and types of interventions with the potential to increase
graduation rates. We conducted
o a case study in 1 state (Washington state) to calculate graduation
rates;
o site visits in 3 states (Georgia, North Carolina, and Washington) to
review data accuracy;
o site visits in 6 states (Georgia, Illinois, Minnesota, New York,
Washington, and Wyoming) to observe interventions and interview program
staff; and
o semi structured telephone interviews in all 20 states to obtain
information on definitions used, implementation status, and guidance
provided by Education.
See table 4 for a list of states selected for site visits and phone
interviews based on the research objective we studied.
Appendix I: Objectives, Scope, and Methodology
Table 4: States Selected for Site Visits and Phone Interviews by Purpose
To address the first research To address the first research To address the
second To address the third question regarding data question regarding
rationale for research question regarding research question regarding
definitions and calculations selecting definitions data accuracy
interventions
a
Washington California Colorado Connecticut Delaware Florida Georgiaa
Illinois Indiana Kansas Massachusetts Minnesota Mississippi New Hampshire
New Mexico New York
a
North Carolina Pennsylvania
a
Washington Wisconsin Wyoming Georgiaa Georgiaa
aa
North Carolina Illinois
a
Washington Minnesotaa New Yorka
a
Washington Wyominga
No. of states: 1 20 3 6
Source: GAO Analysis.
aStates where GAO team conducted site visits.
In our case study we used student data from Washington state for the
2002-03 school year, the most recent school year for which data were
available at the time of our review. Using these data, we conducted an
analysis comparing the results of calculating the high school graduation
rate using two different graduation rate definitions-the cohort definition
and the departure classification definition. Washington state used a
modified cohort formula that was based on tracking student dropouts rather
than on tracking student transfers.1 It also required all students with
"unknown" status to be reported as dropouts. We also used these data to
analyze the effects of allowing schools to make progress toward the
graduation rate target as a means of making AYP and using an estimated
miscount of the number of dropouts on the graduation rate. We interviewed
experts to determine reasonable rates at which dropouts may be in error.
We analyzed data using a set of 444 out of 547 of the state's
1Generally, cohort definitions are based on tracking student transfers.
Appendix I: Objectives, Scope, and Methodology
high schools. The 103 high schools that were not included in our analysis
were those with graduation rates of 10 percent or less. These were
generally alternative high schools, such as those designed to serve
students who had committed serious crimes. We also interviewed a state
official who confirmed our understanding of the omitted schools and agreed
with the reasonableness of the criterion.
Although our analyses were based on a 4-year period, we used the 1 year of
student data and estimated information for the 3 prior years. We did not
obtain student data from prior years because state officials told us that
data accuracy had improved significantly in the 2002-03 school year. We
assessed the reliability of the Washington state data by (1) performing
electronic testing of required data elements for missing data and for
obvious errors, (2) reviewing existing information about the data and the
system that produced them, and (3) interviewing Washington state officials
knowledgeable about the data. However, we did not check the data to source
information. We determined that the data were sufficiently reliable for
the purposes of this report.
To identify interventions with the potential to increase graduation rates,
we used a "snowballing" approach. Using this approach, we reviewed the
literature on interventions and interviewed Education officials and
dropout prevention experts and reviewed Web sites, such as the National
Dropout Prevention Centers Web site (http://www.dropoutprevention.org/),
to identify those that have the potential to increase high school
graduation rates. Based on the research we reviewed and on recommendations
from experts, we selected several interventions at various locations
around the country. For those interventions we selected to visit we
reviewed available evaluations, including the findings related to
outcomes, such as increased graduation rates and improved literacy. We
also assessed the methodological approaches of these evaluations. Based on
our review, we identified 3 interventions that had been rigorously
evaluated and have shown potential to increase graduation rate and visited
3 schools that had implemented these programs. (Rigorous evaluations of 2
other interventions which showed promising results were released
subsequent to our field work. We reviewed the results of these evaluations
and reported their findings.) We also visited schools that had implemented
13 other interventions that experts and research showed promise in
affecting factors that may improve grad rates. However, rigorous
evaluations on these programs had not been done at the time of our review.
Appendix I: Objectives, Scope, and Methodology
To determine how Education assists states, we reviewed Education
regulations, guidance, and other documents and interviewed Education and
state agency officials. We also interviewed these officials to determine
the degree to which Education's actions have enhanced and disseminated
knowledge about interventions. Finally, we interviewed officials from the
National Governors Association, national education organizations, and
other experts in the area of high school graduation rates and reviewed
related research to obtain an understanding of the issues surrounding
these rates and high school reform efforts to address them. We conducted
our work between September 2004 and July 2005 in accordance with generally
accepted government auditing standards.
Appendix II: Comments from the Department of Education
Appendix II: Comments from the Department of Education
Appendix II: Comments from the Department of Education
Appendix II: Comments from the Department of Education
Appendix II: Comments from the Department of Education
Appendix III: GAO Contact and Staff Acknowledgments
GAO Contact
Staff Acknowledgments
Marnie S. Shaul, (202) 512-7215, [email protected]
Harriet Ganson (Assistant Director), Julianne Hartman Cutts
(Analyst-in-Charge), and Jason Palmer (Senior Analyst) managed all aspects
of the assignment. Dan Klabunde made significant contributions to this
report, in all aspects of the work. In addition, Sheranda Smith-Campbell,
Nagla'a El-Hodiri, and Greg Kato provided analytic assistance. Jean
McSween, Karen O'Conor, and Beverly Ross provided technical support. Jim
Rebbe and Sheila McCoy provided legal support, and Corinna Nicolaou
assisted in the message and report development.
Related GAO Products
No Child Left Behind Act: Improvements Needed in Education's Process for
Tracking States' Implementation of Key Provisions. GAO-04-734. Washington,
D.C.: September 30, 2004.
No Child Left Behind Act: Additional Assistance and Research on Effective
Strategies Would Help Small Rural Districts. GAO-04-909. Washington, D.C.:
September 23, 2004.
Special Education: Additional Assistance and Better Coordination Needed
among Education Offices to Help States Meet the NCLBA Teacher
Requirements. GAO-04-659. Washington, D.C.: July 15, 2004.
Student Mentoring Programs: Education's Monitoring and Information Sharing
Could Be Improved. GAO-04-581. Washington, D. C.: June 25, 2004.
Title I: Characteristics of Tests Will Influence Expenses; Information
Sharing May Help States Realize Efficiencies. GAO-03-389. Washington,
D.C.: May 8, 2003.
Title I: Education Needs to Monitor States' Scoring of Assessments.
GAO-02-393. Washington, D.C.: April 1, 2002.
School Dropouts: Education Could Play a Stronger Role in Identifying and
Disseminating Promising Prevention Strategies. GAO-02-240. Washington,
D.C.: February 1, 2002.
Elementary School Children: Many Change Schools Frequently, Harming Their
Education. GAO/HEHS-94-45. Washington, D.C.: February 4, 1994.
Bibliography
Burns, Matthew K., Barbara V. Senesac, and Todd Symington. "The
Effectiveness of the HOSTS Program in Improving the Reading Achievement of
Children At-Risk for Reading Failure." Reading Research and Instruction,
vol. 43, no. 2 (2004): 87-103
Dynarski, Mark, Philip Gleason, Anu Rangarajan, and Robert Wood. Impacts
of Dropout Prevention Programs, Final Report. Princeton, New Jersey:
Mathematica Policy Research, Inc., 1998.
Dynarski, Mark, Philip Gleason, Anu Rangarajan, and Robert Wood. Impacts
of School Restructuring Initiatives, Final Report. Princeton, New Jersey:
Mathematica Policy Research, Inc., 1998.
Dynarski, Mark and Philip Gleason. How Can We Help? What We Have Learned
From Evaluations of Federal Dropout Prevention Programs? A Research Report
from the School Dropout Demonstration Assistance Program Evaluation.
Princeton, New Jersey: Mathematica Policy Research, Inc., 1998.
Gingras, Rosano, and Rudy Careaga. Limited English Proficient Students at
Risk: Issues and Prevention Strategies. Silver Spring, Maryland: National
Clearinghouse for Bilingual Education, 1989.
Greene, J. P. and Marcus A. Winters. Public School Graduation Rates in the
United States (New York: Manhattan Institute for Policy Research, 2002),
http://www.manhattan-institute.org/pdf/cr_31.pdf (accessed June 21, 2005).
Kemple, James J. Career Academies: Impacts on Labor Market Outcomes and
Educational Attainment. New York: Manpower Demonstration Research
Corporation, December 2001.
Kemple, James J., Corinne M. Herlihy, and Thomas J. Smith. Making Progress
towards Graduation: Evidence from the Talent Development High School
Model. New York: Manpower Demonstration Research Corporation, May 2005.
Kerbow, David. "Patterns of Urban Student Mobility and Local School
Reform." Journal of Education for Students Placed at Risk. vol. 1, no. 2
(1996): 149-171.
Bibliography
Lehr, Camilla A. and Cheryl M. Lange. "Alternative Schools Serving
Students with and without Disabilities: What Are the Current Issues and
Challenges." Preventing School Failure, vol. 47, no. 2 (2003): 59-65.
Opuni, K. A. Project GRAD Newark: 2003-2004 Program Evaluation Report,
Houston, Texas: Center for Research on School Reform, February 2005.
Quint, Janet, Howard S. Bloom, Alison Rebeck Black, LaFleur Stephens
LaFleur, and Theresa M. Akey. The Challenge of Scaling Up Educational
Reform: Findings and Lessons from First Things First, New York: Manpower
Demonstration Research Corporation, July 2005.
Rumberger, Russell, and Scott Thomas. "The Distribution of Dropout and
Turnover Rates among Urban and Suburban High Schools." Sociology of
Education, vol. 73, no. 1 (2000): 39-69.
Sinclair, M. F., S. L. Christenson, and M. L. Thurlow. "Promoting School
Completion of Urban Secondary Youth with Emotional or Behavioral
Disabilities." Exceptional Children, (in press).
Snow, Catherine E., M. Susan Burns, and Peg Griffin, Eds. Preventing
Reading Difficulties in Young Children. Washington, D.C.: National Academy
Press, 1998.
Swanson, Christopher B. Keeping Count and Losing Count: Calculating
Graduation Rates for All Students under NCLB Accountability. Washington,
D.C.: Urban Institute, 2003, http://www.urban.org/url.cfm?ID=410843
(downloaded June 21, 2005).
Shannon, Sue G., and Pete Bylsma. Helping Students Finish School: Why
Students Drop Out, and How to Help Them Graduate. Olympia, Washington:
Office of Superintendent of Public Instruction, 2003.
U.S. Department of Education, National Center for Education Statistics,
National Forum on Education Statistics. Forum Guide to Building a Culture
of Quality Data: A School and District Resource. NFES 2005-801.
Washington, D.C.: 2004.
U.S. Department of Education, National Center for Education Statistics.
National Institute of Statistical Sciences / Education Statistics Services
Institute Task Force on Graduation, Completion, and Dropout Indicators.
NCES 2005-105. Washington, D.C.: 2004.
Bibliography
Wagner, Mary. Dropouts with Disabilities: What Do We Know? What Can We Do?
A Report from the National Longitudinal Transition Study of Special
Education Students. Menlo Park, California: SRI International, 1991.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Gloria Jarmon, Managing Director, [email protected] (202)
512-4400Congressional U.S. Government Accountability Office, 441 G Street
NW, Room 7125 Relations Washington, D.C. 20548
Public Affairs Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
PRINTED ON RECYCLED PAPER
*** End of document. ***