Education for Disadvantaged Children: Research Purpose and Design
Features Affect Conclusions Drawn From Key Studies (Letter Report,
08/31/2000, GAO/HEHS-00-168).

Pursuant to a congressional request GAO compared the Department of
Education's Longitudinal Evaluation of School Change and Performance
Plan (LESCP), to its previous assessment program, the Prospect Study,
focusing on: (1) the purpose of each study and how these purposes relate
to the needs of policymakers and educators; (2) the process used to
design and implement the studies, including obtaining feedback from
review panels and releasing the results to Congress and the public ; and
(3) the studies' strengths and limitations in light of their purposes
and determine the effect these strengths and limitations have on the
conclusions that can be drawn from the data.

GAO noted that: (1) while Education staff, contractors, and members of
advisory panels all agreed that the purpose of the Prospects study was
to assess the effectiveness of Title I, of the Elementary and Secondary
Education Act considerably less agreement existed on the precise purpose
of the LESCP study; (2) these differences centered primarily around how
much emphasis the study would place on the Title I program; (3) the lack
of agreement about LESCP's purpose created unclear expectations for the
study, making it difficult to predict the degree to which the final
report will meet the needs of Congress other policymakers, and
educators; (4) the LESCP and Prospects studies used similar processes
for contracting out data collection and analysis, obtaining comments
from review panels, and releasing the results; (5) both studies were
conducted by outside research organizations under contracts with
Education; (6) for both Prospects and LESCP, Education and the
contractors obtained advice from two panels of experts--a
congressionally mandated review panel that advised Education on policy
issues and a separate technical panel that provided feedback specific to
the individual study; (7) the panels that advised the Prospects study
had proportionally greater representation from educators, while LESCP's
panels have proportionally more researchers; (8) for both the LESCP and
Prospects studies, the 3 to 4 year longitudinal data collection and the
complexity of the implementation process posed challenges for providing
study results in time to meet Congress' 5-year reauthorization schedule;
(9) to provide information for reauthorization, Education issued interim
reports to Congress in both cases; (10) for both Prospects and LESCP,
major design features influenced the study's ability to address its
overall purpose; (11) the design of each study has both strengths and
potential limitations; (12) the LESCP study is not yet complete, but the
depth of information provided by its longitudinal focus is likely to be
a key strength; (13) however, LESCP, which uses a smaller,
nonrepresentative sample, suffers from design limitations that will
restrict its ability to fully satisfy any of the three potential
purposes envisioned by Education, contractors, and panel members; and
(14) because several design limitations are directly related to
measuring the effects of Title I and standards based reform, they will
likely have a major effect on the conclusions that could be drawn in
these areas of education policy.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-00-168
     TITLE:  Education for Disadvantaged Children: Research Purpose and
	     Design Features Affect Conclusions Drawn From Key
	     Studies
      DATE:  08/31/2000
   SUBJECT:  Education program evaluation
	     Comparative analysis
	     Economically depressed areas
	     Disadvantaged persons
	     Aid for education
	     Academic achievement
	     Performance measures
	     Educational standards
IDENTIFIER:  Dept. of Education Longitudinal Evaluation of School
	     Change and Performance Plan
	     Dept. of Education Title I Program

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Testimony.                                               **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO/HEHS-00-168

Appendix I: Comparison of Prospects and LESCP Studies

22

Appendix II: Comments From the Department of Education

25

IRP Independent Review Panel

LESCP Longitudinal Evaluation of School Change and Performance

RFP request for proposals

TWG Technical Working Group

Health, Education, and
Human Services Division

B-285997

August 31, 2000

The Honorable William F. Goodling
Chairman, Committee on Education and the Workforce
House of Representatives

The Honorable Michael N. Castle
Chairman, Subcommittee on Early Childhood, Youth
and Families
Committee on Education and the Workforce
House of Representatives

The Honorable Peter Hoekstra
Chairman, Subcommittee on Oversight
and Investigations
Committee on Education and the Workforce
House of Representatives

Title I, the federal government's largest program for elementary and
secondary education, is primarily directed at assisting disadvantaged
children in high-poverty schools. To inform the 1994 and impending
reauthorizations of this important program,1 the Congress has required the
Department of Education to conduct a national assessment of Title I--a broad
study to examine the overall progress of students served by the program and
the implementation of its key provisions. Education has conducted a number
of research studies to support these national assessments. Two of these
studies gathered data on Title I students over several years--the Prospects
study, completed in 1997, and the ongoing Longitudinal Evaluation of School
Change and Performance (LESCP),

which is due to be completed in 2001.2 Recently, concerns have been raised
about the quality and usefulness of the LESCP study. These concerns have
focused on the quality of the study design, whether Education obtained
sufficient input from external experts, and the extent to which Education
provided data from the study to inform the Congress's debate over
reauthorization in a timely manner. You asked us to review and compare LESCP
to its predecessor, the Prospects study. In this report, we (1) review the
purpose of each study and how these purposes relate to the needs of
policymakers and educators; (2) describe the process used to design and
implement the studies, including obtaining feedback from review panels and
releasing the results to the Congress and the public; and (3) analyze the
studies' strengths and limitations in light of their purposes and determine
the effect these strengths and limitations have on the conclusions that can
be drawn from the data.

We reviewed the interim and final reports from the Prospects study, the
interim report from the LESCP study, and the final reports of the 1993
National Assessment of Chapter 1 and the 1999 National Assessment of Title
I.3 We also reviewed internal Education documents relating to the design and
analysis of each study. We analyzed the minutes of all the meetings of the
current Independent Review Panel (IRP), a congressionally mandated advisory
panel to Education, and reviewed the panel's final report to Congress. We
also reviewed the minutes of all the meetings of the LESCP Technical Working
Group (TWG), a panel of expert researchers convened by Education to advise
the Department and its contractors on the study. Finally, we interviewed
Education staff, including Title I program staff and staff of the Planning
and Evaluation Service, which directed both studies; IRP and TWG members;
the project directors for the LESCP and Prospects studies from the primary
contracting organizations; and outside

experts. We performed our work between June and August 2000 in accordance
with generally accepted government auditing standards.

While Education staff, contractors, and members of advisory panels all
agreed that the purpose of the Prospects study was to assess the
effectiveness of Title I, considerably less agreement existed on the precise
purpose of the LESCP study. These differences centered primarily around how
much emphasis the study would place on the Title I program. For example,
although some expected LESCP to serve as another vehicle for evaluating
Title I's overall effectiveness, other individuals expected LESCP to
evaluate specific education reform policies implemented under the 1994
reauthorization of Title I. Still other individuals expected the LESCP study
to focus primarily on changes in instructional practice, with Title I issues
treated more as a contextual factor than as a major focus of the analysis.
This lack of agreement about LESCP's purpose created unclear expectations
for the study, making it difficult to predict the degree to which the final
report will meet the needs of the Congress, other policymakers, and
educators.

The LESCP and Prospects studies used similar processes for contracting out
data collection and analysis, obtaining comments from review panels, and
releasing the results. Both studies were conducted by outside research
organizations under contracts with Education. For both Prospects and LESCP,
Education and the contractors obtained advice from two panels of experts--a
congressionally mandated review panel that advised Education on policy
issues and a separate technical panel that provided feedback specific to the
individual study. The panels that advised the Prospects study had
proportionally greater representation from educators, while LESCP's panels
have proportionally more researchers. For both the LESCP and Prospects
studies, the 3 to 4 year longitudinal data collection and the complexity of
the implementation process posed challenges for providing study results in
time to meet Congress's 5-year reauthorization schedule. To provide
information for reauthorization, Education issued interim reports to
Congress in both cases.

For both Prospects and LESCP, major design features influenced the study's
ability to address its overall purpose. The design of each study has both
strengths and potential limitations. For example, Prospects' large,
representative sample supported fairly strong conclusions about the effect
of Title I, although the study design did not allow researchers to
definitively measure how students would have performed if they did not have
access to Title I services. The LESCP study is not yet complete, but the
depth of information provided by its longitudinal focus is likely to be a
key strength. However, LESCP, which uses a smaller, nonrepresentative
sample, suffers from design limitations that will restrict its ability to
fully satisfy any of the three potential purposes--evaluation of Title I,
standards-based reform, or instructional practices--envisioned by Education,
contractors, and panel members. Because several design limitations are
directly related to measuring the effects of Title I and standards-based
reform, they will likely have a major effect on the conclusions that could
be drawn in these areas of education policy. They will, however, have a less
serious effect on the conclusions that could be drawn about instructional
practices.

Title I of the Elementary and Secondary Education Act was established in
1965 to help schools meet the needs of economically and educationally
disadvantaged students. Title I's $7.9 billion budget is targeted
primarily--but not exclusively--to schools with a relatively high percentage
of students from low-income families. Although it is the federal
government's largest elementary and secondary education program, Title I
accounts for a small share of total education expenditures--about 3 percent.
However, for many poor schools Title I is a key source of funding for items
such as supplementary instruction, professional development, new computers,
and after-school or other extended-time programs.

In 1994, the Congress made several significant changes to the program.
First, the Congress changed the rules for allocating Title I dollars in an
effort to direct more funding to the neediest schools. Second, in the 1994
reauthorization the Congress increased the number of schools eligible to use
Title I funds on a schoolwide basis. Title I has traditionally expected
schools to direct program funds to students who are low-achieving or at
highest risk for school failure (called targeted assistance). However, a
provision known as the schoolwide program allows a school to spend its Title
I funds to improve the school as a whole, rather than targeting Title I
dollars to low-achieving students. Because all students in the school can
benefit, under a schoolwide program it is more difficult to distinguish the
effect of Title I services from the effect of a school's overall
instructional program. The number of schools adopting the schoolwide
approach has increased dramatically, to nearly half of all Title I schools,
since the 1994 reauthorization expanded eligibility for schoolwide status.

In the 1994 legislation, the Congress also established a new reform policy,
commonly called standards-based reform. Under these new requirements, states
(in addition to their general oversight responsibilities) were to collect
and report information on educational outcomes and to hold schools and
districts accountable for results. Specifically, states were required to
develop content standards to describe what students need to know and
performance standards to describe their expected proficiency in at least the
core subject areas of reading or language arts and mathematics. States were
then to establish assessments to measure how students are doing in relation
to these standards. Each state must report the results of assessments. In
addition to the overall results, states must collect and publicly report
assessment results by six specified student categories--gender, racial and
ethnic group, English proficiency status, disability status, migrant status,
and economic status. States must also develop criteria for determining
whether schools and districts are performing satisfactorily.4 These
standards-based reform requirements were designed to increase Title I's
focus on educational outcomes for all students, not just those children who
had been traditionally targeted under the program.

To support the national assessments of Title I, Education issued contracts
to outside researchers to conduct several major projects. For the earlier
reauthorization, Education's major Title I research effort was the Prospects
study, a large longitudinal study of Title I students and schools. The
Prospects study also responded to an additional, more specific congressional
mandate that required Education to conduct a longitudinal study to compare
students who received Title I services with students who did not. Prospects
began gathering student data in 1991; although not completed until 1997, the
study played an important role in informing the 1994 reauthorization of
Title I. In the more recent national assessment, Education relied on data
from several studies. One of these studies, LESCP, is also a longitudinal
study of Title I schools. Although the legislative mandate for the most
recent national assessment required longitudinal data, the legislative
requirements were less specific than those set out in the mandate that
governed the Prospects study.

for LESCP

Education staff, panel members, contractor staff, and the interim and final
reports from the Prospects study consistently described the purpose of the
Prospects study as evaluating the effectiveness of the Title I program. This
purpose reflected the specific legislative mandate for the Prospects study.
Consistent with this purpose, the main finding of the Prospects study was
that students who received Title I services started below the achievement
level of their peers and that this initial gap remained essentially
unchanged as students moved into higher grades. (See app. I for more
information about the findings of the Prospects study.) The Prospects data
directly addressed congressional needs for information on overall program
effectiveness, and it also provided some useful information for other
policymakers and educators.

Considerably less agreement existed among Education staff, panel members,
and contractors on the precise purpose of the LESCP study, especially as it
relates to the Title I program. LESCP's legislative requirements were less
specific than those of the Prospects study, which responded to a specific
congressional mandate. Education developed the following study questions for
LESCP: (1) To what extent are changes occurring in what is being taught in
reading and mathematics in the classrooms in the study? (2) To what extent
are changes occurring in how instruction is being delivered? (3) To what
extent are students showing changes in performance? and (4) How do recent
revisions in Title I contribute to these changes? Despite having these study
questions, Education staff, panel members, and contractors--all of whom were
closely connected with the LESCP project--differed in their view of the
study's purpose. These differences appeared to be primarily concerned with
the emphasis the study would place on the fourth study question--the role of
Title I. For some staff and panel members, the fourth question was the key
research question for the project. For others, the first three questions
were central to the study, and Title I was to be treated more as a
contextual factor than a major focus of analysis. As a result, individuals
seeing the same study questions held very different views on the overall
purpose of the project. For example, while several individuals stated that
LESCP's purpose was to evaluate the effectiveness of the Title I program,
others told us that this was not the study's purpose. Similarly, several
people expected LESCP to evaluate standards-based reform, while others
stated that evaluating standards-based reform was beyond the study's scope.
Yet others expected LESCP to focus primarily on the effect of instructional
practices on student achievement.

LESCP's purpose was not fully clarified by the release of an interim report.
The report reflected a primary emphasis on studying the effect of
instructional practices on student achievement. However, several panel
members expected the interim report to include more information on Title I
or standards-based reform. The report found that certain instructional
practices, such as having students talk in small groups about what they
read, were correlated with improved student performance in fourth grade
reading. The report also found that there was little change in these
instructional practices between 1997 and 1998. (See app. I for more
information on the findings of the interim LESCP report.) Education expects
some findings related to standards-based reform to be included in the final
LESCP report, which is due for release in 2001. In the absence of the final
report, however, the divergent views on LESCP's purpose make it difficult to
predict whose needs the study will address--the Congress's need for
information related to Title I effectiveness, Education's and the states'
needs for information on evaluating the importance of various
standards-based reform practices, or educators' needs for information on the
most effective instructional practices for promoting student achievement.

Release of Data

The process for implementing a longitudinal study--including contracting
with outside research organizations and obtaining expert advice on important
implementation issues--can have a major effect on the quality and timeliness
of the final product. For both Prospects and LESCP, Education emphasized
quality by gathering longitudinal data and consulting with expert panels to
obtain advice and feedback on important implementation issues. In both
cases, however, gathering data over 3 to 4 years and following a multistep
contracting process posed challenges for providing information to the
Congress in time to meet the Title I reauthorization schedule.

Both the Prospects and LESCP studies were conducted by outside research
organizations under contracts that included data collection, analysis, and
report writing. Education used a similar competitive bidding process for
issuing these contracts. Under this process, Education staff developed a
request for proposals (RFP) that included detailed specifications for each
study. For example, the RFP for LESCP specified key design features such as
the sample size, characteristics of the sample, data elements (including
both state assessments and standardized exam results, for example), and the
time frame for data collection. Potential contractors then submitted
proposals to Education. Education staff and peer reviewers evaluated these
proposals on the basis of the quality of the proposal, the contractors'
experience in education evaluation, and the contractor's bid. After the
award, a designated Education staff member supervised the project and
monitored contractor progress. For both projects, the process for awarding
contracts took several months.

LESCP Studies

For both the Prospects and LESCP studies, Education convened two types of
expert panels--one to provide broad policy guidance and another to provide
expert advice on the technical aspects of the study. The policy panel--the
IRP--advised Education on broad issues that cut across several studies,
while the TWG provided advice and guidance on technical issues specific to
each study.

For each of the two previous Title I reauthorizations, the Congress required
Education to establish an IRP to advise the Department on its research
efforts concerning the National Assessment of Title I. In its quarterly
meetings, the IRP has provided feedback on the relationship between broad
education reform issues and Education's research efforts and has raised
issues about the limitations of specific research projects. Each IRP has
also chosen to submit its own report to Congress to accompany Education's
report on the National Assessment of Title I. These reports provide the
panels' perspective and recommendations on important reauthorization issues,
such as ensuring high-quality professional development for teachers.

Congress required the IRPs to include researchers, state and local
practitioners, and other appropriate individuals. Both panels included
representatives from academia, education associations, state and local
education agencies, and others. However, the current panel includes a
greater percentage of researchers and association representatives, while the
1993 panel included a greater percentage of educators, especially from the
local level. Fig. 1 details the affiliations of IRP members in 1993 and
1999.

1999

In contrast to the IRPs, the TWGs are designed to assist the Department and
the contractors on a single study. Initially, contractors recommend TWG
panel members in their proposals, and Education selects the panel once the
award has been made. TWG members are primarily researchers, selected to
provide subject area and methodological expertise in areas important to the
study. The TWG's role is to provide guidance and advice in implementing the
study design, analyzing the data, and writing the report. For example, the
LESCP TWG reviewed the survey instruments and made suggestions to reduce the
length of the proposed teacher questionnaire. However, the TWG members are
generally not involved in the early study design because the group is not
formed until after many of the major design decisions have already been
made. Moreover, TWG input cannot result in major modifications to the study
design because the contract, which includes detailed design specifications,
has already been finalized and may need to be re-competed.

TWGs were used on both the Prospects and LESCP studies. The Prospects study
also sought advice from a "stakeholders group," which included
representatives from state education agencies, local educators, and
advocates. However, Education staff told us that the two panels often
duplicated efforts, with both groups raising similar issues. For the LESCP
study, Education incorporated representatives from state education agencies
into the TWG, instead of convening a stakeholders group. Combined with the
reduced number of educators on the IRP, the elimination of the stakeholders
group resulted in fewer educators--especially at the local level--involved
in advising Education on LESCP as compared with the Prospects study. Several
individuals involved with the LESCP study stated that additional input from
educators, particularly in the early stages of study design, would have been
helpful. However, Education staff and some panel members also told us that
it was sometimes difficult to get educators to participate in their advisory
panels.

Meeting Reauthorization Schedule

Both the Prospects and LESCP studies faced challenges in trying to provide
Congress with timely information to inform reauthorization of Title I. Title
I was scheduled to be reauthorized every 5 years. However, for each study,
more than 5 years were needed to design the study, issue the contract,
obtain the required clearance for the study instruments from the Office of
Management and Budget, collect and analyze several years' worth of data, and
complete the final report. Both Prospects and LESCP provided some
information for reauthorization through interim reports. Education also may
provide to Congress preliminary information from the studies before their
release. According to Education officials, the decision to provide such
information is generally based on several considerations, including (1) the
quality of the data, (2) Education's confidence that the preliminary
findings will be consistent with the final results, (3) the importance and
usefulness of the information to policymakers, and (4) in some cases,
feedback from IRP and TWG members.

For both the LESCP and Prospects studies, key design features influence the
studies' ability to draw conclusions on Title I, school reform, or
instructional practice. The design of each study has both strengths and
potential limitations. For example, Prospects' large, representative sample
supported fairly strong conclusions about the effect of Title I, although
the study design did not allow researchers to definitively measure how
students would have performed if they did not have access to Title I
services. The LESCP study is not yet complete, but the depth of information
provided by its longitudinal focus is likely to be a key strength. However,
limitations to the LESCP study--particularly the study's small,
nonrepresentative sample--will restrict researchers' ability to draw strong
conclusions from the data, especially about Title I or standards-based
reform.

Draw Strong Conclusions About Title I

The Prospects study was one of the largest and--at a total cost of $28.8
million--most expensive studies Education has conducted on the Title I
program. Prospects data were gathered on a large, nationally representative
sample of 372 schools from 1991 to 1994. Students in these
schools--including some students that moved--were followed for 3 to 4 years.
The Prospects study also collected information on a large number of
important factors at the student, family, teacher, school, and district
levels. These factors included student and family characteristics,
characteristics of the school, teacher standards and expectations, and
instructional practices. (See app. I for more information.)

Its rigorous and comprehensive design allowed the Prospects study to support
fairly strong conclusions about the effect of the Title I program on student
achievement. For example, because Prospects collected data from a large,
nationally representative sample, researchers could draw stronger
conclusions about the effect of Title I services nationwide than would have
been possible with a smaller, nonrepresentative sample. The Prospects data
provided information on how Title I services affected student achievement
over a 3-year period, rather than providing only a "snapshot" of the
program's effect. In addition, the Prospects study captured the experiences
and outcomes of highly mobile students--an often underserved population that
is at risk for low achievement and dropping out of school.

Prospects' extensive and detailed information at the district, school,
teacher, parent, and student levels created a rich database on students
across the country, allowing researchers to explore a variety of factors
associated with student educational outcomes. For example, the Prospects
study found that the poverty level of the school (over and above the
economic status of an individual student) was negatively related to
standardized achievement scores and that mobility also had a negative effect
on students' academic performance. The researchers also explored
relationships between teachers' instructional practices and student
outcomes. The study found that an emphasis on comprehension and the
development of writing skills were generally positively related to student
achievement in reading, and an emphasis on remedial instruction was
positively related to math test scores in first through third grade.

Although its comprehensive design clearly strengthened the final study,
several design choices limited the conclusions that could be drawn from
Prospects' data.

ï¿½ Prospects' large sample size, although clearly a major strength, led
researchers to rely on self-reported survey data in several areas, including
instructional practice. Because respondents tend to select socially
desirable choices and to overestimate their own progress, the quality of
these data may be questionable.

ï¿½ The Prospects study concentrated largely on students in the elementary
grades. While elementary school students constitute the majority of those
served by Title I, the program also extends to middle and high school
students. Prospects' data provided less information about these students'
progress under Title I.5

ï¿½ The Prospects study attempted to compare Title I students with similar
students that did not receive Title I services.6 However, because Title I
students were selected by principals and teachers who believed these
students most needed extra help, the students who received Title I services
may have been more educationally disadvantaged than their non-Title I
counterparts. Because students were selected for Title I rather than being
randomly assigned, the study design did not allow researchers to
definitively measure how the same students would have performed in the
absence of the program.

ï¿½ The Prospects study also did not develop clear criteria to measure the
effectiveness of the Title I program. As a result, different individuals
have interpreted Prospects' findings differently in terms of the success or
failure of Title I. Some individuals have interpreted Prospects' results as
evidence that the Title I program may not be effective because Title I
students did not rise to the achievement level of their peers. Others
disagree, saying that without Title I, gaps between Title I students and
their peers would have grown over time rather than remain constant.

ï¿½ Prospects did not provide information on the cost of Title I services in
conjunction with their effect on student achievement, limiting policymakers'
ability to determine whether the program is a good investment of federal
dollars.

Another key limitation of Prospects came about not because of the study
design, but because major changes to the Title I program were implemented
after the data were gathered in 1991 through 1994 and before the final
Prospects report was issued. Consequently, it is difficult to determine how
to apply Prospects' results to the redesigned Title I program. For example,
at the time the Prospects data were collected, relatively few Title I
schools were operating schoolwide programs. By 1999, however, nearly half of
Title I schools were operating schoolwide programs. Because schoolwide
programs may differ from targeted assistance schools in the way they deliver
services, the results from the Prospects study may be less applicable to the
Title I program as it is currently structured.

Especially About Title I or Standards-Based Reform

The ongoing LESCP study is smaller and (at a total cost of about $9 million)
considerably less costly than the Prospects study. Between 1997 and 1999,
LESCP gathered data on a small, nonrepresentative sample of 71 Title I
schools from 18 districts in 7 states. LESCP measured student achievement
using a national standardized test, similar to the test used in Prospects;
however, LESCP also collected student achievement data as measured by state
assessments. Like Prospects, the LESCP study also collected information on a
large number of factors at the student, family, teacher, school, and
district levels. These factors included student and family characteristics,
characteristics of the school, teacher professional development, and
instructional practices. (See app. I for more detailed information on
LESCP's study design).

Because the LESCP study is ongoing, complete information is not available to
fully assess the study's strengths and limitations. However, as was true for
Prospects, LESCP's design features will contribute to the strengths and
limitations of the study and directly affect the conclusions that can be
drawn from the data. For example, one strength is that LESCP's data will
describe the study schools over a period of several years. In addition,
because LESCP gathered detailed data at the district, school, teacher,
parent, and student levels, the study can be expected to provide a depth of
information on a variety of issues facing these high-poverty schools.

Despite these important strengths, the LESCP study is unlikely to fully
satisfy any of its three potential purposes--evaluation of Title I,
standards-based reform, or instructional practices--as envisioned by
Education, contractors, and panel members. LESCP's major design limitations
include the following:

ï¿½ Most important, LESCP results cannot be generalized beyond the small,
nonrepresentative sample of schools, districts, and states used in the
study. The small sample size may also make it difficult to compare study
results across special populations of students attending Title I schools,
such as students with disabilities or students with limited proficiency in
English.

ï¿½ Because district officials chose the schools that participated in the
study, the sample is likely to include a greater percentage of
higher-achieving schools compared with the population as a whole.

ï¿½ LESCP's study design, like that of Prospects, relied primarily on
self-reported survey data to provide information on important variables such
as the implementation of standards-based reform policies and teachers'
instructional practices. Because respondents tend to select socially
desirable choices and to overestimate their own progress, the quality of
these data may be questionable.

ï¿½ Because LESCP primarily focused on schools rather than students, the study
did not follow students who changed schools. Since mobility is associated
with lower student achievement, the absence of students who moved is likely
to result in overestimates of student achievement in the sample as a whole.
Furthermore, this effect may be stronger for some schools in the sample than
for others.

In addition to the broad limitations described above, several LESCP design
features will restrict the study's ability to draw conclusions about the
effects of standards-based reform and Title I, which are important areas for
Congress to consider in the reauthorization of Title I. However, these
additional limitations of the study design are likely to have a less serious
effect on potential conclusions about instructional practices. Additional
limitations include the following.

ï¿½ No consensus has emerged on how to define and measure standards-based
reform. Furthermore, state and local implementation of standards-based
reform varies in ways that can be difficult to measure consistently across
states and districts.7 For example, state standards vary considerably in
content and level of detail. LESCP measured teachers' ratings of how
familiar they were with state standards and assessments, but by design such
measures do not capture the content of the standards themselves. The
challenges in measuring these practices will make it more difficult to
interpret LESCP's results.

ï¿½ Changes over time in the implementation of standards-based reform policies
may also limit LESCP's ability to tie specific reform policies to student
achievement. The seven states in the sample were selected to include five
states that had already implemented policies related to standards-based
reform and two that had not yet done so. However, by the time most of the
data were collected, the states in the sample that had been slower to
implement these policies had largely caught up with the others in the
sample. As a result, there was limited variation in standards-based reform
across the seven states, making it more difficult to link this reform to
student achievement.

ï¿½ Because LESCP's sample included only schools that receive Title I dollars,
the study design will not allow researchers to compare Title I schools with
schools that did not receive Title I funding. Without this comparison, it
will be difficult to draw conclusions about the program's overall
effectiveness.

ï¿½ Although Title I services are provided to students in preschool through
high school, LESCP data extended only to students in the third through fifth
grades, providing an incomplete picture of the effect of Title I services.

ï¿½ The large majority of schools (58 of 71) in the LESCP sample operated
schoolwide programs, and these schools generally had higher poverty rates
than the 13 targeted assistance schools in the sample. With such a small
number of schools, LESCP will not be able to address a major gap in existing
Title I research by drawing conclusions about how the performance of
schoolwide programs compares to targeted assistance schools with similar
poverty rates.

ï¿½ Like Prospects, LESCP did not include information on the cost of Title I
services in conjunction with their effect on student achievement, thus
limiting policymakers' ability to determine whether the program is a good
investment of federal dollars.

In addition to the limitations imposed by the study design, LESCP's ability
to provide information on the effectiveness of Title I may be complicated by
changes in the program since the 1994 reauthorization. Most important,
growth in the number of schoolwide programs has complicated efforts to
determine the effect of Title I because, under the schoolwide approach, it
is more difficult to distinguish the effect of Title I services from the
effect of the school's overall instructional program. Consequently, using
LESCP data to assess the effect of Title I funds would be even more
difficult than such an analysis was for the Prospects study.

In discussing the limitations of the LESCP study, Education staff, panel
members, and contractors pointed to two factors that they believe made this
study especially challenging to design and implement. First, they pointed
out that, over the life of the project, the budget for the LESCP study will
be less than one-third of what was spent on Prospects, even without
adjusting for inflation. With additional funding, Education staff and panel
members told us, LESCP could have included a larger, more representative
sample, a wider span of grades, and more qualitative and quantitative data
on instructional practice and the implementation of standards-based reform.
Second, some individuals stressed the inflexibility of the contracting
process, which requires Education to determine detailed study specifications
in advance. For example, several panel members pointed out that Education
and its contractors may have only a limited ability to respond to panel
members' comments on the study's design because making design changes may
require Education to re-open the competitive bidding process.

Both the Prospects and LESCP studies produced valuable information, but no
single study can fulfill the diverse information needs of the Congress,
Education, the states, and principals and teachers. Moreover, individuals
often have different views about which research questions are most important
to address. Given this diversity, it is especially important to clarify the
purpose of research studies so that the potential users of the research can
anticipate how they will be able to use the information. In the case of
Prospects, there was general agreement on the study's purpose, and the
completed study provided information that was closely related to that
purpose. There was much less agreement, however, on the overall purpose of
the LESCP project. As a result, even those individuals closely involved with
LESCP did not always know what information to expect from the study.
Furthermore, a clear statement of purpose is not sufficient, in and of
itself, to guarantee a strong research design--especially in a complex field
such as education where many related factors contribute to student
achievement. Nonetheless, when the purpose of a study is clear, it becomes
easier for researchers to identify and minimize the effect of important
limitations in the study design. This in turn will enable them to draw
stronger conclusions.

Education provided comments on this report, which appear in app. II. In the
Department's comments, the Deputy Secretary of Education stated that we
accurately described the process used to conduct both the Prospects and
LESCP studies. Concerning our finding that the purpose of LESCP was not
clearly understood by those involved, Education said that the study was
designed to examine both the effects of standards-driven reforms and
instructional practices in high-poverty schools. Education also stated that
these two purposes are not inconsistent with one another. We agree that a
study can have more than one purpose. However, Education's position on the
purposes of LESCP was not clearly articulated by the research questions, nor
was this view consistently reflected by Department staff, panel members, and
study contractors. Furthermore, as our analysis shows, the major limitations
of the study's design seriously restrict the conclusions that can be drawn
about standards-based reform on the basis of LESCP data. In its comments,
Education also stated that GAO recognized that LESCP's design limitations
reflected "the need to gather rich, in-depth data within the constraint of
limited Title I funding." We did not analyze the extent to which LESCP's
funding level influenced the study design; rather, we merely reported that
Education identified the funding level as a factor that affected the study.
Education also submitted technical comments, which we incorporated where
appropriate.

We are sending copies of this report to the Honorable Richard Riley,
Secretary of Education, appropriate congressional committees, and other
interested parties. We will also make copies available to others upon
request. If you or your staffs have any questions about this report, please
call me at (202) 512-7215. Major contributors to this report included Jeff
Appel, Sarah L. Glavin, and Sara L. Schibanoff.

Marnie S. Shaul
Associate Director,
Education, Workforce and
Income Security Issues

Comparison of Prospects and LESCP Studies

                               Prospects                    LESCP
 Time frames
 Data gathered        1991-94                     1997-99
 Interim report
 released             July 1993                   July 1999
 Final report
 released             April 1997                  Not yet released
 Study cost
 Estimated study cost $28.8 million               $9 million
 Objectives, findings, methodology, and key limitations
                                                  As written in interim
                                                  report: To analyze
                                                  variation and changes
                      As written in final report: over time in students'
                      To examine the effects of   performance and teachers'
                      Title I on student          instructional practices
                      achievement and other       in a set of high poverty
 Study objectives     school-related educational  elementary schools.
                      outcomes.
                                                  As described in
                      As described in interviews: interviews: To analyze
                      Same as in report.          the effectiveness of
                                                  standards-based reform
                                                  and to evaluate the
                                                  effectiveness of Title I.
                      Students who received Title
                      I services began below the
                      achievement level of their
                      peers. This initial gap
                      remained essentially
                      unchanged as students moved
                      into higher grades.         Some instructional
                                                  practices, such as
                      Characteristics of students frequent repetition, were
                      and their families account  associated with improved
                      for the largest part of     student performance in
                      overall variation in        reading and mathematics.
                      student achievement.
                                                  There was little change
                      The poverty level of the    in these instructional
                      school (over and above the  practices between 1997
                      economic status of an       and 1998.
 Major findings       individual student) was
                      negatively related to       Teacher participation in
                      standardized achievement    professional development
                      scores.                     was only modestly
                                                  associated with
                      Academic standards varied   differences in classroom
                      between high- and           practices across
                      low-poverty schools.        teachers. Furthermore,
                                                  individual teachers did
                      An emphasis on              not appear to change
                      comprehension and the       their practices as a
                      development of writing      result of professional
                      skills was generally        development activities.
                      positively related to
                      student achievement in
                      reading. An emphasis on
                      remedial instruction was
                      positively related to math
                      test scores in grades 1-3.
                                                  Teacher surveys,
                                                  standardized tests, state
                      Teacher surveys,            and local assessments,
                      standardized tests, student interviews with
 Data collection      and parent surveys,         principals and district
 strategies           principal surveys, and a    Title I coordinators,
                      district coordinator        classroom observations,
                      survey.                     focus groups of teachers
                                                  and parents, review of
                                                  policies and programs of
                                                  family involvement.
                                                  The small,
                                                  nonrepresentative sample
                                                  limited the study's
                                                  ability to draw general
                                                  conclusions.

                                                  Because district
                                                  officials chose the
                                                  schools that participated
                                                  in the study, the sample
                                                  was likely to include a
                                                  greater percentage of
                                                  higher achieving schools
                                                  compared with the
                                                  population as a whole.

                                                  The small sample size may
                                                  make it difficult to
                                                  compare study results
                                                  across special
                                                  populations of students
                                                  attending Title I
                      The quality of              schools, such as students
                      self-reported data may be   with disabilities or
                      questionable.               students with limited
                                                  English proficiency.
                      Information on students in
                      grades above the sixth      Students that moved were
                      grade was limited.          not followed.

                      Students in comparison      Information was provided
                      groups may have different   only for students in
                      educational needs than      grades 3 to 5.
                      Title I students, which
                      limited the study's ability The quality of
                      to draw conclusions.        self-reported data may be
 Key limitations                                  questionable.
                      The study lacked clear
                      criteria to measure the     The difficulty defining
                      effectiveness of the Title  and measuring
                      I program.                  standards-based reform
                                                  makes it more difficult
                      Major changes occurred in   to interpret the study's
                      the Title I program before  results.
                      the final Prospects report
                      was issued.                 The status of reform
                                                  efforts is changing over
                      No cost information was     time, making it difficult
                      given.                      to link standards-based
                                                  reform to student
                                                  achievement.
                                                  Title I schools could not
                                                  be compared with
                                                  non-Title I schools.

                                                  Under the schoolwide
                                                  approach, it is more
                                                  difficult to distinguish
                                                  the effect of Title I
                                                  services from the effect
                                                  of the school's overall
                                                  instructional program,
                                                  which creates
                                                  difficulties in assessing
                                                  the impact of Title I.

                                                  Results for schoolwide
                                                  and targeted assistance
                                                  schools could not be
                                                  compared.

                                                  No cost information was
                                                  given.
 Measures used

                      National standardized       National standardized
 Student achievement  test--Comprehensive Test of test--Stanford 9--and
                      Basic Skills (CTBS).        state and local
                                                  assessments.
                      Student attendance and
 Other outcomes       teacher ratings of student  None.
                      performance.
                                                  Frequency, duration, and
                      Frequency of use of various emphasis on various
                      instructional strategies    instructional strategies
                      (such as using computers    (such as reading aloud,
 Instructional        and leading discussion      working at a computer, or
 practices            groups) and curricular      using calculators) and
                      areas (such as whole number curricular areas (such as
                      operations, fractions, and  vocabulary, phonics, or
                      vocabulary).                multidigit
                                                  multiplication).
                      Student information on
                      grade retention; preschool  Teachers' self-reported
                      attendance; teacher ratings familiarity with policy
                      of student ability and      instruments (such as
                                                  content standards and
 Other major          motivation; family          curricular frameworks);
 variables            characteristics; teacher    teachers' reports of the
                      background, certification,
                      experience; estimates of    amount, type, content,
                      school climate; and school  and usefulness of
                      and district                professional development
                      characteristics.            activities.
 Sample
 characteristics      .
                      Selection was random to
                      ensure a nationally         The selection was
 Sample selection     representative sample;      purposive and stratified
 method               however, the 12 largest     by the implementation of
                      districts were selected     standards-based reform in
                      with certainty.             the states and districts.
 Number of states     Not available               7
 Number of districts  120                         18
 Number of schoolsa   372                         71
 Number of studentsb  28,369                      2,323
 Schools in sample
 that received Title  66                          100
 I funding (%)
 Schoolwide schools
 in the sample (%)    Not available               81
 Targeted assistance
 schools in the       Not available               19
 sample (%)
 Average poverty rate
 for schools in the   50                          74
 sample (%)
 School sample with
 poverty rates over   25                          35
 75% (%)
 Urbanicity of school
 sample (%)           49                          Not available

aFor the Prospects study, this number includes only the original
longitudinal cohort.

bThe number of students is the largest number in the 1-year longitudinal
cohort.

Comments From the Department of Education

(102004)

Figure 1: Primary Affiliation of Independent Review Panel Members, 1993 and
1999 12
  

1. Title I has been scheduled for reauthorization every 5 years, but the
1999 reauthorization has not yet been completed.

2. Abt Associates, Inc., Prospects: Final Report on Student Outcomes,
prepared under contract to the Planning and Evaluation Service, U.S.
Department of Education (Washington, D.C.: U.S. Department of Education,
Apr. 1997); Abt Associates, Inc., Prospects: The Congressionally Mandated
Study of Educational Growth and Opportunity, Interim Report, prepared under
contract to the Planning and Evaluation Service, U.S. Department of
Education (Washington, D.C.: U.S. Department of Education, July 1993);
Policy Studies Associates, Inc. and Westat, Inc., The Longitudinal
Evaluation of School Change and Performance in Title I Schools, Interim
Report, prepared under contract to the Planning and Evaluation Service, U.S.
Department of Education (Washington, D.C.: U.S. Department of Education,
July 1999).

3. Before the 1994 reauthorization, the Title I program was called "Chapter
1." Throughout this report, we refer to the program as Title I.

4. States had until the 1997-98 school year to develop content and
performance standards. Education, as authorized by statute, extended the
deadline for performance standards for many states to coincide with the
deadline for assessments, which were to be finalized by the 2000-01 school
year.

5. Prospects' research design included a seventh grade cohort that was
followed through the ninth grade, but limited analysis was done on this
group. Title I participation by these students was also lower than for
elementary school students.

6. To construct this comparison group, Prospects' researchers selected those
first grade students who attend schools with over 50 percent of students
from low income families and who also scored in the bottom quartile of the
national standardized test. Within this selected group, the researchers
compared Title I students with those students who did not receive Title I
services. The Prospects study did not construct similar comparison groups
for the third and seventh grade cohorts.

7. For more information about this variation, see Title I Program: Stronger
Accountability Needed for Performance of Disadvantaged Students
(GAO/HEHS-00-89, June 1, 2000).
*** End of document. ***