Title I: Education Needs to Monitor States' Scoring of
Assessments (01-APR-02, GAO-02-393).
Concerned that Title I of the Elementary and Secondary Education
Act (ESEA) had not significantly improving the educational
achievements of children at risk, Congress mandated major changes
in 1994. States were required to adopt or develop challenging
curriculum content and performance standards, assessments aligned
with content standards, and accountability systems to measure
progress in raising student achievement. In return, states were
given greater flexibility in the use of Title I and other federal
funds. The No Child Left Behind Act of 2001 augments the
assessment and accountability requirements that states must
implement and increases the stakes for schools that fail to make
adequate progress. The 1994 legislation required states to comply
with the requirements by January 2001 but allowed the Department
of Education to extend that deadline. Education has granted
waivers to 30 states to give them more time to meet all
requirements. If states fail to meet the extended timeliness,
they are subject to the withholding of some Title I
administrative funds. Title I directors indicated that a state's
ability to meet the 1994 requirements improved when both state
leaders and state agency staff made compliance a priority and
coordinated with one another. Most directors said that inadequate
funding hindered compliance. Many of the states reported taking
action to ensure that Title I assessments were scored accurately,
that any exemptions for students with limited English proficiency
were justified, and students with disabilities were receiving
appropriate testing accommodations. As of March 2002, 17 states
had complied with the 1994 assessment requirements; 35 states had
not.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-393
ACCNO: A02849
TITLE: Title I: Education Needs to Monitor States' Scoring of
Assessments
DATE: 04/01/2002
SUBJECT: Academic achievement
Accountability
Aid for education
Disadvantaged persons
Education program evaluation
Internal controls
Performance measures
Schools
Standards evaluation
State-administered programs
Students
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-393
United States General Accounting Office
GAO Report to the Secretary of Education
April 2002
TITLE I
Education Needs to Monitor States' Scoring of Assessments
GAO-02-393
Contents
Letter
Results in Brief
Background
Most States Are Not in Compliance with the 1994 Title I
Accountability and Assessment Requirements States Cited Support and
Coordination as Furthering Compliance Most States are Taking Some Action To
Ensure Accurate Scoring,
Justification for Exemptions, and Appropriate Accommodations,
but Actions are Limited Conclusions Recommendation Agency Comments 1
2 4
9 13
15 18 19 19
Appendix I Scope and Methodology
Appendix II Comments from the Department of Education
Tables
Table 1: Accountability and Assessment Requirements under the 1994 and 2001
Reauthorizations of Title I 6 Table 2: Status of States' Compliance with
1994 Title I Assessment Requirements as of March 2002 10 Table 3: Timeline
Waiver Status for 30 States in Meeting the 1994 Title I Assessment
Requirements as of March 2002 11
Abbreviations
ESEA Elementary and Secondary Education Act
GAO General Accounting Office
LEA local education agency
United States General Accounting Office Washington, DC 20548
April 1, 2002
The Honorable Roderick R. Paige The Secretary of Education
Dear Mr. Secretary:
Title I of the Elementary and Secondary Education Act (ESEA), the largest
source of federal funding for education, will provide states with $10.3
billion in fiscal year 2002 to improve the educational achievement of
children at risk. Title I serves about 12.5 million children in all 50
states, the District of Columbia, and Puerto Rico. In response to concerns
that Title I funding was not significantly improving the educational
achievement of at-risk students, in 1994 Congress mandated major changes to
Title I, including changes to how states measure student performance. As
part of the 1994 ESEA reauthorization, states were required to adopt or
develop challenging curriculum content and performance standards,
assessments aligned with content standards, and accountability systems to
assess schools' and districts' progress in raising student achievement. In
exchange for meeting these requirements, states were given increased
flexibility in the use of Title I and other federal funds. New legislation
that reauthorizes ESEA - The No Child Left Behind Act of 2001 - has since
been passed. The new legislation does not reduce or eliminate any of the
1994 requirements. Instead, it augments the assessment and accountability
requirements that states must implement and increases the stakes for schools
that fail to make adequate progress. This report provides a snapshot of how
close states had come to meeting the 1994 requirements when the 2001
requirements were signed into law.
In light of the increasingly important role played by the required
assessment and accountability measures, we have collaborated with other
audit organizations to review aspects of how states are implementing these
measures and ensuring that data used to assess schools' progress in raising
student achievement are complete and accurate. This report is one of several
on this topic to be issued separately by the various collaborating audit
organizations, which include the U.S. Department of Education's Office of
the Inspector General (OIG), the Texas State Auditor's Office, the
Pennsylvania State Auditor's Office, and the City of Philadelphia
Controller's Office. In this report, we provide GAO's findings regarding (1)
the status of states' compliance with key 1994 Title I requirements, (2)
factors that have helped or hindered states in meeting the requirements,
and (3) the actions states are taking to ensure that Title I assessments are
scored accurately, exemptions for students with limited English proficiency
are justified and students with disabilities are appropriately accommodated
during testing according to Title I regulations.
To assess states' progress in implementing important assessment and
accountability requirements introduced by Title I, we (1) interviewed
Education officials and reviewed Education memoranda and reports regarding
states' compliance1 with the 1994 requirements, (2) surveyed State Education
Agency (SEA) Title I directors regarding the challenges faced by states with
and without approved assessment systems, and (3) conducted telephone
interviews with Title I and other state officials in states that are in
compliance with Title I to develop detailed information on strategies for
overcoming key barriers. We also conducted similar interviews in some states
still working to attain compliance with the 1994 requirements to determine
factors that have hindered their progress. We sent surveys to the 50 states,
the District of Columbia, and Puerto Rico and obtained 50 responses (96
percent). We conducted our work in accordance with generally accepted
government auditing standards between June 2001 and March 2002. (See app. I
for specifics on our scope and methodology.)
As of March 2002, 17 states were in compliance with the 1994 Title I
assessment requirements; however, 35 states and instrumentalities were not.2
The 1994 legislation required states to be in full compliance with the
requirements by January 2001 but allowed the Department of Education to
extend that deadline. For states that have not met the requirements,
Education distinguishes between states that are near compliance and those
that still have a significant amount of work remaining, granting timeline
waivers to the former and compliance agreements to the latter. Education has
granted timeline waivers to 30 states to give them more time to meet all
requirements. According to Education, by January 31,
Results in Brief
1 In this report, we refer to a state as compliant when Education has fully
approved its assessment system for meeting the 1994 final assessment
requirements. We do not intend to imply that these states are necessarily in
compliance with all requirements of the 1994 law.
2 In this report, when we refer to states we will be including the
instrumentalities of the District of Columbia and Puerto Rico.
2004, all of these states will be in compliance and meet all 1994 Title I
requirements, none of which are reduced or eliminated by the 2001
legislation. If states fail to meet extended timelines for implementing the
1994 requirements, they are subject to the withholding of some Title I
administrative funds. Education has asked the remaining five states to enter
into compliance agreements that will establish the final date by which they
must meet all requirements. Education has held public hearings in each of
these states. Under the law, without a compliance agreement, these states
may lose funding. Noncompliant states most frequently have not met the
specific requirements to assess all students and to report the data by
subgroups of students, but some also have more intractable problems, such as
assessments that are not aligned with state standards. Because the majority
of states have not met the requirements of the 1994 law, many states may not
be well positioned to meet the deadlines for implementing the additional
requirements in the 2001 legislation.
Title I directors indicated that a state's ability to meet the 1994
requirements improved when the necessary players-both state leaders and
state agency staff-made compliance a priority and coordinated with one
another to achieve it; most directors indicated that state compliance was
hindered by inadequate funding. Each compliant state that we interviewed
said that the state's governor, legislature, department of education
leaders, or business leaders first prompted compliance initiatives, such as
establishing a blue ribbon committee to address the issue. According to
Title I directors, state leader initiatives that endorsed compliance as a
high-priority, the backing of requirements by state and local staff involved
in implementation, and technical expertise at the state level were among the
types of support that contributed most to states' compliance. Coordination
between staff in different offices and levels of government was another
factor identified as important to compliance. Inadequate funding was the
most often cited factor that hindered state compliance with requirements,
according to our survey of Title I directors. In interviews, the directors
said that their investment of time and money in systems of assessment that
predated and conflicted with the requirements of the 1994 legislation was an
obstacle to compliance.
Most of the states reported taking some action to ensure that Title I
assessments were scored accurately, that any exemptions for students with
limited English proficiency were justified, and students with disabilities
were receiving appropriate testing accommodations. Almost all states hire a
contractor to score the Title I assessments; however, 16 of
these states reported that they did not monitor the scoring done by the
contractor. Most of those who did monitor the scoring reported they did so
by selecting a sample of answer sheets to compare with the contractor's
results to ensure their accuracy. Others compared school and district test
results with the results from previous years to identify large discrepancies
that might suggest a problem. Several states have experienced errors in
scoring done by contractors. For some states, these errors resulted in the
incorrect identification of schools in need of improvement and students in
need of additional services. Thirty-three states also reported taking some
actions to ensure that any exemptions for students with limited English
proficiency were justified and 41 reported actions to ensure students with
disabilities received appropriate accommodations during testing. For
example, states reported they had developed standards for districts to
follow in accommodating these students so that assessments can yield valid
measures of their performance. However, states reported few actions that
would ensure that these guidelines were being followed. Many states are
still developing procedures to ensure that any exemptions are justified and
accommodations are appropriate. Education performs compliance reviews of
grantee programs and is in the process of redesigning this review process.
However, the redesigned reviews do not specifically include the monitoring
of states' actions with regard to contractors' scoring of assessments.
To reduce the potential for undetected errors in test scoring that could
have material effects on educational decisions or damage confidence in the
test results, we are recommending that Education specifically include
monitoring of state actions regarding contractors and scoring provisions in
its state compliance reviews. Education agreed to our recommendation in its
official agency comments. A copy of the comments is printed in Appendix II.
Background The original Title I legislation was passed in 1965, but the 1994
reauthorization of ESEA mandated fundamental changes to the Title I program.
One of the key changes involved the development of state systems of
standards and assessments to ensure that students served by Title I were
held to the same standards of achievement as all other children. Prior to
1994, some states had already implemented assessment systems, but these
tended to be norm-referenced-students' performance
was judged in relation to the performance of other students. The 1994
legislation required assessments that were criterion-based-students'
performance was to be judged against an objective standard.3 Every state
applying for Title I funds since 1994 agreed to implement the changes
described in the 1994 law and to bring its assessment systems into
compliance. States are also required to develop a definition of adequate
yearly progress based on the assessments to hold schools accountable for
educational progress. To help states that could not meet the proposed 2001
timeline, Education had authority to grant timeline waivers and compliance
agreements to states under certain conditions. In its 2001 ESEA
reauthorization, Congress increased testing requirements for states as well
as the consequences for not improving test scores in schools and did not
eliminate any of the requirements of the 1994 legislation. As shown in table
1, the 1994 and 2001 legislative requirements for assessment and
accountability concern developing standards for content and performance;
measuring improvement; implementing and administering assessments, including
assessing students with limited English proficiency; reporting assessment
data; and applying consequences for not meeting performance goals.
3 A norm-referenced test is an objective test that is standardized on a
group of individuals whose performance is evaluated in relation to the
performance of others. Criterion-referenced tests are assessments that
measure the mastery of specific skills or subject content and focus on the
performance of an individual as measured against a standard or criterion
rather than the performance of others taking the test.
Table 1: Accountability and Assessment Requirements under the 1994 and 2001
Reauthorizations of Title I
1994 requirements 2001 requirements
Developing standards for content and performance
Develop challenging standards for what students should know in math and
reading or language arts. In addition, for each of these standards, states
should develop performance standards representing three levels: partially
proficient, proficient, and advanced. The standards must be the same for all
children. If the state does not have standards for all children, it must
develop standards for Title I children that incorporate the same skills,
knowledge, and performance expected of other children.
In addition, develop standards for science content by 2005-06. The same
standards must be used for all children.
Implementing and administering assessments
Develop and implement assessments aligned with the content and performance
standards in at least math and reading or language arts.
Add assessments aligned with the content and performance standards in
science by the 2007-08 school year. These science assessments must be
administered at some time in each of the following grade ranges: from grades
3 through 5, 6 through 9, and 10 through 12.
Use the same assessment system to measure Title I students as the state uses
to measure the performance of all other students. In the absence of a state
system, a system that meets Title I requirements must be developed for use
in all Title I schools.
Use the same assessment system to measure Title I students as the state uses
to measure the performance of all other students. If the state provides
evidence to the secretary that it lacks authority to adopt a statewide
system, it may meet the Title I requirement by adopting an assessment system
on a statewide basis and limiting its applicability to Title I students or
by ensuring that the Title I local educational agency (LEA) adopts standards
and aligned assessments.
Include in the assessment system multiple measures of student performance,
including measures that assess higher-order thinking skills and
understanding.
Unchanged.
Administer assessments for math and reading in each of the following grade
spans: from grades 3 through 5, 6 through 9, and 10 through 12.
Administer reading and math tests annually in grades 3 through 8, starting
in the 2005-06 school year (in addition to the assessments previously
required sometime within grades 10 through 12).
States do not have to administer math and reading or language arts tests
annually in grades 3 through 8 if Congress does not provide specified
amounts of funds to do so, but states have to continue to work on the
development of the standards and assessments for those grades.
Have students in grades 4 and 8 take the National Assessment for Educational
Performance (NAEP) exams in reading and math every other year beginning in
2002-03, as long as the federal government pays for it.
Implement controls to ensure the quality of the data collected from the
assessments.
Unchanged.
Including students with limited English proficiency and with disabilities in
assessments
Assess students with disabilities and limited English proficiency By
2002-03, annually assess the language proficiency of students
according to standards for all other students. with limited English
proficiency. Students who have attended a
Provide reasonable adaptations and accommodations for U.S. school for 3
consecutive years must be tested in English
students with disabilities or limited English proficiency, to include unless
an individual assessment by the district shows testing in a
testing in the language and form most likely to yield accurate and native
language will be more reliable.
reliable information on what they know and can do.
1994 requirements 2001 requirements Reporting data
Report assessment results according to the following: by state, LEA, school,
gender, major racial and ethnic groups, English proficiency, migrant status,
disability, and economic disadvantage.
Unchanged.
LEAs must produce for each Title I school a performance profile with
disaggregated results and must publicize and disseminate these to teachers,
parents, students, and the community. LEAs must also provide individual
student reports, including test scores and other information on the
attainment of student performance standards.
Provide annual information on the test performance of individual students
and other indicators included in the state accountability system by 2002-03.
Make this annual information available to parents and the public and include
data on teacher qualifications. Compare high- and low-poverty schools with
respect to the percentage of classes taught by teachers who are "highly
qualified," as defined in the law, and conduct similar analyses for
subgroups listed in previous law.
Measuring improvement
Use performance standards to establish a benchmark for improvement referred
to as "adequate yearly progress." All LEAs and schools must meet the state's
adequate yearly progress standard, for example, having 90 percent of their
students performing at the proficient level in math. LEAs and schools must
show continuous progress toward meeting the adequate yearly progress
standard. The state defines the level of progress a school or LEA must show.
Schools that do not make the required advancement toward the adequate yearly
progress standard can face consequences, such as the replacement of the
existing staff.
In addition to showing gains in the academic achievement of the overall
school population, schools and districts must show that the following
subcategories of students have made gains in their academic achievement:
pupils who are economically disadvantaged, have limited English proficiency,
are disabled, or belong to a major racial and ethnic group. To demonstrate
gains among these subcategories of students, school districts measure their
progress against the state's definition of adequate yearly progress.
States have 12 years for all students to perform at the proficient level.
Consequences for not meeting the adequate yearly progress standard
LEAs are required to identify for improvement any schools that fail New
requirements are more specific as to what actions an LEA
to make adequate yearly progress for 2 consecutive years and to must take to
improve failing schools. Actions are defined for each
provide technical assistance to help failing schools develop and year the
school continues to fail leading up to the 5th year of
implement required improvement plans. After a school has failed failure when
a school must be restructured by changing to a
to meet the adequate yearly progress standard for 3 consecutive charter
school, replacing school staff, or state takeover of the
years, LEAs must take corrective action to improve the school. school
administration. The new law also provides that LEAs offer options to
children in failing schools. Depending on the number of years a school has
been designated for improvement, these options may include going to another
public school with transportation paid by the LEA or using Title I funds to
pay for a private tutor.
Source: P.L. 103-382 and P.L. 107-110.
Almost all states employ contractors to perform services to help them meet
these requirements. Among states that we interviewed, contractors included
private companies, universities, nonprofit organizations, and individual
consultants. These entities were hired to provide services that may include
assessment development, administration, scoring, analysis, and reporting of
results. Some of these entities can provide combinations of services to
states, such as test development and test scoring. States are responsible
for monitoring contractor performance.
Congress allowed states to phase in the 1994 ESEA requirements over time,
giving states until the beginning of the 2000-01 school year to fully
implement them with the possibility of limited extensions. Education is
responsible for determining whether or not a state is in compliance with
these requirements and is authorized under ESEA, to give states more time to
implement the requirements as long as states are making adequate progress
toward this goal. States submit evidence to Education showing that their
system for assessing students and holding schools accountable meets Title I
requirements. Education has contracted with individuals with expertise in
assessments and Title I to review this evidence. The experts provide
Education with a report on the status of each state regarding the degree to
which a state's system for assessing students meets the requirements and
deserves approval. Using this and other information, the Secretary sends
each state a decision letter that summarizes the experts' review and
communicates whether a state is in full compliance, in need of a timeline
waiver, or more seriously, a compliance agreement. Education may withhold
funds if a state does not meet the terms of its compliance agreement. The
1994 legislation was not specific in the amount of administrative funds that
could be withheld from states failing to meet negotiated timelines, but the
2001 legislation states that Education must withhold 25 percent of state
administrative funds until the state meets the 1994 requirements including
the terms of any timeline waivers or compliance agreements.
In June 2000, we issued a report on states' efforts to ensure compliance
with key Title I requirements.4 At that time, we expressed concern about the
number of states that were not positioned to meet the deadlines in the 1994
law. To increase compliance, we made two recommendations. We recommended
that the Department of Education should (1) facilitate among states the
exchange of information and best practices regarding the implementation of
Title I requirements and (2) implement additional measures to improve
research on the effectiveness of different services provided through Title I
to improve student outcomes. Education continues to work on the
implementation of these recommendations. In addition, we said that Congress
should consider requiring that states' definitions of adequate yearly
progress apply to disadvantaged children, as
4 General Accounting Office, "Title I Program: Stronger Accountability
Needed for Performance of Disadvantaged Students" GAO/HEHS-00-89,
(Washington, D.C.: June 1, 2000).
well as to the overall student population. The 2001 legislation does require
that states apply adequate yearly progress requirements and report on the
results by subgroups, including students in poverty, with disabilities, and
with limited English proficiency.
Most States Are Not in Compliance with the 1994 Title I Accountability and
Assessment Requirements
As of March 2002, 17 states were in compliance with the 1994 Title I
assessment requirements; however, 35 were not. (See table 2.) Departmental
approval of timeline waivers to give states more time to reach compliance
has been granted for 30 states. Education has asked five states to enter
into compliance agreements that will establish the final date by which they
must be in compliance before losing Title I funding. Among other
requirements, states that are not in compliance have most frequently not met
the specific requirements to assess all students and break out assessment
data by subcategories of students. The 2001 legislation requires states to
implement additional assessments through 2008, thus substantially augmenting
current assessment requirements. Education has published a notice of
proposed negotiated rulemaking in the Federal Register and has solicited
comments from outside parties in preparation for establishing state
compliance standards for the 2001 legislation.
Table 2: Status of States' Compliance with 1994 Title I Assessment
Requirements as of March 2002
Compliant (17) Noncompliant (35)
Source: Department of
Education.
When Education
determines that a state
is not in compliance
with the 1994 Title I
assessment requirements,
it may grant the state a
timeline waiver for
meeting those
requirements. A waiver
may not exceed 3 years.
Education officials
indicate that the agency
grants waivers to states
that have a history of
success in implementing
significant portions of
their assessment
systems, have a clear
plan with a definite
timeline for complying
with the Title I
requirements, and have the capacity to carry out the plan and thus meet
those requirements. When a state requests a waiver, it must provide
Education with a plan that includes a timeline for addressing deficiencies
in the state's assessment system. Education reviews this information to
decide whether the waiver should be granted and its duration. So far,
Education has granted timeline waivers to 30 states. (see table 3.)
Table 3: Timeline Waiver Status for 30 States in Meeting the 1994 Title I
Assessment Requirements as of March 2002
Illinois Georgia
North Dakota
Kentucky Hawaii
Oklahoma New
Hampshire Iowa South
Carolina New York
Michigan South Dakota Utah Mississippi Tennessee Washington Nebraska
Wisconsin
Note: Each timeline waiver has an exact ending date.
1 year = to be complete December 31, 2002 or sooner.
2 year = to be complete December 12, 2003 or sooner.
3 year = to be complete January 31, 2004 or sooner.
Source: Department of Education.
A compliance agreement is deemed necessary when Education determines that a
state will not complete the implementation of its assessment system in a
timely manner. According to Education officials, a state requiring a
compliance agreement generally does not have a history of successful
implementation, has not met a significant number of Title I requirements,
and does not have a plan in place for meeting those requirements. Education
recommends a compliance agreement so that a state may continue to receive
Title I funds. Before Education may enter into a compliance agreement, a
public hearing must be held in which the state has the burden of persuading
Education that full compliance with the Title I requirements is not feasible
until a future date. The state must be able to attain compliance within 3
years of the signing of the compliance agreement by the state and Education.
The state then negotiates the terms of the agreement with Education.
Education's written findings and the terms of the compliance agreement are
published in the Federal Register. A state that enters into a compliance
agreement to address requirements of the 1994 Title I law and subsequently
fails to meet the requirements of the agreement can be subject to loss of
some state Title I administrative funds. Education is presently working on
five compliance agreements (Alabama, Idaho, Montana, West Virginia, and the
District of Columbia) and has held public hearings for each.
The 2001 reauthorization of ESEA was signed into law on January 8, 2002. The
act provides states not in compliance with the 1994 Title I requirements at
the time of the signing of the 2001 legislation with a 90-day period that
started on January 8, 2002 to negotiate changes in the dates by which they
must be in compliance with the 1994 requirements. After the conclusion of
this 90-day period, the legislation prohibits further extensions for
compliance with the 1994 requirements. States failing to meet these
negotiated timelines will be subject to loss of some of their Title I
administrative funds. According to senior Education officials, this loss
could be significant to states, as many use federal program administrative
funds to pay the salaries of state department of education staff.
A review of documents from Education shows that noncompliant states have
most commonly not met two Title I requirements-assessing all students and
breaking out assessment data by subcategories of students. Title I does not
permit states to exempt any student subgroup from their assessments and
Education's guidance states that individual exemptions may be permitted by
the states in extraordinary circumstances. Nonetheless, many states allow
substantial exemptions for students with disabilities and limited English
proficiency. Several states reported that they have only recently amended
laws that prohibited testing of some students with limited English
proficiency. Title I also requires states, local districts, and schools to
report the performance of students overall and in a variety of
subcategories. These categories are gender, race, ethnicity, English
proficiency status, migrant status, disability status, and economic
disadvantage. Many states disaggregated data for some but not all of these
categories. Documents from Education show that data for the disabled,
migrant, and economically disadvantaged subcategories are the most common
subgroups excluded from state, district, and school reports. In addition,
many states lag in other areas, such as aligning assessments to state
content standards.
To achieve compliance with the 2001 legislation, states will need to add new
standards and increase assessment efforts, as detailed in table 1. In
responding to our survey, 48 states reported that they have developed
content standards in science, but only 16 reported having annual assessments
for math and 18 reported annual assessments for reading in all grades 3
through 8. In addition, states will not have the 2 to 3 year timeline
waivers available to them as they had when they worked to meet the 1994
requirements. New 2001 requirements listed in table 1 have deadlines that
vary according to the requirement, and the Secretary of
Education can give states 1 additional year from those deadlines to meet the
new requirements, but only in case of a "natural disaster or a precipitous
and unforeseen decline in the financial resources of the state." Since the
majority of states have not met the requirements of the 1994 law, it appears
that many states may not be well-situated as they work to meet the schedule
for implementing new requirements that build upon the 1994 requirements.
States Cited Support and Coordination as Furthering Compliance
States successful in meeting key Title I requirements attributed their
success primarily to four factors. These factors were (1) the efforts of
state leaders to make Title I compliance a priority; (2) coordination
between staff of different agencies and levels of government; (3) obtaining
buy-in from local administrators, educators, and parents; and (4) the
availability of state-level expertise. Survey respondents identified
inadequate funding as an obstacle to compliance. The state Title I officials
we interviewed said that their states' commitment of resources to
norm-referenced assessments that conflicted with the 1994 Title I
requirements contributed to this obstacle.
Almost 80 percent of the respondents identified state leaders' efforts as a
factor that facilitated their meeting the 1994 Title I requirements. In
every state that had attained compliance with the Title I requirements, the
officials that we interviewed said that the governor, legislators, or
business leaders made compliance with the Title I requirements a
high-priority. States described the development of high-level committees,
new state legislation, and other measures to raise the visibility and
priority of this issue. For example, one governor spearheaded a plan that
used commissions to develop content standards and assessments aligned with
those standards. Some state officials we interviewed reported that efforts
by state department of education leaders resulted in major organizational
changes in the state education department. For example, according to one
Title I Director, the state changed the organizational structure and
reporting relationships of state offices to organize them by function rather
than by funding streams and to enhance coordination; according to another
Title I Director, state leaders who did not support changes necessary to
achieve compliance with Title I were replaced with staff that did support
the changes.
In responding to our survey, over 80 percent of the Title I officials
identified the ability of staff or agencies to coordinate their efforts with
one another as a factor that helped them meet requirements. In our
interviews, state officials cited the necessity of coordination between
state and local staff working in the areas of assessment, instruction, and
procurement. Two of the states we interviewed specifically noted that when
the assessment office shared a physical location with the Title I office,
coordination was easier and the ability to achieve compliance with Title I
was enhanced.
Title I and other officials we interviewed in those states that had met the
1994 Title I assessment requirements noted that they had made great efforts
to obtain buy-in from other state officials, local administrators,
educators, and the public. They said that efforts to ensure buy-in paved the
way for changes meant to ensure compliance with the assessment and
accountability requirements of the1994 legislation. Several officials we
interviewed reported holding public meetings and focus groups to obtain
input from parents, teachers and local administrators regarding how the
state should implement Title I requirements. They also reported conducting
public relations campaigns to educate the public about the importance of
complying with Title I requirements for standards-based assessment. One
state, for example, conducted 6 years of focus groups and hearings and
conducts a conference annually to allow local education officials to gain
advice from experts regarding any concerns or problems they are having in
implementing Title I requirements.
In responding to our survey, over 80 percent of state Title I directors
identified the availability of state level expertise as a factor that
facilitated their efforts to meet Title I requirements. State officials we
interviewed reported that training for teachers and district personnel was
often needed to apply new content standards in the classroom and to
administer assessments correctly. Two states, for example, used regional
centers to educate local staff on assessments and standards.
Fifty percent of survey respondents identified inadequate funding as an
obstacle in moving toward compliance and noncompliant states cited this
problem more often than compliant ones. In our interviews, Title I and
assessment officials from noncompliant states reported that progress toward
compliance with Title I requirements was stalled because of investments they
had made in assessment systems that predated and conflicted with the
requirements of the 1994 Title I reauthorization. Respondents said that they
had made substantial investments of time and money in systems of assessment
that often relied upon norm-referenced assessments and did not meet the 1994
requirement for criterion-based tests. They noted that it took their states
several years to change from the
Most States are Taking Some Action To Ensure Accurate Scoring, Justification
for Exemptions, and Appropriate Accommodations, but Actions are Limited
old system of assessment to one meeting the requirements specified by the
1994 reauthorizing legislation. According to the officials we interviewed,
building support to start again on another system and obtaining the funding
made it more difficult to make the necessary changes in a timely manner. In
addition, one survey respondent from a very small state noted that due to
the state's size it has a small number of staff and does not have the
technical expertise needed to develop a new system, thus hampering the
state's ability to meet the requirements.
Most states are taking some action to ensure that Title I assessments are
scored accurately, that any exemptions for students with limited English
proficiency are justified, and that students are receiving appropriate
accommodations when these are needed to gather an accurate assessment of
their abilities. Most states hire a contractor to score Title I assessments
and about two-thirds of these states monitored the scoring performed by the
contractor. Some states that hire contractors have found errors in the
scoring the contractors did, and in some cases, these errors have had
serious negative consequences for schools and students. Most states reported
taking some actions to ensure that students with limited English proficiency
and disabilities received appropriate accommodations during testing.
Education is redesigning its current compliance and monitoring program to
better monitor states' implementation of Title I.
Problems Found in
Contractor Scoring
According to our survey results, most states (44) hire a contractor for test
scoring, but 16 of these states identified no monitoring mechanism to ensure
the accuracy of their contractor's scoring and reporting. Among those states
that did report one or more monitoring mechanisms, 15 reported that they
monitored the contractor's scoring by comparing a sample of original student
test results to the contractor's results. A few states also reported, in
interviews with us, that they compared their most recent test scores with
those from previous years and looked for significant variations that
suggested potential errors in scoring. However, in our interviews, some
assessment officials indicated that they use this type of monitoring rather
informally.
The problems identified in assessment scoring suggest that these approaches
do not always provide adequate assurance of complete and accurate results.
Indeed, several of the states that use contractors to score tests reported
that they have had problems with errors in scoring whether or not they had
monitoring measures in place. In some cases, contractors
marked correct answers as incorrect and in other cases the contractors
calculated the scores incorrectly. The errors were discovered by a number of
individuals, including local district officials, parents, and state agency
staff. These scoring errors had impacts on students, families, and school
and district resources. Based on erroneous scores calculated by a
contractor, one state sent thousands of children to summer school in the
mistaken belief that their performance was poor enough to meet the criterion
for summer intervention. In addition to disrupting families' summer plans
and potentially preventing student promotions, this may have drawn resources
away from other necessary activities. In another case, based on a
contractor's erroneous scoring, a state incorrectly identified several
schools as "in need of improvement," a designation that carries with it both
bad publicity and extra expense, for example, districts may have to fund the
needed improvements.
A few state officials that we interviewed told us that they have begun
instituting processes to check the accuracy of scoring. For example, three
states said that they had hired individuals who were experts in test scoring
or they hired other third parties to conduct independent audits. States that
were in compliance with 1994 Title I assessment requirements generally had
more complete monitoring systems, including measures such as technical
advisory committees to review results, conduct site visits, and follow a
sample of tests through the scoring and reporting process. In contrast,
several states indicated they are still relying on contractor
self-monitoring to ensure accurate scoring.
Although Education is obligated under the Federal Managers' Financial
Integrity Act of 19825 and the Single Audit Act6 to ensure that states that
receive federal funds comply with statutory and regulatory requirements to
monitor contractors, it currently takes limited action regarding states'
monitoring of assessment contractors. Education's inspector general has
reported deficiencies in an important vehicle for such oversight -
-Education's compliance reviews of state programs.7 The compliance reviews
are conducted on a 4-year cycle and include an on-site visit that
5 U.S. General Accounting Office, "Standards for Internal Control in the
Federal Government" GAO/AIMD-00-21.3.1, (Washington, D.C.: Nov. 1999).
6 P.L. 98-502.
7 See "Review of the Office of Elementary and Secondary Education's
Monitoring of Formula Grants: Final Audit Report" (ED-OIG/A04-A0013),
November 2001.
lasts 1 week. Specifically, the OIG cited insufficient time to conduct the
reviews, lack of knowledge among Education staff about areas they were
reviewing, and a lack of consistency in how the reviews were conducted.
Senior Education officials told us the department is redesigning the current
compliance and monitoring program used for its on-site visits to better
focus on outcomes and accountability in Title I and that it is addressing
the OIG's recommendations. However, a senior Education official who is
working on the redesign of the compliance reviews told us that the current
draft plans did not include specific checks on state monitoring of
assessment scoring. Confidence in the accuracy of test scoring is critical
to acceptance of the test results' use in assessing school performance.
Some Actions Taken To Ensure Exemptions Justified and Accommodations
Appropriate
According to our surveys and interviews, 33 states have taken at least
minimal actions to ensure any exemptions for students with limited English
proficiency are justified and 41 states take actions to ensure
accommodations for students with disabilities are appropriate.8 Most states
reported that they had developed standards for districts to follow in
accommodating these students so that assessments could yield accurate
measures of their performance. However, states reported few actions that
would ensure that these guidelines were being followed. For example, 17
states reported that they compare the number of students with limited
English proficiency tested within a given year against the number for the
previous year. They used this comparison as their means of verifying that
the numbers of students receiving exemptions were reasonable. As the pool of
students in a particular school can change substantially from year to year,
this comparison has obvious limitations. Moreover, students' status, for
example with respect to English proficiency, can change from year to year.
Similarly, 37 states reported using an annual comparison of the number of
students with disabilities being tested as a check for appropriate
accommodations. However, it is not evident how such comparisons would allow
states to ascertain the appropriateness of the accommodations.
Survey results and interviews did indicate that more states are taking
actions to monitor accommodations for students with disabilities than for
8 Accommodations for testing modify the circumstances for the student while
taking a test. For example, students with learning disabilities may be
allowed more time to take a test.
students with limited English proficiency. For example, while 25 states
reported that they had standards for accommodating students with limited
English proficiency, 36 had standards for accommodating students with
disabilities. The state officials that we interviewed told us that this was
because districts built upon steps they had taken under the Individuals with
Disabilities Education Act (IDEA)9 to document the accommodations needed by
students with disabilities. In general, states said that the districts have
more experience and technical expertise for assessing and supporting
students with disabilities because of working under IDEA for many years. In
contrast, some states lacked consistent standards for identifying students
with limited English proficiency and more states were still working to
develop alternate assessments or accommodations for these students.
Augmenting the 1994 requirements, the new 2001 legislation requires that
states annually assess the language proficiency of students with limited
English proficiency by the 2002-03 school year. States do conduct cyclical
monitoring of the implementation of all their programs that might be used to
assess the appropriateness of district policy and practice with regard to
testing accommodations. However, in a recent review,10 we found that states
varied dramatically in the frequency of their on-site visits. The average
time between visits to districts ranged from 2 years or less (6 states) to
more than 7 years (17 states).
Conclusions This snapshot of the states' status with respect to the 1994
Title I requirements suggests that many states may not be well-positioned to
meet the requirements added in 2001. Only 17 states were in compliance with
the assessment requirements of the 1994 law in March of 2002; therefore, the
majority of states will still be working on meeting the 1994 requirements as
they begin work toward meeting the new requirements.
In addition, despite the enhanced emphasis on assessment results, states
still appear to be struggling with ensuring that assessment data are
complete and correct. The 1994 and 2001 ESEA reauthorizations raised student
assessments to a new level of importance. The assessments are
9 IDEA was enacted in 1970 and, among other things, requires schools to
provide assessments for students with disabilities and provide whatever
accommodations they may need, including specific accommodation for testing,
such as more time to take tests, to ensure that the tests provide valid
reflections of their abilities.
10 GAO/HEHS-00-89, June 1, 2000.
intended to help ensure that all students, including those who have
disabilities and those who have limited English proficiency are meeting
challenging standards. In addition, assessment results are a key part of the
mechanism for holding both schools and states accountable for improving
educational performance. Thus, ensuring the completeness and accuracy of
assessment data is central to measuring students' progress and ensuring
accountability. Without adequate oversight of assessment scoring, efforts to
identify and improve low-performing schools could be hindered by lack of
confidence in assessment results or uncertainty regarding whether particular
schools have been appropriately identified for improvement. Education's
current monitoring does not include specific oversight of how states ensure
the quality of scoring contractors' work, but Education's revision of its
monitoring process provides the agency with an opportunity to help states
ensure that scoring done by contractors is accurate.
To enhance confidence in state assessment results, we recommend that when
the Department of Education monitors state compliance with federal programs,
it include checks for contractor monitoring related to Title I, Part A.
Specifically, Education should include in its new compliance reviews a check
on the controls states have in place to ensure proper test scoring and the
effective implementation of these controls by states.
Recommendation
Agency Comments We provided Education with a draft of this report for
review. The Department's official comments are printed in appendix II. In
its comments, Education agreed with our recommendation. Education also
provided us with technical comments that we incorporated in the report as
appropriate.
We are sending copies of this report to appropriate congressional committees
and other interested parties. If you have any questions about
this report, please contact me on (202) 512-7215 or Betty Ward-Zukerman on
(202) 512-2732. Key contributors to this report were Mary Roy, G. Paul
Chapman II, Laura Pan Luo, Corinna Nicolaou, and Patrick DiBattista.
Sincerely yours,
Marnie S. Shaul, Director Education, Workforce, and Income Security Issues
Appendix I: Scope and Methodology
We conducted this review in conjunction with our partners in the Domestic
Working Group. The Domestic Working Group's objective is to allow officials
in the federal, state, and local governmental audit communities to interact
on a personal and informal basis on various topics of mutual concern. The
group consists of 18 (6 federal, 6 state, and 6 local) top officials and is
intended to complement the work of the intergovernmental audit forums and
other professional associations.
For this review, the Texas State Auditor's Office conducted a detailed
assessment of data quality at the state and local levels in Texas, while the
Department of Education's Office of Inspector General did so at the state
and local levels in California and conducted additional work on control
processes at the Department of Education. In Pennsylvania, the Pennsylvania
Department of Auditor General conducted work at the state level and the
Philadelphia Controller's Office pursued the same goal within the city of
Philadelphia.
To complement these efforts, GAO surveyed all states and conducted detailed
interviews with several regarding their experiences in implementing major
provisions of Title I. Specifically, we reviewed three key questions: (1)
the status of states' compliance with key 1994 Title I assessment
requirements; (2) factors that have hindered or helped states move toward
meeting the requirements; and (3) the actions states are taking to ensure
that Title I assessments are scored accurately, exemptions for students with
limited English proficiency are justified, and students with disabilities
are accommodated during testing according to federal regulations.
We obtained information on the first objective from the Department of
Education. We met with Education officials and obtained updated listings of
compliance throughout the audit. In addition, we reviewed state decision
letters, peer reviews of state assessment systems, and reports completed or
commissioned by Education's Planning and Evaluation Service.
To address the second and third questions, we used both a state survey of
Title I directors and detailed interviews with state Title I officials and
other state officials who played a key role in Title I compliance - often
assessment officials and sometimes Special Education, program evaluation,
and information technology officials. We sent the survey to all 50 state
directors and to the District of Columbia and Puerto Rico. We received 50
completed surveys. We followed up with 19 states to clarify and expand on
questions in the interview related to contracting for the
Appendix I: Scope and Methodology
scoring of tests. We interviewed officials from 5 states that had assessment
systems approved by Education and 3 states that were still trying to attain
compliance. We also interviewed two expert reviewers, Education officials
with responsibility for Title I and program review, three officials at
Education's regional assistance centers, and officials at the Council of
Chief of State School Officers. We coordinated our work and findings with
our audit partners, who provided us with information relative to their
states' activities.
Appendix II: Comments from the Department of Education
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its Web
site daily. The list contains links to the full-text document files. To have
GAO e-mail this list to you every afternoon, go to www.gao.gov and select
"Subscribe to daily e-mail alert for newly released products" under the GAO
Reports heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office P.O. Box 37050 Washington, D.C. 20013
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Visit GAO's Document GAO Building
Distribution Center Room 1100, 700 4th Street, NW (corner of 4th and G
Streets, NW) Washington, D.C. 20013
To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm,
Waste, and Abuse in E-mail: [email protected], or
Federal Programs 1-800-424-5454 or (202) 512-7470 (automated answering
system).
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D.C. 20548
*** End of document. ***