Education Research: Education Should Improve Assessments of R&D  
Centers, Regional Labs, and Comprehensive Centers (24-JAN-02,	 
GAO-02-190).							 
								 
Activities conducted by the Research and Development (R&D)	 
Centers, Regional Labs, and Comprehensive Centers reflect their  
legislative mandates and support the Department of Education's	 
research agenda to various degrees. Because statutes define	 
different missions and activities for these programs, the amount 
and focus of the research and other research-based activities	 
they support varies. Education shapes the priorities that guide  
the research conducted by the R&D Centers and targets the	 
technical assistance provided by the Comprehensive Centers	 
through requirements in agreements with these entities. However, 
Education has limited control over the activities of the Regional
Labs because, unlike most federal education programs, neither	 
federal government or state governments have oversight		 
responsibility for their programs. The R&D Centers, Regional	 
Labs, and Comprehensive Centers reported collaborating and	 
coordinating with each other and Education and cited a variety of
factors that facilitated and hindered such activities. They said 
that they were most likely to engage in these activities when	 
they shared a common interest in a specific student population,  
such as English language learners, or in a specific topic, such  
as assessment. Current evaluation practices for assessing the R&D
Centers, Regional Labs, and Comprehensive Centers have provided  
only limited information about the performance of these 	 
organizations and have not been useful for making future funding 
decisions.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-02-190 					        
    ACCNO:   A02601						        
  TITLE:     Education Research: Education Should Improve Assessments 
of R&D Centers, Regional Labs, and Comprehensive Centers	 
     DATE:   01/24/2002 
  SUBJECT:   Educational grants 				 
	     Educational research				 
	     Research and development				 
	     Cooperative agreements				 
	     Performance measures				 
	     Agency missions					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Testimony.                                               **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-02-190
     
Report to Congressional Requesters

United States General Accounting Office

GAO

January 2002 EDUCATION RESEARCH

Education Should Improve Assessments of R& D Centers, Regional Labs, and
Comprehensive Centers

GAO- 02- 190

Page i GAO- 02- 190 Education Research Letter 1

Results in Brief 2 Background 7 Activities Reflect Legislative Mandates and
Support Education?s

Agenda, but Education Has Limited Control over Regional Labs 16
Collaboration and Coordination among the R& D Centers, Regional

Labs, Comprehensive Centers, and Education Is Ongoing 24 Education Needs to
Improve Its Assessments of the R& D Centers,

Regional Labs, and Comprehensive Centers 33 Conclusions 37 Matters for
Congressional Consideration 38 Recommendations for Executive Action 38
Agency Comments and Our Evaluation 39

Appendix I GAO Contacts And Staff Acknowledgments 40 GAO Contacts 40
Acknowledgments 40

Tables

Table 1: Key Features of the R& D Centers, Regional Labs, and Comprehensive
Centers 15 Table 2: Primary Focus and Mandated Activities of the R& D

Centers, Regional Labs, and Comprehensive Centers 18

Figures

Figure 1: R& D Centers Reported Spending Most Core Funds from OERI on
Research 3 Figure 2: Comprehensive Centers Reported Spending Most Core

Funds from OESE on Technical Assistance 4 Figure 3: Regional Labs Reported
Spending Core Funds from OERI

on a Variety of Activities 5 Figure 4: Education Offices with Responsibility
for Education

Research, Research- Based Activities, and Technical Assistance 8 Figure 5:
Locations of the R& D Centers 10 Figure 6: Regions of the Regional Labs 12
Figure 7: Regions of the Comprehensive Centers 14 Contents

Page ii GAO- 02- 190 Education Research

Figure 8: Organizational Structure of an R& D Center and Its Affiliations
with Other R& D Centers 28 Figure 9: Regional Lab Parent Organization, In-
House Programs,

and Services Provided to Partners 30

Abbreviations

ESEA Elementary and Secondary Education Act OBEMLA Office of Bilingual
Education and Minority Languages

Affairs OERI Office of Education Research and Improvement OESE Office of
Elementary and Secondary Education R& D Research and Development

Page 1 GAO- 02- 190 Education Research

January 24, 2002 The Honorable John Boehner Chairman, Committee on Education
and the Workforce House of Representatives

The Honorable Mike Castle Chairman, Subcommittee on Education Reform
Committee on Education and the Workforce House of Representatives

The Honorable Pete Hoekstra Chairman, Subcommittee on Select Education
Committee on Education and the Workforce House of Representatives

The Congress and the administration have emphasized the importance of a
scientifically sound research base to improve teaching and learning in the
nation?s schools. To develop this research base and promote the use of
research- based practices, the Department of Education (Education) is
charged with sponsoring research and developing and disseminating research-
based activities, such as technical assistance. Education administers over a
thousand grants, cooperative agreements, 1 and contracts that fund
educational research, development of materials, new methods of instruction
and practices in teaching, dissemination of research results, and technical
assistance activities. Some of these grants and contracts are administered
through the Office of Educational Research and Improvement (OERI), but
others are administered elsewhere in Education.

The number of programs these grants, cooperative agreements, and contracts
support, as well as changes in the laws authorizing them, have raised many
questions about the activities the programs perform, the extent to which the
programs collaborate and coordinate with each other and Education, and the
usefulness of the evaluations that have assessed their performance.

1 A cooperative agreement is a type of grant used by agencies when
substantial involvement and interaction is expected of both the agency and
the recipient in carrying out the activities in the agreement.

United States General Accounting Office Washington, DC 20548

Page 2 GAO- 02- 190 Education Research

Consequently, you asked us to conduct a series of studies on Education-
funded research, including this examination of three programs: Research and
Development Centers (R& D Centers), Regional Educational Laboratories
(Regional Labs), and Regional Comprehensive Assistance Centers
(Comprehensive Centers). Specifically, we agreed to answer:

1. To what extent are the activities of the R& D Centers, Regional Labs, and
Comprehensive Centers consistent with their legislative mandates and to what
extent can Education shape and control these activities to support its
research agenda?

2. To what extent do the R& D Centers, Regional Labs, and Comprehensive
Centers collaborate and coordinate with each other and with Education and
what factors foster or hinder collaboration and coordination?

3. To what extent are Education?s practices for assessing the R& D Centers,
Regional Labs, and Comprehensive Centers useful for evaluating their
performance?

To answer these questions, we sent questionnaires to the 12 R& D Centers, 10
Regional Labs, and 15 Comprehensive Centers; reviewed documents from the R&
D Centers, Regional Labs, Comprehensive Centers, and Education; examined
relevant statutes; and interviewed Education officials. We also conducted 14
site visits to R& D Centers, Regional Labs, and Comprehensive Centers to
gather illustrative examples and to more fully understand the processes
involved in their funding and operations. In addition, we examined the
standards Education uses to assess R& D Centers and Regional Labs and the
most recent Education- funded assessments of the R& D Centers, Regional
Labs, and Comprehensive Centers. We performed our work between January and
December 2001 in accordance with generally accepted government auditing
standards. We did not independently verify survey information reported by
the R& D Centers, Regional Labs, and Comprehensive Centers.

Activities conducted by the R& D Centers, Regional Labs, and Comprehensive
Centers reflect their legislative mandates and support Education?s research
agenda to various degrees. Because statutes define different missions and
activities for these programs, the amount and focus of the research and
other research- based activities they support varies. For example, law
mandates that R& D Centers engage in national research, development, and
dissemination activities. In a manner consistent with Results in Brief

Page 3 GAO- 02- 190 Education Research

this mandate, in fiscal year 2000 they reported spending 87 percent of the
$35.5 million in core funding they received from OERI on these activities.
(See fig. 1.)

Figure 1: R& D Centers Reported Spending Most Core Funds from OERI on
Research

In contrast, legislation authorizes the Comprehensive Centers to provide
technical assistance and training to help state and local educational
agencies implement federal programs established by the Elementary and
Secondary Education Act (ESEA). Accordingly, in fiscal year 2000,
Comprehensive Centers reported spending 83 percent of the $28.6 million of
their core funds from Office of Elementary and Secondary Education (OESE) on
technical assistance. (See fig. 2.)

73% 9%

5% 4%

4% 3% 1% Technical Assistance

Evaluation Collaboration Other

Development Dissemination Research

Page 4 GAO- 02- 190 Education Research

Figure 2: Comprehensive Centers Reported Spending Most Core Funds from OESE
on Technical Assistance

The law requires Regional Labs to conduct a wide range of activities that
reflects primarily the needs of the regions they serve. In line with this
requirement, in fiscal year 2000 the Regional Labs reported spending fairly
equal amounts as compared to the R& D Centers and Comprehensive Centers of
their $65.2 million in core funds from OERI on research, development,
dissemination and technical assistance activities. (See fig. 3.)

83% 5%

4% 2%

3% 2%

1% Research Technical Assistance Evaluation

Collaboration Other Development

Dissemination

Page 5 GAO- 02- 190 Education Research

Figure 3: Regional Labs Reported Spending Core Funds from OERI on a Variety
of Activities

To a large extent, Education shapes the priorities that guide the research
conducted by the R& D Centers and targets the technical assistance provided
by the Comprehensive Centers through requirements in agreements with these
entities. However, Education has limited control over the activities of the
Regional Labs because regional governing boards, mandated by the legislation
authorizing Regional Labs, establish regional priorities and are responsible
for directing the Regional Labs in fulfilling the terms of their contracts
with Education. Therefore, the Regional Labs are unlike most federal
education programs because neither the federal government nor state
governments have oversight responsibility for their programs. As the
Congress reauthorizes the Regional Labs, it may wish to consider giving
Education responsibility for the agenda of the Regional Labs and the quality
of the products and services they produce or giving states the
responsibility by providing these funds to each state for subsequent
distribution.

The R& D Centers, Regional Labs, and Comprehensive Centers reported
collaborating and coordinating with each other and Education and cited a
variety of factors that facilitated and hindered such activities. These
programs provided a variety of examples of collaboration and coordination.
They said that they were most likely to engage in these activities when they
shared a common interest in a specific student

21% 17%

8% 16% 7% 6%

25%

Research Technical Assistance Evaluation

Collaboration Other

Development Dissemination

Page 6 GAO- 02- 190 Education Research

population, such as English language learners, or in a specific topic, such
as assessment. Relationships between the R& D Centers, Regional Labs, and
Comprehensive Centers facilitated collaboration and coordination. Moreover,
Education played a proactive role in encouraging such activities by
requiring joint activities such as conferences as part of their funding
agreements, and by identifying areas where collaboration would be
beneficial.

Current evaluation practices for assessing the R& D Centers, Regional Labs,
and Comprehensive Centers have provided only limited information about the
performance of these organizations and have not been useful for making
future funding decisions. The law requires that Education use peer review a
process that relies on knowledgeable individuals to make independent
assessments of research?s technical and scientific merit to evaluate OERI
activities, including those undertaken by the R& D Centers and Regional
Labs. Although peer review is well accepted and widely used throughout the
government to assess the merit of research proposals and the scientific
soundness of research, it does not directly assess research usefulness,
outcomes, or effects. Moreover, Education?s procedures for peer reviews had
a potential for bias and were cumbersome, which limited the usefulness of
their findings. Finally, peer review is of limited value for Regional Labs
because most of their activities do not involve research. With regard to the
Comprehensive Centers, Education?s 2000 evaluation of the Comprehensive
Centers provided useful information about the Comprehensive Centers as a
network but limited information on the performance of individual centers.
Thus, the information from the evaluation could not be used to inform
decisions pertaining to funding of individual Comprehensive Centers or to
improve ongoing practices. Because of the shortcomings of the evaluations of
R& D Centers, Regional Labs, and Comprehensive Centers, we are suggesting
that the Congress consider directing Education to use other accepted
evaluation techniques to assess applied research, development,
dissemination, and technical assistance activities. Moreover, we are
recommending to Education that it revise its peer review standards to allow
for division of labor and greater concentration on assessing the quality of
projects, services and products, rather than reviewing procedural materials.
We are also recommending that it design future evaluations of the
Comprehensive Centers to provide information on individual centers.

Page 7 GAO- 02- 190 Education Research

The R& D Centers, Regional Labs, and Comprehensive Centers share
responsibility with other programs created by the Congress for education
research, research- based activities, and technical assistance. 2 Many of
these programs are located in six different offices throughout Education.
(See fig. 4.)

2 Education funds many additional research and support entities, including
Special Education and Early Childhood Regional Resource Centers, Eisenhower
Regional Math and Science Consortia, and National Centers for Vocational
Education. Background

Page 8 GAO- 02- 190 Education Research

Figure 4: Education Offices with Responsibility for Education Research,
Research- Based Activities, and Technical Assistance

Comprehensive Centers

Bilingual Education Research

Eisenhower Regional Mathematics and Science Consortia

Regional Technology in Education Consortia

Field Initiated Studies

American Oversees Research Centers

International Research and Studies

Centers for International Business Education

National Centers for Career and Technical Education

Adult EducationNational Leadership Activities

Secretary of Education

Office of Special Education and Rehabilitative Services (OSERS)

National Institute on Disability and Rehabilitation Research

Special Education Technical Assistance and Dissemination

Special Education Studies and Evaluations

Special Education Research and Innovation to Improve Services and Results
for Children with Disabilities

Early Childhood Technical Assistance

Rehabilitation Services Demonstration and Training Special Demonstration
Office of

Vocational and Adult Education (OVAE) Office of

Postsecondary Education (OPE) Office of

Educational Research and Improvement (OERI)

Office of Bilingual Education and Minority Languages Affairs (OBEMLA) Office
of

Elementary and Secondary Education (OESE)

R& D Centers

Regional Labs

Page 9 GAO- 02- 190 Education Research

For example, the Individuals with Disabilities Education Act established a
special education research and innovation program as well as technical
assistance centers to improve services and results for children with
disabilities. The Rehabilitation Act established a National Institute for
Disability and Rehabilitation Research. The Office of Special Education and
Rehabilitative Services administers these programs. The amount of funding
received by these programs in fiscal year 2000 ranged from $70,000 for 11
American Overseas Research Centers to $86.5 million for the National
Institute on Disability and Rehabilitation Research.

OERI is Education?s lead office for educational research and development.
Its goals are to promote quality and equity in education by funding
research; developing new learning materials, teaching techniques, and
methods of organizing schools; demonstrating and evaluating promising
educational practices; disseminating research- based information; and
collecting data related to schools in the United States and other nations.
Unlike other Education offices, OERI?s activities span all grade levels,
from preschool through adult education, and all major content areas of
instruction.

The R& D Centers, established in the 1960s to increase fundamental knowledge
in education, are administered by OERI. Over the years, legislative changes
have repositioned their placement in Education. Most recently, the
Educational Research, Development, Dissemination and Improvement Act of 1994
reorganized OERI, implementing measures that changed the way the R& D
Centers related to Education. First, the act created the National
Educational Research Policy and Priorities Board to work with the Assistant
Secretary of OERI to establish a long- term national agenda for research,
development, and dissemination activities. Unlike previous boards, this
board was charged with improving research priorities and developing
standards for evaluating OERI research, including that done by the R& D
Centers. Second, the act established five national research institutes 3
within OERI, each with its own research focus, and placed the R& D Centers,
as well as field- initiated studies and other research- related programs,
under the appropriate institute. The Congress is expected to start the
reauthorization of OERI in 2002; the last

3 The 5 OERI institutes are relatively small. Each employed between 8 to 14
professional and support staff. In contrast, the National Institute of
Mental Health, one of the Department of Health and Human Services 19
institutes, employed over 400 staff at its headquarters office.

Page 10 GAO- 02- 190 Education Research

reauthorization was in 1994 and was intended to be on a 5- year cycle.
Figure 5 shows the locations of the R& D Centers.

Figure 5: Locations of the R& D Centers

Legislation has changed the focus of the Regional Labs over time. Originally
established to resemble the labs funded by the Atomic Energy Commission,
such as the Los Alamos National Laboratory, the Congress

ME MA

NH NY

VT RI CT NJ PA WV

VA MD DE

NC SC GA TN

MS AL FL KY

IN OH MI

WI AR

LA OK

TX NM AZ

IL MT

IA MN

ND SD

NB KS CO WY

MO UT ID

NV CA

OR WA

Center for the Study of Teaching and Policy (CTP)

National Center for Postsecondary Improvement (NCPI)

Center for Research on Evaluation, Standards, and Student Testing (CRESST)

Center for Research on Education, Diversity and Excellence (CREDE)

National Center for Early Development and Learning (NCEDL)

Center for Research on the Education of Students Placed At- Risk (CRESPAR)
Consortium for

Policy Research in Education (CPRE)

National Research Center on the Gifted and Talented (NRC/ GT)

National Center for Study of Adult Learning and Literacy (NCSALL) National
Research and

Development Center on English Learning and Achievement (CELA) Center for the
Improvement

of Early Reading Achievement (CIERA) National Center for Improving

Student Learning and Achievement in Mathematics and Science (NCISLA)

HI AK

Page 11 GAO- 02- 190 Education Research

created the Regional Labs in 1965 to conduct long- term activities to
address national educational problems. However, funding to support these
activities was never made available, limiting the scope of the Regional Labs
to smaller- scale projects. Further, in the late 1970s, the Congress?s
negative reactions to federally supported curriculum projects prompted
Education and the Regional Labs to discontinue all large- scale nationally
oriented curriculum projects. As a result, Regional Labs developed an
increasingly regional agenda. In 1994, the Congress gave the governing board
of each Regional Lab sole responsibility for determining if the Regional Lab
fulfilled the terms of its contract with Education and its regional agenda.
The Congress mandated that each governing board reflects a balanced
representation of states in the region, as well as interests and concerns of
regional constituencies, including teachers and education researchers.
Figure 6 shows the states included in the regions of the 10 Regional Labs.

Page 12 GAO- 02- 190 Education Research

Figure 6: Regions of the Regional Labs

a PREL includes: American Samoa, Commonwealth of the Northern Mariana
Islands, Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap),
Guam, Hawaii, Republic of Marshall Islands, and the Republic of Palau.

NC SC GA MS AL

FL WV

VA TN

KY AR

LA OK

TX NM

IN OH MI

WI IL IA MN ND

SD NB

KS CO WY

MO AZ

UT NV

CA MT

ID OR

WA ME

MA NH

NY VT

RI CT

NJ PA MD DE

HI AK

WestEd Pacific Resources for Education and Learning (PREL)

Northwest Regional Educational Laboratory (NWREL)

Mid- continent Regional Educational Laboratory (McREL)

North Central Regional Educational Laboratory (NCREL)

Northeast and Islands Laboratory at Brown University (LAB)

Laboratory for Student Success (LSS)

Appalachia Educational Laboratory (AEL)

Southeastern Regional Vision for Education (SERVE) Southwestern Educational

Development Laboratory (SEDL)

a

Page 13 GAO- 02- 190 Education Research

Created in 1994, the Comprehensive Centers were established more recently
than the R& D Centers and the Regional Labs. The Improving America?s Schools
Act of 1994 consolidated the functions of 48 categorical technical
assistance centers 4 that supported programs authorized under the ESEA,
including Title I, Migrant Education, and Indian Education, into 15
Comprehensive Centers. The Congress created the Comprehensive Centers to
support comprehensive, cross- program assistance as envisioned in the law,
and placed them under the Office of Elementary and Secondary Education
(OESE) and the Office of Bilingual Education and Minority Languages Affairs
(OBEMLA). Figure 7 show the states included in the regions served by the
Comprehensive Centers.

4 Categorical programs typically permit funds to be used only for specific,
narrowly defined purposes and populations, such as migrant, Indian, or
economically disadvantaged children.

Page 14 GAO- 02- 190 Education Research

Figure 7: Regions of the Comprehensive Centers

a Region XV includes: American Samoa, Commonwealth of the Northern Mariana
Islands, Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap),
Guam, Hawaii, Republic of Marshall Islands, and the Republic of Palau.

HI MT

WY ID

OR WA

Region X NM AZ

CO UT NV

Region IX Region XV

IN OK

IL NB

KS MO Region VII TX Region VIII

FL Region XIV GA MS AL

AR LA

Region V WV

VA NC SC TN

KY Region IV NJ PA

MD DE OH

Region III NY Region II

ME MA

NH VT RI CT Region I

CA Region

XII MI

WI IA MN

ND SD AK

Region XIII

a

Region XI

Region VI

Page 15 GAO- 02- 190 Education Research

The amount of resources allocated to the R& D Centers, Regional Labs, and
Comprehensive Centers differs, but the overall investment in these programs
is modest. For example, to operate in fiscal year 2000, each R& D Center
received from $1.5 million to $6.6 million, each Regional Lab received from
$3.8 million to $8.6 million, while each Comprehensive Center received from
$0.9 million to $2.7 million. The core budget for all these programs totaled
about $130 million.Table 1 shows the key features of the R& D Centers,
Regional Labs, and Comprehensive Centers.

Table 1: Key Features of the R& D Centers, Regional Labs, and Comprehensive
Centers Program

Responsible Education office

FY 2000 core funding a

FY 2000 average and range of funding per unit to support core program

Allowable award recipients

R& D Centers Office of Educational Research and Improvement $35.5 million b
Average: $3 million

Range: $1.5 million to $6. 6 million

University partnerships consisting of universities and other not- for-
profit organizations Regional Labs

Office of Educational Research and Improvement $65.2 million b Average: $6.
5 million

Range: $3.8 million to $8. 6 million

Not- for- profit organizations and universities

Comprehensive Centers Office of Elementary and

Secondary Education and Office of Bilingual Education and Minority Languages
Affairs

$28.6 million Average: $1. 9 million Range: $0.9 million to $2. 7 million

Public or private not- for- profit organizations, universities, and
consortia of these institutions

a The core funding figures do not include supplemental funds that R& D
Centers, Regional Labs and Comprehensive Centers may have received during
fiscal year 2000 from Education or other agencies for special activities.

b These figures are based on amounts reported to us by R& D Center, Regional
Lab, and Comprehensive Center officials. Education, however, provided
somewhat different figures, reporting $34.9 million in core funding for the
R& D Centers and $65 million for the Regional Labs.

Because of the importance of education research, research- based activities
and technical assistance in improving schools, many studies have focused on
the R& D Centers, Regional Labs, and Comprehensive Centers. Education has
funded various assessments of the R& D Centers and the Regional Labs and has
recently conducted an evaluation of the Comprehensive Centers. In addition,
the R& D Centers and the Regional Labs have been studied and discussed by
numerous independent organizations, including the National Academy of
Sciences, the Brookings Institution, the RAND Corporation, and the National
Educational Research

Page 16 GAO- 02- 190 Education Research

Policy and Priorities Board. 5 Several studies, including the 1992 National
Academy of Sciences report and a 2000 Brookings Institute report, have
concluded that the funds available to OERI to support research have been,
and continue to be, insufficient to support long- term, large- scale
efforts.

Laws define different missions and roles for the R& D Centers, Regional
Labs, and Comprehensive Centers, and these differences are reflected in how
these programs spend their money. R& D Centers focus on national research
priorities, such as student assessment. Although both Regional Labs and
Comprehensive Centers have a regional orientation, Regional Labs focus on
meeting the needs of the regions. Comprehensive Centers focus on assisting
customers in their regions implement federal education agendas, such as
helping to close the achievement gaps for federally targeted groups like
disadvantaged students. Education uses funding documents and program
oversight to direct and prioritize the activities of the R& D Centers,
Regional Labs, and Comprehensive Centers and shape the agendas of the R& D
Centers and Comprehensive Centers. Unlike R& D Centers and Comprehensive
Centers, the Regional Labs have governing boards. Because these governing
boards determine regional agendas and oversee Regional Lab activities,
Education has limited ability to shape the agendas of Regional Labs or
ensure accountability for their products and services.

5 The National Academy of Sciences, Center for Education, Committee on
Scientific Principles in Education Research, Science, Evidence, and
Inference in Education: Report of a Workshop (Washington, D. C.: National
Academy Press, 2001); The National Academy of Sciences, Committee on
Scientific Principles in Education Research, Research and Educational
Reform: Roles for the Office of Educational Research and Improvement

(Washington, D. C.: National Academy Press, 1992); The Brookings
Institution, Can We Make Education Policy on the Basis of Evidence? What
Constitutes High Quality Education Research and How Can It Be Incorporated
Into Policymaking?? (Washington, D. C.: 1999); RAND Corporation, Setting
Standards for Education Research (forthcoming); National Educational
Research Policy and Priorities Board, U. S. Department of Education,

Investing in Learning : A Policy Statement with Recommendations on Research
in Education (Washington, D. C.: 1999) and A Blueprint for Progress in
American Education

(Washington, D. C.: 2000). Activities of the Three

Programs Reflect Legislative Mandates and Support Education?s Agenda, but
Education Has Limited Control over Regional Labs

Page 17 GAO- 02- 190 Education Research

The Congress created a separate primary focus for the R& D Centers, Regional
Labs, and Comprehensive Centers and gave them the responsibility of
performing specific activities. Because of differences in their mandates,
the R& D Centers, Regional Labs, and Comprehensive Centers have different
roles in supporting Education?s research agenda. The Educational Research,
Development, Dissemination, and Improvement Act of 1994 places the R& D
Centers under education research institutes in OERI, each of which addresses
a specific content area, and requires them to carry out the purposes for
which the institutes were created by conducting research and development. In
contrast, rather than requiring the Regional Labs to address a particular
content area, the act requires them to use research- based knowledge to
address the issues in the regions they serve and assigns them an expansive
array of activities to perform. Like the Regional Labs, the Comprehensive
Centers have extensive mandates that require the Comprehensive Centers to
focus on certain customers by giving priority to schools with schoolwide 6
programs and the highest number of children in poverty. The law, however,
allows Education to guide the general direction of the mandated activities.
Table 2 provides the primary focus of the R& D Centers, Regional Labs, and
Comprehensive Centers and summarizes their mandated activities.

6 Schoolwide programs combine resources from various Education programs,
such as those authorized by the Individuals with Disabilities Education Act
and the Safe and Drug Free Schools and Communities Act, to enhance teaching
and learning for all students in a school. Laws Mandate Missions

and Activities of the R& D Centers, Regional Labs, and Comprehensive Centers

Page 18 GAO- 02- 190 Education Research

Table 2: Primary Focus and Mandated Activities of the R& D Centers, Regional
Labs, and Comprehensive Centers Program Primary focus Mandated activities

R& D Centers Address national research priorities on specified content areas
or student groups

Conduct basic research, applied research, and dissemination, which may also
include development

Regional Labs

Serve needs of geographical regions regarding the implementation of broad-
based systemic school improvement strategies

 Conduct applied research projects

 Develop and disseminate educational research products

 Develop a plan for identifying and serving the needs of its region

 Serve the educational development needs of the region

 Facilitate communication between education experts, school officials,
teachers, parents, and librarians

 Provide support, training, and technical assistance

 Collaborate and coordinate with other technical assistance providers

 Bring teams of experts together to develop and implement school
improvement plans

 Collaborate with the OERI institutes

 Consult with state educational agencies and libraries

 Develop strategies to use schools as components in reforming education and
reviving rural communities

 Report and disseminate information on overcoming obstacles faced by rural
schools

 Identify successful practices that have been developed by the Regional
Labs or other educational entities in the region Comprehensive Centers
Provide technical assistance and

training related to the administration and implementation of programs
authorized under ESEA

 Provide support, training, and assistance to state and local educational
agencies, tribal divisions of education, and other recipients of Education
funds

 Improve the quality of instruction, curricula, assessments, and other
aspects of school reform

 Implement effective schoolwide programs

 Meet the needs of children served by programs funded by Education

 Implement high quality professional development

 Improve the quality of bilingual education

 Create safe and drug free environments

 Implement educational applications of technology

 Evaluate programs

 Expand the involvement of parents in the education of their children

 Reform schools, school systems, and the governance and management of
schools

 Meet the special needs of schools and children in urban and rural areas

 Provide technical assistance, and coordinate and cooperate with Regional
Labs, Eisenhower regional consortia, literacy centers, and other entities
engaged in research, development, dissemination, and technical assistance

The spending patterns of the R& D Centers, Comprehensive Centers, and
Regional Labs reflect their mandates and missions. For example, the Congress
authorized the R& D Centers to conduct research and development in order to
increase fundamental knowledge of central issues in education. To support
this mission, they reported spending 73 percent of the $35.5 million in core
funding they received from OERI in fiscal year

Page 19 GAO- 02- 190 Education Research

2000 on research and an additional 14 percent on development and
dissemination. 7 The Congress created the Comprehensive Centers to help
state educational agencies, school districts, and schools within an assigned
region implement federal elementary and secondary school programs by
providing technical assistance and training. Accordingly, the Comprehensive
Centers reported spending most of their fiscal year 2000 $28.6 million core
funds from OESE 83 percent on technical assistance. Regional Labs were
authorized to conduct a wide range of research- based activities, including
applied research, 8 development, dissemination, and technical assistance.
Their spending reflected these purposes: the Regional Labs reported spending
25 percent of their fiscal year 2000 $65.2 million core funds from OERI on
research, 17 percent on development, 16 percent on dissemination, and 21
percent on technical assistance.

Most R& D Centers, three Regional Labs, and two Comprehensive Centers used
funds from other organizations, including other federal agencies, state and
local educational agencies, and foundations, to expand work they were
performing for Education. For example, one R& D Center, the National Center
for Postsecondary Improvement, used funding from the Pew Charitable Trust to
extend the scope of a study examining the impact of state and university
admission standards on secondary schools and students. Similarly, 10 R& D
Centers reported that they leveraged additional money from other federal and
state agencies and not- for- profit organizations to promote Education?s
research agenda. Two of these R& D

7 We pretested the survey with officials of the R& D Centers and
Comprehensive Centers. These officials agreed that the general categories of
activities- research, dissemination, technical assistance, collaboration,
development and evaluation- were sufficiently distinct. However, in follow-
up interviews with respondents, some noted the difficulty in separating
certain interrelated activities from one another. For example, respondents
reported difficulty in separating technical assistance from dissemination,
since both might take place during the course of one activity. Consequently,
percentages reported are estimates.

8 The terms ?basic research? and ?applied research? are found in the
authorizing legislation for the R& D Centers and ?applied research? is found
in the authorizing legislation for the Regional Labs, but the legislation?s
definition for research does not distinguish between basic and applied
research. Similarly, the congressionally mandated National Education
Research Policies and Priorities Board discusses these terms in Investing in
Learning : A Policy Statement with Recommendations on Research in Education
(Washington, D. C.: 1999) but gives no formal definitions. Further, the
Board concluded that Education conducts no basic research. Rather, such
research is conducted in other federal agencies, most notably the National
Institute for Child Health and Human Development, the Office of Naval
Research, and the National Science Foundation. Education concentrates on
applied research and, according to the Board, is contributing some of the
important applied research aimed at comprehensive or standards- based
reform, testing, and assessment.

Page 20 GAO- 02- 190 Education Research

Centers reported receiving more funding from these other sources of funding
than from OERI. Three Regional Labs also reported receiving funds from non-
OERI sources, with these funds composing between 1 to 27 percent of their
budgets. Two Comprehensive Centers reported receiving external funding and
these funds accounted for 0. 3 to 2 percent of those Centers? funding.

Education shapes the activities of the R& D Centers, Regional Labs, and
Comprehensive Centers through its funding documents and program monitoring.
Education uses funding documents, such as grant announcements and statements
of work, to guide and direct activities included in the mandates and to
obtain help in implementing department activities. In addition, Education
assigns program officers to oversee the activities of the R& D Centers,
Regional Labs, and Comprehensive Centers.

Although priorities in R& D Center cooperative agreements are broad and do
not impose particular methods for researching a topic, they are specific
enough to shape the direction and breadth of the R& D Centers? research
agenda. For example, in spelling out the priorities for the R& D Center for
enhancing young children?s development and learning, Education identified
topics, theories, and research areas that the Center should address.
Similarly in spelling out the priorities for the R& D Centers for meeting
the needs of diverse student populations, Education identified topics,
theories, and student populations to be studied.

In addition, the R& D Centers? cooperative agreements require that a portion
of R& D Centers? funds be spent on tasks defined by OERI. R& D Centers?
cooperative agreements require them to reserve 5 percent of their core funds
to carry out OERI initiated activities that assist OERI in carrying out its
responsibilities. For example, OERI may require an R& D Center to write a
briefing paper or conduct a research project.

Contracts between Education and the Regional Labs give the Regional Labs
control over most of their activities, but also give Education the
opportunity to guide some of their work. Regional Labs identify the critical
issues in their region and develop plans to address these issues. However,
these contracts also allow Education to assign each Regional Lab a broad
specialty area for example, early childhood education or educational
technology- that reflects the national education agenda and is aligned
Education?s Ability to

Shape R& D Center, Regional Lab, and Comprehensive Center Agendas Varies

Page 21 GAO- 02- 190 Education Research

with a dominant theme of an OERI research institute with which they are
associated. 9 In making Regional Labs responsible for a particular specialty
area, the contracts required that they (1) conduct development, applied
research, and dissemination in that area; (2) keep abreast of developments
in their designated field; (3) provide subject area expertise to other labs;
and (4) work cooperatively with OERI institutes as appropriate. Although
Education defines the general type of activities for Regional Labs in
specialty areas, the governing boards determine the focus of the activities
and the extent to which they met the requirements of the contract.
Additionally, the contracts require Regional Labs to work together on areas
of concern to all Regional Labs, such as how to effectively disseminate
their products and develop a telecommunications network. Regional Labs
reported spending about 25 percent of their funds on these required national
activities and on other activities with a national purpose. Unlike the
cooperative agreements for the R& D Centers, Regional Lab contracts do not
require them to do work to support OERI activities. Prior to 1995, the
contracts required that the Regional Labs spend 1 percent of their core
funds from OERI to support OERI. However, this requirement was dropped
because of objections from the Regional Labs. Currently, Regional Labs may
agree to perform work for OERI in return for additional funding.

Although the Comprehensive Centers are not research entities, Education
shapes their activities by setting priorities for them in funding documents.
Such priorities include, for example, meeting with school district officials
to review and provide advice on district procedures for meeting federal
requirements and assisting school districts in the development of student
assessments. In addition, Comprehensive Centers are required to engage in
common activities identified by Education such as conducting annual
conferences on school improvement activities that promote Education?s
agenda.

In addition to funding documents, program officers who administer grants and
contracts may play an important role in influencing activities of the R& D
Centers and Comprehensive Centers by ensuring that work

9 The contracts awarded for December 1995 through December 2000 contained
tasks for specialty areas. The contracts awarded for December 2000 through
December 2005 replaced ?specialty area? tasks with ?national leadership
area? tasks, such as educational leadership and teacher development, and
required that the leadership role include synthesizing research,
disseminating information, and providing training to other Regional Labs.

Page 22 GAO- 02- 190 Education Research

performed is consistent with work proposed and that funds are being used as
effectively and efficiently as possible. Program officers are supposed to
perform the following types of activities: (1) help to develop funding
documents and ensure that R& D Centers and Comprehensive Centers are in
compliance with these documents; (2) review progress reports, financial
reports, and products; and (3) approve dissemination plans, staffing
changes, and activities funded by other sources. Program officers reported
using these oversight functions to ensure that the activities of R& D
Centers and Comprehensive Centers are consistent with their proposals. For
example, Education program officials reported that they identified
activities that were inconsistent with those in the R& D Centers? proposals
and subsequently negotiated alternative activities, indicated where
collaboration between some R& D Centers would be beneficial, and encouraged
R& D Centers to drop nonpromising lines of research. Program officers also
may play a role in determining supplemental and future funding decisions.

Even though regional governing boards are responsible for the oversight of
Regional Labs, Education?s program officers, nonetheless, may still have the
potential to influence the activities of these labs. A program officer
assigned to a Regional Lab described their role as that of ?critical
friends? who use their professional expertise and interpersonal
relationships with Regional Labs? staff to influence the activities of the
Regional Labs. In addition, Regional Lab program officers, like R& D Center
program officers, may play a role in determining supplemental and future
funding decisions.

Time and authority may limit program officers? ability to exercise their
influence. In OERI, one program officer is generally assigned to each R& D
Center and Regional Lab. The program officers we interviewed reported
spending about 50 percent of their time on monitoring activities related to
the R& D Centers and Regional Labs because of other assigned
responsibilities. 10 Because the R& D Centers and Regional Labs are complex
organizations and prolific producers of products and services, officers have
to be very selective in targeting their own time. Only one program officer
is assigned to monitor all 15 Comprehensive Centers.

10 Program officers said they spend the other 50 percent of their time on a
wide range of activities including working on field initiated studies and
interagency research projects, writing statements of work and cooperative
agreement announcements, coordinating with a number of Education offices,
and monitoring other OERI programs.

Page 23 GAO- 02- 190 Education Research

The Congress has consistently given education program oversight to either a
federal agency- usually Education or to the states, since the states are
generally responsible for the education of their students. The Regional Labs
are unlike other federal education programs because neither the federal
government nor state governments have oversight responsibility for programs.
Specifically, Education has little control over Regional Labs because
regional boards govern them. This occurs in spite of the fact that the
Regional Labs get the largest share of the federal dollars devoted to these
three programs that conduct or support research. The law requires that the
Regional Labs establish governing boards with regional representatives. The
regional boards have sole responsibility for determining the regional agenda
and for determining whether the Labs are fulfilling the responsibility of
their contracts, even though these contracts are funded by Education. 11
Even for the national specialty areas, Education sets only wide parameters
while the governing boards determine specific activities. Education limits
its communications with Regional Labs mainly to administrative issues,
according to a director we interviewed at a Regional Lab. In addition,
Education?s Regional Lab program officers told us their comments on Regional
Lab work and products were only advisory.

The law requires each Regional Lab to establish a governing board that
reflects both a balanced representation of the states in the region and the
concerns of regional constituencies and includes teachers and education
researchers. In addition, Education requires that every chief state school
officer in the area served by the Regional Labs be offered an opportunity to
serve on the board or to designate a representative. These safeguards,
however, do not assure that priorities of each state in the region will be
met. A variety of factors potentially make achieving balanced regional and
state representation challenging. First, Regional Labs are not given
guidance on how to obtain balance between states and regional interests, and
regions, unlike states, have no formal governing body to establish
educational priorities. Second, the governing boards, not the states, have
the authority to determine how members are nominated and selected. Finally,
states within regions vary substantially in size, population density,
poverty levels, and ethnic composition. These factors may determine state
educational priorities in a way that would make them vary widely. For
example, California- a diverse and large state in size and population- is

11 Education provides financial oversight, but does not oversee the Regional
Labs? products and services. Regional Labs? Governance

Structure Limits Education?s Ability to Provide Product Oversight and Shape
Their Agendas

Page 24 GAO- 02- 190 Education Research

included with Utah, Nevada and Arizona in one region. New York also a
diverse and large state- is included with Vermont and other New England
states in another region.

R& D Centers, Regional Labs, and Comprehensive Centers provided many
examples of collaborative and coordinated activities. R& D Centers, Regional
Labs, and Comprehensive Centers are required to collaborate and coordinate
with each other and Education. The programs reported they are most likely to
engage in collaborative and coordinated activities when they share a common
interest in a specific student population, such as English language
learners, or a specific topic, such as assessment. Partnerships between
programs, common memberships in consortia, and staff members who are
employed by more than one of these organizations facilitate collaboration
and coordination and help leverage resources. Education plays a proactive
role in promoting collaboration and coordination by including requirements
for certain activities as part of its funding documents and in its ongoing
negotiations with these organizations. However, certain factors differences
in student populations, funding uncertainty, and competition reduce
opportunities for collaboration and coordination.

R& D Centers, Regional Labs, and Comprehensive Centers collaborate and
coordinate with each other and Education as required by law and the
documents that control their funding. Certain types of collaboration and
coordination flow naturally from overlapping needs, interests, and
resources. For example, Regional Labs and Comprehensive Centers with
populations of students with limited English proficiency would naturally
make use of the R& D Centers, Regional Labs, and Comprehensive Centers that
have expertise in that area.

R& D Centers, Regional Labs, and Comprehensive Centers reported a variety of
collaborative efforts for fiscal year 2000, including joint projects and
training. Joint projects included:

 Two Comprehensive Centers (the Northern California Comprehensive Center
and the Southern California Comprehensive Center) worked together to produce
teleconferences to help low performing schools.

 Ten Regional Labs and an R& D Center (the National Center for Early
Development and Learning) produced a training guide entitled

Continuity in Early Childhood: A Framework for Home, School, and Community
Linkages.

Collaboration and Coordination among the R& D Centers, Regional Labs,
Comprehensive Centers, and Education Is Ongoing

R& D Centers, Regional Labs, and Comprehensive Centers Collaborate and
Coordinate in Various Ways

Page 25 GAO- 02- 190 Education Research

 An R& D Center (the Center for Research on the Education of Students
Placed At- Risk) and a Regional Lab (the Northwest Regional Lab) produced a
joint publication about parent involvement in schools.

 A Regional Lab (the Northwest Regional Lab) and an R& D Center (the Center
for Research on Evaluation, Standards, and Student Testing) created the
Classroom Assessment Tool Kit.

Examples of training included:

 A Comprehensive Center (the Southern California Comprehensive Center)
trained other Comprehensive Centers to teach instructors how to coach
children learning to read.

 An R& D Center (the Center for Research on Evaluation, Standards, and
Student Testing) trained a Regional Lab (WestEd) to use a data collection
tool.

 An R& D Center (the National Center for Improving Student Learning and
Achievement in Mathematics and Science) worked with a Comprehensive Center
(the Region VI Comprehensive Center) to conduct a professional development
project by teaching people to train math and science teachers.

 A Regional Lab (the Southwest Educational Development Laboratory) provided
training to a staff member of a Comprehensive Center (the Southeast
Comprehensive Center) in the use of Flashlight and Compass: A Collection of
Tools to Promote Instructional Coherence a tool for establishing teacher
study groups.

R& D Center, Regional Lab, and Comprehensive Center staff also provided many
examples of coordination efforts meant to ensure that each was aware of the
others? projects. Some examples include the following:

 As a follow- up to Education?s National Awards for Model Professional
Development, three Regional Labs, WestEd, the North Central Regional Lab,
and the Mid- Continent Regional Lab, studied how the awarded districts
supported districtwide teacher and student learning. An R& D Center, the
Center for the Study of Teaching and Policy, shared its data on resource
allocation among some of the same districts and contributed to the research
questions and design. The Regional Labs shared the findings with Education.

 An R& D Center, Center for Research on Education, Diversity, and
Excellence, reviewed work on Spanish Writing Assessment done by a Regional
Lab, Northwest Regional Lab.

Page 26 GAO- 02- 190 Education Research

Legislation requires that the Regional Labs and Comprehensive Centers
collaborate and coordinate with each other and with the R& D Centers, but
their mandates differ in the amount of collaboration and coordination they
require. The Regional Labs are required to collaborate and coordinate with
each other, Education- funded technical assistance providers, and OERI
institutes, and to share and plan joint activities with other
Educationfunded state, and federal programs. They are also required to
establish a network for sharing information, planning activities involving
multiple regions, and working on national projects. The Comprehensive
Centers are required to share information, coordinate services, and work
cooperatively with the Regional Labs, R& D Centers, Education?s regional
offices, state and local educational agencies, and all other Educationfunded
research, development, dissemination, and technical assistance programs. The
R& D Centers do not have specific legislative requirements to collaborate
and coordinate with other Education- funded programs; however, they are
required to do so in their cooperative agreements.

R& D Centers, Regional Labs, and Comprehensive Centers collaborate and
coordinate when they have an interest in the same student population. The
following examples illustrate how student populations provide a focal point
for collaboration and coordination.

 A member of the Mid- Continent Regional Lab, which includes states with
large numbers of Native Americans, sat on the steering committee of the
National Research Center on the Gifted and Talented, an R& D Center. This
committee oversaw the production of a publication on talented American
Indian and Alaskan Native students.

 Two R& D Centers, the Center for Research on Education, Diversity, and
Excellence and the National Center for Improving Student Learning and
Achievement in Mathematics and Science, published a newsletter on issues
related to how diverse students learn math and science.

 All 15 of the Comprehensive Centers created an Internet mailing list about
English- language learners to share information from their regions, identify
staff with proficiency in meeting the needs of Englishlanguage learners, and
disseminate information.

The structures of the R& D Centers, Regional Labs, and Comprehensive Centers
foster collaboration and coordination among them and other entities. In some
cases, partnerships may encourage collaboration. In other cases,
collaboration occurs because one single entity operates both an R& D Center,
Regional Lab, and/ or Comprehensive Center. Legislative Requirements

and Common Interests Foster Collaboration and Coordination

Interlacing Organizational Relationships Generally Facilitate Collaboration
and Coordination

Page 27 GAO- 02- 190 Education Research

Partnerships may encourage collaboration by establishing bridges between
programs. R& D Centers, Regional Labs, and Comprehensive Centers are
programs that operate through parent organizations universities, not- for-
profit organizations, and educational agencies. These parent organizations
create consortia- formal partnerships with other universities, not- for-
profit organizations, for- profit corporations, and educational agencies- to
run R& D Centers and Comprehensive Centers. Participants in partnerships
that run R& D Centers range from 2 universities to 29 universities and not-
for- profit organizations. Figure 8 shows one R& D Center that is a
consortium of 5 universities and the affiliations they have with other R& D
Centers.

Page 28 GAO- 02- 190 Education Research

Figure 8: Organizational Structure of an R& D Center and Its Affiliations
with Other R& D Centers

Unlike R& D Centers and Comprehensive Centers, the parent organizations that
run Regional Labs do not form partnerships with other universities,

Michigan State University University of

Virginia

U. S. Department of Education (administered by the Office of Educational
Research

and Improvement)

University of Michigan

University of Southern

California University of

Minnesota Center for the

Improvement of Early Reading Achievement

(Parent Organization) Center for

the Study of Teaching and Policy (CTP)

Funds Granted

Consortium for Policy Research in Education (CPRE)

National Center for Postsecondary Improvement (NCPI)

Center for Research on Education, Diversity and Excellence (CREDE)

Affiliations With Other R& D Centers

Page 29 GAO- 02- 190 Education Research

not- for- profits, or educational agencies to run the Regional Labs. They
may, however, be asked by a Comprehensive Center or R& D Center to enter
into a partnership in order to provide specialized services. We found four
parent organizations that operated Regional Labs that had formed
partnerships with Comprehensive Centers. Figure 9 shows an example of such a
partnership.

Page 30 GAO- 02- 190 Education Research

Figure 9: Regional Lab Parent Organization, In- House Programs, and Services
Provided to Partners

Participants in partnerships that run Comprehensive Centers range from three
to eight universities, not- for- profit organizations, and educational
agencies. Most parent organizations that run Comprehensive Centers partner
with other organizations to obtain expert services in specialized

Provides 3 technical assistance and training staff for New England

Comprehensive Center (Region I) Operates Arizona office of

Southwest Comprehensive

Center (Region IX)

WestEd

(Parent Organization) Western Regional Lab Northern

California Comprehensive

Center (Region XI)

Other Federal, State, and Privately Funded Programs Eisenhower

Regional Consortium

Funded by U. S. Department of Education (administered by Office of
Elementary

and Secondary Education) Funded by

U. S. Department of Education (administered by Office of Educational
Research

and Improvement) Other Funding

In House Programs Services Provided to Partners

Page 31 GAO- 02- 190 Education Research

areas, such as migrant education or Indian education. Because each of these
specialized partners may work with as many as four Comprehensive Centers,
these partnerships may establish bridges that forge coordination in
particular topical areas. For example, ESCORT, a former categorical
technical assistance center, specializes in migrant education and partners
with four Comprehensive Centers to provide services in that area.

Collaboration also results when a single organization operates both an R& D
Center, Regional Lab, or a Comprehensive Center. For example, staff from the
Western Regional Lab and the Northern California Comprehensive Center
participated in a workgroup that developed a guide on how schools could
better obtain student perspectives and suggestions to improve school
planning. As shown in figure 9, both the Western Regional Lab and the
Northern California Comprehensive Center are run by the same parent
organization. Five of the 10 parent organizations that run a Regional Lab
also run a Comprehensive Center. One of the 12 parent organizations that
operates an R& D Center also operates a Comprehensive Center.

These interlacing organizational relationships allow R& D Centers, Regional
Labs, and Comprehensive Centers to leverage resources. For example, an
Education official working with the Comprehensive Centers told us that
through their parent organizations and partnerships Comprehensive Centers
are able to leverage the personnel and expertise needed to perform their
work. In those cases where parent organizations run two programs, we were
told that staff divide their time between programs to leverage expertise.
Similarly, a director of a parent organization that runs a Regional Lab and
a Comprehensive Center stated that the Comprehensive Center draws upon
experts assigned primarily to other projects to obtain skills needed to
implement particular activities.

Individual relationships, such as staff members holding multiple
appointments within R& D Centers, Regional Labs, or Comprehensive Centers
also facilitate collaboration and coordination among R& D Centers, Regional
Labs, and Comprehensive Centers. For example, a principal investigator for
the Consortium for Policy Research in Education, an R& D Center, also works
on projects for the National Center for Improving Student Learning and
Achievement in Mathematics and Science, another R& D Center.

Page 32 GAO- 02- 190 Education Research

The funding agreements between Education and the R& D Centers, Regional
Labs, and Comprehensive Centers reflect Education?s interpretation and
implementation of legislative requirements for collaboration and
coordination. The cooperative agreements for the R& D Centers require them
to collaborate and coordinate with Regional Labs, Comprehensive Centers,
other federal programs, policy institutions, and advocacy groups. R& D
Centers also agree, in their funding documents, to conduct an annual meeting
to share research with Education and other research and development
programs, and collaborate with OERI. Regional Lab contracts with Education
require that they participate in at least two meetings a year convened to
discuss issues related to Education- funded programs. In addition,
Education?s contracts for Regional Labs require Regional Lab representatives
to meet with OERI annually and chairpersons of the governing boards to meet
with OERI when their contract begins. Regional Labs also have an option in
their contracts with Education that allows them to earn supplemental funds
by agreeing to perform work in collaboration and coordination with OERI,
including sponsoring meetings and panels and writing briefs. Cooperative
agreements for the Comprehensive Centers require them to meet with seven
different Education- funded programs, including the Regional Labs, to
discuss collaboration and coordination; plan a national conference; engage
in a common project to improve teaching; and collaborate with each other and
local and state educational agencies.

Cooperative agreements for the R& D Centers and Comprehensive Centers also
outline Education?s responsibilities to facilitate collaboration and
coordination. For example, the funding agreement for the Comprehensive
Centers specifies that Education officials will work with Comprehensive
Centers in planning conferences and identifying areas for collaboration and
coordination. The funding agreements that Education has with Regional Labs
do not outline Education?s responsibilities for facilitating collaboration
and coordination.

Education officials may also identify appropriate areas for collaboration
and coordination. For example, an institute director told us that she
contacted an R& D Center in another institute to discuss possible areas for
collaboration and coordination with the R& D Center in her institute.
Similarly, some Comprehensive Center officials stated that the program
officer assigned to them identified areas for cross- program collaboration
and coordination and communicated with them frequently. For example, the
program officer suggested collaborating and coordinating on the creation of
a common framework for working with low performing schools in Comprehensive
Center regions. Education Plays a

Proactive Role in Encouraging Collaboration and Coordination

Page 33 GAO- 02- 190 Education Research

Despite the efforts made by the R& D Centers, Regional Labs, Comprehensive
Centers, and Education in fostering collaboration and coordination, barriers
exist. R& D Centers, Regional Labs, and Comprehensive Centers cited
differences in student populations and topics, uncertain funding, and
competition as hindrances to collaboration and coordination. R& D Centers
have different research focuses and conduct research on different topics.
Regional Labs and Comprehensive Centers serve diverse geographical areas
with different interests. These differences potentially reduce opportunities
for collaboration and coordination. For example, an R& D Center with a focus
on postsecondary education would have little or no reason to collaborate or
coordinate on substantive issues with other R& D Centers, Regional Labs, and
Comprehensive Centers that focus on research related to elementary and
secondary education. Funding uncertainties also make collaboration and
coordination difficult. Directors said they were often reluctant to write
collaborative activities into their proposals because they did not know
which organizations would win future funding competitions. Competition also
limits collaboration and coordination. The education research and technical
assistance business is a competitive industry. Like others in competitive
industries, R& D Center, Regional Lab, and Comprehensive Center staff are
protective of ideas and practices that give them advantages over other
organizations that they perceive as competitors for future sources of
funding.

The recently funded evaluations of the R& D Centers, Regional Labs, and
Comprehensive Centers provided limited information on outcomes of the
activities conducted by these programs. OERI is required to use peer review
to evaluate the R& D Centers and Regional Labs. Peer review is well accepted
and widely used for assessing the merit of research proposals and the
scientific soundness of research. Unlike the R& D Centers, research is only
a relatively small part of what Regional Labs do. Their other activities
dissemination and technical assistance would have been more appropriately
evaluated using methods other than peer review. The peer review processes
that Education used to evaluate the R& D Centers and Regional Labs have
shortcomings that limited the usefulness of the findings. First, the peer
review process used has the potential for bias because the R& D Centers and
the Regional Labs selected most of the products that were reviewed. Second,
the processes were cumbersome. For example, Education required each member
of a review team to write an individual assessment report and to review all
contracts, contract modifications, progress reports, and annual updates for
a three- year period. In addition, with regard to the Comprehensive Centers,
the Differences in Student

Population and Topics, Funding Uncertainty, and Competition Reduce
Opportunities for Collaboration and Coordination

Education Needs to Improve Its Assessments of the R& D Centers, Regional
Labs, and Comprehensive Centers

Page 34 GAO- 02- 190 Education Research

customer satisfaction survey evaluations of the Comprehensive Centers did
not provide information on individual centers.

Traditionally, Education has used peer review to evaluate activities carried
out by OERI, including those conducted by the R& D Centers and the Regional
Labs. The Educational Research, Development, Dissemination, and Improvement
Act of 1994 codified this practice by requiring OERI to develop peer review
standards for evaluating and assessing the performance of recipients of
grants, cooperative agreements, and contracts that exceed $100,000, as well
as for selecting proposals for funding and identifying exemplary and
promising educational programs.

Historically, peer review has been used extensively in the selection of
proposed research projects and, to a lesser extent, to evaluate research and
development programs. Peer review entails an independent assessment of the
technical or scientific merit of research by peers who are scientists with
knowledge and expertise equal to that of the researchers whose work they
review. It is sometimes used to evaluate research when the ultimate outcomes
of the research are unpredictable.

Peer review may be appropriate for evaluating some activities conducted by
the R& D Centers but other evaluation techniques are better suited for
evaluating the many activities of the R& D Centers and Regional Labs.
According to the National Academy of Sciences, although peer review is well
suited for assessing theory- driven research with potential long- term
effects, it is less appropriate for assessing applied research, development,
technical assistance, and dissemination efforts in which outcomes are
anticipated and can be measured over a relatively short period of time. 12
The R& D Centers and Regional Labs engage in many research activities that
are designed to achieve practical outcomes. Evaluation methods that measure
outcomes customer surveys, comparisons with similar programs, and controlled
case studies may be better suited for evaluating these activities.

12 National Academy of Sciences, Committee on Science, Engineering, and
Public Policy,

Evaluating Federal Research Programs (Washington, D. C.: National Academy
Press, 1999). Recent Peer Reviews of

the R& D Centers and Regional Labs Provided Limited Information

Page 35 GAO- 02- 190 Education Research

Education?s peer review processes have the potential for bias and were
cumbersome. Both of these conditions limited the usefulness of findings in
addressing key issues.

 The self- selection of materials for review by R& D Centers and Regional
Labs did not provide a representative cross section of their products and
services. R& D Center staff were involved in deciding which products were to
be reviewed and each Regional Lab nominated five or six major products or
services, two of which were selected for review. While this approach allowed
reviewers an in- depth look at major program initiatives, it did not provide
reviewers with a cross section of the R& D Centers or Regional Labs? work,
nor did it allow them to generalize from these works to other activities.

 Selecting as reviewers for the R& D Centers and Regional Labs some
individuals who have previously evaluated the merits of the grant
applications or proposals raised questions about objectivity. It is likely
that the individuals who selected these organizations as grant or contract
recipients might want their original choices validated.

 Requiring peers to have a broad mix of skills made selection and
scheduling of reviewers difficult. Unlike other agencies that select peers
solely on the basis of their expertise in the area of work, OERI requires
that review panels include individuals with a broad range of knowledge and
experience. For example, OERI standards require peer review panels to
include individuals with in- depth knowledge of education policy or practice
and in- depth knowledge of theories and methods of study related to the
subject area. These requirements complicated the identification of peer
review panels.

 The amount of material to be reviewed was extensive in terms of the short
time frames of the assessments and the complexity of the organizations. Over
a short period of time- for example, 5 days on site for the review of a
Regional Lab, with half that time devoted to data gathering and the other
half to writing the reports- all reviewers were required to read immense
amounts of material, including funding documents, statements of work,
proposals, and progress reports, to learn, in detail, how the programs
performed their work and to write individual reports. Some reviewers
complained that they only had time to ?scratch the surface,? and that much
of the material they had to review was repetitive. If responsibilities could
have been shared, peer reviewers would have been able to gather more in-
depth knowledge.

Page 36 GAO- 02- 190 Education Research

 In some cases, materials chosen for review by the R& D Centers were
incomplete. Reviewers of some programs noted that materials addressing the
rationale, hypotheses, controls, and usefulness of studies were often
insufficient for them to make informed judgements.

 Assessments took place midway in the funding cycles. The reviews generally
took place during the third year of a 5- year contract. As a result,
reviewers were hesitant to be critical in reports because many studies were
on going and therefore could improve by the end of the contract.

 The dual purposes of the peer reviews inhibited candor. On one hand,
Education designed the reviews to be formative evaluations- evaluations that
were to focus on the performance of the programs in terms of their missions
and the technical quality of their products. In this regard, reviews were to
examine the overall quality of the work of the R& D Centers and Regional
Labs, the extent to which R& D Centers and Regional Labs performed their
work on time and met professional standards. However, the reviews were also
designed to assess the usefulness, outcomes, and effects of their work to
help OERI determine if the R& D Centers and Regional Labs merited continued
funding. The peer review process depended exclusively upon expert opinion
rather than directly measuring how useful the research was or its effects.
Reviewers discussed ?the potential? value of activities and were not able to
predict the ability of the entities to contribute substantially to the
field. Moreover, they did not believe their findings should have influence
over funding decisions, which affected what they wrote in their reports.

Education?s evaluation of the Comprehensive Centers met the requirements of
the law but provided little information that would help Education determine
if each Comprehensive Center was meeting the needs of its customers. The
Elementary and Secondary Education Act requires the Secretary of Education
to collect information about the availability and quality of services
provided by the Comprehensive Centers and to conduct surveys to determine if
populations served by the Comprehensive Centers are satisfied with their
access to services and the quality of those services. As part of the year
2000 evaluation, a contractor surveyed the satisfaction of customers who had
participated in either of two activities offered by a Comprehensive Center.
These activities were selected from among all the activities offered by the
Comprehensive Center because they were long- term or intensive. The
contractors surveyed customers by randomly selecting them from a list
prepared by each Comprehensive Center. The contractor also surveyed a
nationally Evaluation of the

Comprehensive Centers Did Not Provide Comparable Information on Individual
Centers

Page 37 GAO- 02- 190 Education Research

representative sample of state and district officials that they identified
as a likely pool of customers for the centers and a sample of gatekeepers-
individuals that had requested or negotiated for services on behalf of their
school or school district. In randomly selecting customers to survey, the
contractor did not choose a number large enough from each Comprehensive
Center?s list to allow any reliable generalizations to be made about a
particular Comprehensive Center. Likewise, the sample of potential customers
and gatekeepers was not suitable to draw conclusions about an individual
Comprehensive Center. The inclusion of representative activities and
customers for each Comprehensive Center would have increased the cost of the
mail survey. However, not drawing a sample that was representative at the
center level reduced the usefulness of the evaluation because Education
could not identify variation across Comprehensive Centers or obtain
information to improve practices at individual Comprehensive Centers.

The Regional Labs are unlike most federal education programs because neither
the federal government nor state governments have oversight responsibility
for their programs. Under the current structure, Education is accountable
for the activities it funds through the Regional Labs, but current laws
limit its ability to oversee those activities. Not only is federal oversight
limited, states also have limited control over the regional agenda. Although
the requirement that governing boards have a balanced representation of
states in the region may ensure state input for the agenda of the Regional
Lab, the regional priorities that the Regional Labs serve may not correspond
to the educational priorities of all states in the regions. Thus, neither
the states nor Education can ensure that the Regional Labs are meeting the
needs of the states.

Congressionally mandated peer reviews of the R& D Centers and Regional Labs
have produced limited information about the overall performance of each
organization, the services they provide, or the extent to which teaching and
learning are improved by the products R& D Centers and Regional Labs
produce. In part, this was because reviewers could not divide the tasks
among themselves, as each reviewer was required to do a full, independent
assessment. In addition, given the present practice allowing the Regional
Lab directors to choose products and services for review, the potential
exists for bias and therefore calls into question the quality of the
assessments. As a result, Education lacks information that would be useful
in making funding decisions or improving the performance of each
organization. Unless standard program evaluation techniques, such as
customer surveys or controlled case studies, are Conclusions

Page 38 GAO- 02- 190 Education Research

introduced into the evaluation process of these entities, these problems are
likely to continue.

The value of the mandated year 2000 evaluations of the Comprehensive Centers
was limited. We recognize that addressing this problem would involve
expanding the sample sizes and, for the consumer survey, the number of
activities selected. However, currently, neither the Comprehensive Centers
themselves nor Education can determine if the customers of a particular
Comprehensive Center are satisfied with some or all of its products and
services. As a result, problems at any given center could go unchecked.
Moreover, Education cannot assess the relative strengths and weakness of
individual Comprehensive Centers so it can improve the services in all
centers and make better funding decisions.

If the Congress wishes to ensure greater accountability to a governmental
entity for the Regional Labs, it could consider either giving Education
responsibility for determining the regional agenda and overseeing the
products and services of the Regional Labs or Education could provide these
funds to states, possibly as part of a larger formula grant, for subsequent
distribution by each state. This would give states greater control in
purchasing research- based educational products and services.

If the Congress wants to increase the usefulness of the assessments of the
R& D Centers and Regional Labs, the Congress should consider allowing
Education to use methods other than peer review when such methods are more
appropriate than peer review for evaluating the activities of R& D Centers
and Regional Labs.

To improve the assessments of the R& D Centers and Regional Labs, we
recommend that the Secretary of Education direct the Assistant Secretary of
OERI to

 use random selection of projects, services and products to be reviewed
when conducting future evaluations of R& D Centers and Regional Labs, and

 revise the peer review standards to allow for division of labor and
greater concentration on assessing the quality of projects, services and
products.

To improve the performance and usefulness of the Comprehensive Centers, the
Secretary of Education should direct the Assistant Secretary Matters for

Congressional Consideration

Recommendations for Executive Action

Page 39 GAO- 02- 190 Education Research

of Elementary and Secondary Education and the Assistant Secretary of
Bilingual Education and Minority Languages Affairs to

 design future evaluations of the Comprehensive Centers to produce findings
pertaining to individual Comprehensive Centers.

We provided a draft of this report to the Department of Education for
comment. Education?s Executive Secretariat confirmed by e- mail that
Department officials had reviewed the draft and had no comments except for a
few technical clarifications regarding funding.

We are sending copies of this report to the Secretary of Education, the
appropriate congressional offices, and other interested parties. If you have
any questions, please contact me at (202) 512- 7015 or Eleanor L. Johnson
(202) 512- 7209. Other contributors can be found in appendix I.

Marnie S. Shaul Director, Education, Workforce,

and Income Security Issues Agency Comments

Appendix I: GAO Contacts and Staff Acknowledgments

Page 40 GAO- 02- 190 Education Research

Eleanor L. Johnson (202) 512- 7209 Kathleen D. White (202) 512- 8512

In addition to those named above, Malcolm Drewery, Jr., Tahra N. Edwards,
Richard B. Kelley, and Sarah Moorhead made key contributions to this report.
Appendix I: GAO Contacts and Staff

Acknowledgments GAO Contacts Acknowledgments

(102010)

The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO?s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO?s Web site (www. gao. gov) contains abstracts and full- text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
site daily. The list contains links to the full- text document files. To
have GAO e- mail this list to you every afternoon, go to www. gao. gov and
select "Subscribe to daily e- mail alert for newly released products" under
the GAO Reports heading.

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U. S. General Accounting Office P. O. Box 37050 Washington, D. C. 20013

To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061

GAO Building Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)
Washington, D. C. 20013

Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail:
fraudnet@ gao. gov, or 1- 800- 424- 5454 or (202) 512- 7470 (automated
answering system).

Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S.
General Accounting Office, 441 G. Street NW, Room 7149, Washington, D. C.
20548 GAO?s Mission

Obtaining Copies of GAO Reports and Testimony

Order by Mail or Phone Visit GAO?s Document Distribution Center

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs
*** End of document. ***