Education Discretionary Grants: Awards Process Could Benefit From
Additional Improvements (Letter Report, 03/30/2000, GAO/HEHS-00-55).

Pursuant to a congressional request, GAO reviewed the Department of
Education's redesigned discretionary grants process, focusing on: (1)
Education's progress in implementing the redesigned awards process; (2)
the management controls used to help ensure a fair peer review process
and the costs of peer review; and (3) the extent to which grant awards
are consistent with the results of peer review and the legislation
governing individual grant programs.

GAO noted that: (1) Education's redesigned grant awards process has
helped the agency improve the timeliness of awards and provide better
service to applicants and grantees; (2) as a result of the redesign,
Education provides a greater percentage of new awards earlier in the
year, which allows grantees more time to prepare for implementing their
projects; (3) also, Education has made grant information more available
and accessible to applicants and increased grantees' flexibility to make
certain administrative changes to their projects without prior approval;
(4) Education's peer review process, which in fiscal year 1998 cost
approximately $2.4 million for the nine programs GAO reviewed, has many
of the management controls necessary to help ensure that reviewers
fairly assess the merit of applications; (5) for example, Education
seeks to select peer reviewers who have relevant experience, requires
them to certify they are free of actual or perceived financial conflicts
of interest, and trains them to apply evaluation criteria to score the
merit of grant applications; (6) however, weaknesses in some management
controls could raise concerns about the fairness or perceived fairness
of the process; (7) for example, Education generally does not require
reviewers to certify that they have no nonfinancial conflicts of
interest (that is, personal and professional conflicts) and it lacks
guidance to assist officials in dismissing poorly performing peer
reviewers; (8) moreover, few of the programs assessed the effectiveness
of management controls to ensure that the process was conducted fairly;
(9) for example, although several programs collected information about
the peer review process, few analyzed peer reviewers' feedback on their
review experiences or tracked the performance of reviewers; (10) also,
none of the programs assessed the effect that some peer reviewers'
scoring variations had on applicants' chances to receive funding; (11)
grant awards for the programs GAO reviewed were generally consistent
with peer review scoring, with funds typically awarded to applicants
with the highest peer review scores; (12) awards also were consistent
with legislative objectives for each program and specific measurable
objectives, where specified; (13) about half the grantees in one of
these programs and about one-third in the other had 5 to 20 times the
national average number of students with limited English proficiency.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-00-55
     TITLE:  Education Discretionary Grants: Awards Process Could
	     Benefit From Additional Improvements
      DATE:  03/30/2000
   SUBJECT:  Internal controls
	     Grant award procedures
	     Research grants
	     Conflict of interests
	     Discretionary grants
	     Educational grants
IDENTIFIER:  National Performance Review

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Testimony.                                               **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************

GAO/HEHS-00-55

Appendix I: Scope and Methodology

22

Appendix II: Analysis of Grantee Characteristics for Specific
Programs

30

Appendix III: Comments From the Department of Education

36

Appendix IV: GAO Contacts and Staff Acknowledgements

42

Table 1: Discretionary Programs Reviewed 7

Table 2: Goals of Education's Redesign Process and Actions Taken 9

Table 3: Fiscal Year 1998 Discretionary Grant Data for Selected
Programs 23

Table 4: Results of Panel Score Analysis on One Program 27

Table 5: Results of Panel Score Analysis Across Programs 28

Table 6: Number and Percentage of Grantees in Office of Bilingual Education
and Minority Languages Affairs Programs, by
Percentage of LEP Students 30

Table 7: Number and Percentage of Grantees by Percentage of
Children in Poverty 31

Table 8: Distribution of Grantees by Percentage of Children in
Poverty by Urban Status 32

Table 9: Number and Percentage of Grantees by Per-Pupil
Expenditure 33

Table 10: Distribution of Grantees by Per-Pupil Expenditure
Controlling for Urban Status 34

Table 11: Percentage of Grant Applicants by Urban Status 35

CMSA consolidated metropolitan statistical area

GAPS Grant Administration and Payment System

LEA local education agencies

LEP limited English proficient

MSA metropolitan statistical area

OBEMLA Office of Bilingual Education and Minority Languages Affairs

OMB Office of Management and Budget

Health, Education, and
Human Services Division

B-282710

March 30, 2000

The Honorable Peter Hoekstra
Chairman, Subcommittee on
Oversight and Investigations
Committee on Education and
the Workforce
House of Representatives

Dear Mr. Chairman:

The Department of Education (Education) annually provides billions of
dollars in grants to state and local education agencies, school districts,
colleges and universities, and other organizations to conduct various
program and research activities. These funds are distributed
noncompetitively, through formula grants based on certain applicant
characteristics; and competitively, through discretionary grants based on
the Secretary's judgment. In fiscal year 1999, the Congress appropriated
about $1.7 billion for use in 88 discretionary grant programs that made new
awards during the fiscal year. Education uses the results of peer reviews,
in which a set of experts evaluates the merit of applications, to assist in
making awards decisions.

In the past, applicants expressed concern that the awards process was
time-consuming and burdensome. According to a September 1993 National
Performance Review report, Education estimated that the awards process
required 26 weeks, with only 6 to 8 weeks of that time available for
applicants to prepare applications.1 Applicants expressed frustration with
the limited time allowed to file applications, and grant recipients
(grantees) complained that they had limited time to prepare plans to
implement their projects after receiving their awards. To respond to these
concerns, in December 1995, Education undertook a major effort to totally
redesign its grant awards process. Among other actions, Education
decentralized its grant-making activities, giving authority for this
function to its six principal offices, and streamlined the process. In April
1999, because of your concern that Education's grant awards process was
still too time-consuming and burdensome, you asked that we provide
information on (1) Education's progress in implementing the redesigned
awards process, (2) the management controls used to help ensure a fair peer
review process and the costs of peer review, and (3) the extent to which
grant awards are consistent with the results of peer review and the
legislation governing individual grant programs.

To address each objective, we interviewed Education officials and reviewed
agency documents. We reviewed management controls for peer review using a
framework based on potentially vulnerable areas identified in our previous
work.2 We also reviewed Education's management controls for the peer review
process against government standards that establish criteria for assessing
the effectiveness of such controls. We selected nine discretionary grant
programs that made awards in fiscal year 1998 in four of Education's six
principal offices to obtain a general sense of how the peer review process
is conducted. These programs accounted for about 15 percent of the agency's
61 discretionary grant programs that made awards in fiscal year 1998 and
almost 43 percent of the approximately $696 million in new awards in fiscal
year 1998. We also compared agency awards data with peer review scores and
the legislative objectives of each program. App. I provides a more detailed
discussion of our scope and methodology and information about the nine
programs; app. II provides an analysis of grantee characteristics for
specific programs. We conducted our work between May 1999 and February 2000
in accordance with generally accepted government auditing standards.

Education's redesigned grant awards process has helped the agency improve
the timeliness of awards and provide better service to applicants and
grantees. Prior to the redesign, the process included several repetitive
steps and review layers, which impaired the agency's ability to award funds
to grantees in a timely manner. As a result of the redesign, Education
provides a greater percentage of new awards earlier in the year, which
allows grantees more time to prepare for implementing their projects. For
example, in fiscal year 1999, Education awarded 57 percent of its new awards
by the end of May, compared to 12 percent for the same time period prior to
the redesign. Also, Education has made grant information more available and
accessible to applicants and increased grantees' flexibility to make certain
administrative changes to their projects without prior approval. Education
is considering making additional changes to the process, which could
increase efficiency and reduce costs. However, these changes could present
tradeoffs. For example, one proposal that would allow programs to hold
separate competitions for inexperienced applicants could be viewed by other
applicants as unfair because it could reduce the amount of funding that
would otherwise be available for the general competition.

Education's peer review process, which in fiscal year 1998 cost
approximately $2.4 million (or about 41 percent of the funds authorized for
peer review) for the nine programs we reviewed, has many of the management
controls necessary to help ensure that reviewers fairly assess the merit of
applications. For example, Education seeks to select peer reviewers who have
relevant experience, requires them to certify they are free of actual or
perceived financial conflicts of interest, and trains them to apply
evaluation criteria to score the merit of grant applications. However,
weaknesses in some management controls could raise concerns about the
fairness or perceived fairness of the process. For example, Education
generally does not require reviewers to certify that they have no
nonfinancial conflicts of interest (that is, personal and professional
conflicts) and it lacks guidance to assist officials in dismissing poorly
performing peer reviewers. Moreover, few of the programs assessed the
effectiveness of management controls to ensure that the process was
conducted fairly. For example, although several programs collected
information about the peer review process, few analyzed peer reviewers'
feedback on their review experiences or tracked the performance of
reviewers. Also, none of the programs assessed the effect that some peer
reviewers' scoring variations had on applicants' chances to receive funding.
We make recommendations to Education to enhance and better assess the
effectiveness of its controls.

Grant awards for the programs we reviewed were generally consistent with
peer review scoring, with funds typically awarded to applicants with the
highest peer review scores. Awards also were consistent with legislative
objectives for each program and specific measurable objectives, where
specified. For example, the objective of two programs was to improve
schoolwide programs for limited English-proficient students at schools
serving these students. About half the grantees in one of these programs and
about one-third in the other had 5 to 20 times the national average number
of students with limited English proficiency.

The majority of Education's $38.4 billion fiscal year 1999 total budget
funded formula grants, which award funds noncompetitively to applicants who
meet certain program requirements. For example, one such program provides
grants to states to fund local adult education programs and literacy
services. Education also distributed approximately $1.7 billion in
competitive financial assistance, in the form of discretionary grants to
eligible grantees, which included local education agencies, colleges and
universities, and other applicants.3 Education's six principal offices are
responsible for administering these discretionary grant programs. These
offices are the Office of Bilingual Education and Minority Languages
Affairs, the Office of Educational Research and Improvement, the Office of
Elementary and Secondary Education, the Office of Postsecondary Education,
the Office of Special Education and Rehabilitative Services, and the Office
of Vocational and Adult Education. Table 1 lists the nine programs we
reviewed, by principal office. Within each principal office, there are
multiple program offices which administer the agency's discretionary grant
programs on a daily basis. The program office is responsible for the entire
grant awards process, which consists of publishing the grant awards
announcement, coordinating peer reviews to assess the merit of applications,
making funding recommendations, and awarding grants to successful
applicants. To fund these peer review activities, Education generally can
use up to 1 percent of program appropriations.

Table 1: Discretionary Programs Reviewed

 Principal office                   Programs
                                    Bilingual Education Comprehensive
 Office of Bilingual Education and  School Grants
 Minority Languages Affairs
                                    Program Development Implementation
                                    Grants
                                    21st Century Community Learning Centers

                                    Fund for the Improvement of Education:
 Office of Educational Research and State Partnerships for Character
 Improvement                        Education

                                    Technology Innovation Challenge Grant
                                    Program
                                    Safe and Drug Free Schools and
                                    Communities: Model Demonstration
 Office of Elementary and Secondary Programs to Create Safe and Orderly
 Education                          Learning Environments in Schools

                                    Magnet Schools Assistance
                                    Strengthening Institutions Program
 Office of Postsecondary Education
                                    Talent Search Program

Peer review, the foundation of Education's discretionary awards process,
seeks to identify high-quality projects most worthy of funding. Each grant
program has a separate peer review competition in which individuals, largely
from outside the government, review and score applications, usually at a
central review location. The number of individuals selected to serve as peer
reviewers depends on the number of applications received for a specific
grant awards competition. These peer reviewers are usually grouped together
in panels of three or more members to review applications. Therefore, the
number of peer reviewers selected determines the number of panels that will
be used for each program competition. Each peer reviewer independently reads
and scores a group of applications randomly assigned to the panel, generally
using a numerical scoring system, against program criteria based on
legislative and regulatory requirements. After this, the reviewers convene
in their respective panels to discuss each reviewer's scores and the
strengths and weaknesses of each application. Reviewers may modify their
original scores based on the panel discussion, but are not required to agree
on a common score.

During the peer review process, program officials review all material for
accuracy and completeness and, when the review is completed, conduct a final
review of all materials. Program officials develop a single score for each
application--usually by averaging the scores of all peer reviewers on

the panel that reviewed the application or, less frequently, using a
statistical technique to equalize unusual scoring variances among reviewers.
Program officials may add points to the score if an applicant meets certain
program priorities, such as being in an empowerment zone or enterprise
community. The program office uses the final score to develop a rank-ordered
listing of applications reviewed by all panels, from the highest-scoring to
the lowest-scoring application. Once the rank order list is completed, each
office develops a cutoff point on the list, based on the appropriations for
the program. Next, each program office performs a cost analysis on those
applications that scored high enough to be considered for funding to ensure
that the costs of an applicant's budget are justified. After the cost
analysis is completed, the program office creates a list of applications
that it recommends for funding and includes recommended funding levels. This
list is forwarded for approval, usually to the senior management level. Once
it is decided who will receive funding, awards are made.

Education has achieved many of the goals of its redesigned discretionary
grant awards process, which were to increase the timeliness of awards,
streamline the process, and increase access to and availability of grant
information, among other improvements. In addition, Education has proposed
to make further changes to the process that could offer several advantages,
but could also present tradeoffs. For example, one proposal that would allow
programs to hold separate competitions just for inexperienced applicants
could reduce the amount of funding for the general competition.

Before October 1996, Education administered its discretionary grant awards
process through a 487-step process that took about 6 months from the time
Education publicized the awards to the time grantees received the funding.
The process involved numerous duplicative administrative steps and review
layers, which left Education staff little time to address applicant and
grantee concerns. In December 1995, Education began to redesign the process
to improve the timeliness of awards and improve Education staffs',
applicants', and grantees' satisfaction with the process. Some of the major
changes resulting from the redesign include eliminating unnecessary and
time-consuming processes, decentralizing the grants-making function and
providing a single point of contact for grants administration and
programmatic information, and establishing partnerships with grantees to
help ensure successful project outcomes. In order to accomplish these goals,
Education reported in 1995 that it would reduce the 487-step process to 221
steps. During our review, however, Education officials reported that,
because they had moved to a more results-oriented approach, the focus on
individual steps was no longer relevant. Ultimately, however, Education
reported that it has to date reduced the number of steps to 192. Education
officials stated that many of the steps in the redesigned process are
different from those in the original process and some are actually optional.
Table 2 summarizes the six specific goals of the redesign, the actions
taken, and the resulting changes.

Table 2: Goals of Education's Redesign Process and Actions Taken

            Redesign goal                   Action taken and results
                                      Reduced the number of steps in the
                                      awards process to the current total
                                      of 192,

                                      Reduced the time needed for Office of
                                      Management and Budget (OMB) clearance
 Reduce unnecessary, burdensome, and  and approval of application packages
 time-consuming administrative steps  used by applicants when submitting
 and approval layers, which included  grant proposals from 120 days to 60
 a commitment to reduce the 487-step  days or 10 days, depending on the
 process to 221 steps                 criteria used for the discretionary
                                      grant competition,a and

                                      Trimmed about 30 days from the
                                      approval process for Technical Review
                                      Plans, which describe the procedures
                                      used to review applications.

 Reduce the time needed to make new   Awarded 57 percent of the new fiscal
 discretionary grant awards,          year 1999 awards by the end of May,
 including a specific target of       compared to 12 percent for the same
 making 48 percent of new fiscal year time period before the redesign;
 1999 awards by the end of May        earlier awards provide grantees with
                                      more time to implement their projects
                                      Education decentralized the
 Establish a single point of contact  grants-making function from its
 ("one-stop shopping") for both       Grants Division to its six principal
 programmatic and grants management   offices.
 information to better serve
 potential applicants and grantees    Grant teams in each office serve as a
                                      single point of contact for
                                      applicants and grantees.
                                      Through a website, grantees can have
                                      immediate access to grant
                                      information, and applicants can
                                      identify funding opportunities and
                                      retrieve grant applications.

 Increase the availability of and     Education automated the payment
 access to grant information          process for discretionary grants
                                      through its Grants Administration and
                                      Payment System (GAPS) to provide
                                      grantees with timely access to
                                      information about their grants and
                                      payments and to help program offices
                                      to track grantee expenditure
                                      activity.
                                      Through its expanded authorities for
                                      discretionary grants, Education
 Provide grantees with more           allows grantees a one-time project
 flexibility to manage their projects extension of up to 1 year, without
                                      prior approval, among other
                                      flexibilities.

 Establish partnerships with grantees Education now holds postaward
 to facilitate successful project     conferences with some grantees on
 outcomes                             complex or large projects to agree on
                                      expected project outcomes.

aThe OMB clearance process has been reduced to 60 days for competitions in
which programs use their own program-specific criteria to run the
competition for discretionary grants. The process for programs that use
existing Education criteria has been reduced to 10 days.

A preliminary independent survey of applicants and grantees, after these
changes were implemented, indicates that the majority of them are more
satisfied with various phases of the grant process, such as the availability
of information about grants and the application process.4 Education staff
report that the increased timeliness in making grant awards has also allowed
more time to address applicant and grantee concerns.

Education is considering proposals to amend its regulations that govern
discretionary grant programs to provide greater opportunity for
inexperienced applicants to compete for funding and increase the options
available for reviewing and selecting grants. According to Education
officials, these proposed changes offer additional options for reviewing and
selecting grants and are not mandatory for use in any Education program.
Before implementing these proposals, Education officials stated that they
will publish a Notice of Proposed Rulemaking in the Federal Register and
request public comment. Prompted by recent statutory requirements to provide
for the optional use and acceptance of electronic documents, such as the
Government Paperwork Elimination Act of 1998 (P.L. 105-227), Education is
also considering initiatives to automate the application and review process.
Implementing these changes could offer certain benefits, but could also have
drawbacks. Education is considering the following changes:

ï¿½ Giving special consideration to grant applicants who have never received a
grant or who have not had an active discretionary grant from the federal
government in the 5-year period prior to the application deadline. Education
had previously considered allowing programs to set aside 10 percent of their
funds for inexperienced applicants but determined that such a threshold
would not serve the best interests of programs. It is now considering
allowing programs to hold a separate competition just for inexperienced
applicants or giving preference to these applicants in the general
competition by adding points to their scores. This would address
inexperienced applicants' perception that they are at a disadvantage in
competing with experienced applicants for funding. On the other hand,
allowing programs to hold separate competitions for inexperienced applicants
could reduce the amount of funding that would otherwise be available for
experienced applicants to receive.

ï¿½ Offering reviewers the option of rating the merits of applications using a
qualitative description ranging from excellent to unacceptable, rather than
a numerical score. Under this proposal, applications of comparable merit
would be grouped together in quality bands, thereby allowing Education to
select any of the applications in the band. Selecting projects in this
manner, however, may lead to a less defensible justification for
applications that were not selected because of the increased discretion in
drawing distinctions between applications. While Education officials believe
that using a qualitative rating that relies on detailed reviewer comments
would have more than adequate justification, the greater degree of
discretion could make it more difficult to explain why some applications in
the same quality band were selected over others.

ï¿½ Providing applicants with the opportunity to file applications on-line to
Education's database via the agency's website, which could expedite the
applications process by eliminating the need for Education to manually log
applications and send acknowledgement letters to applicants through the
mail.5 However, applicants are concerned that data could be lost or changed
during transmission and that the transmission of the application could be
delayed, causing them to miss the application deadline.

ï¿½ Providing programs with the option of conducting peer reviews on-line,
which could eliminate staff time spent on peer review logistics, such as
renting hotel space and the costs of physically convening reviewers in a
central location to review applications. Under this initiative, Education
would send applications to peer reviewers electronically, which would allow
reviewers to review applications at their homes or offices. Education
officials believe that electronic reviews would give Education staff greater
capability to monitor reviewers performance. However, conducting reviews
on-line could make the process vulnerable to system failures and delays.
Additionally, on-line reviews could change the nature of panel deliberations
that are key to this process. Further, for competitions with large numbers
of reviewers, it could be a challenge for Education to effectively monitor
the activities and discussions of reviewers, given the large volume of
information exchanged.

Could Be Improved

Education's peer review process includes many of the management controls
needed to help ensure that applications are reviewed fairly. The nine
programs we reviewed conducted their peer reviews similarly and program
officials reported that they spent $2.4 million to conduct their peer
reviews in fiscal year 1998. To be fair, a peer review process must have
several management controls, such as measures to ensure that reviewers
possess relevant expertise and are unbiased in their reviews of grant
applications. In addition, government standards require measures to be in
place to assess the effectiveness of existing controls. However, weaknesses
in controls in some areas raise concerns about the actual or perceived
fairness of the process. For example, Education generally does not require
that peer reviewers certify that they are free of nonfinancial conflicts of
interest and does not have guidance on how to handle poorly performing peer
reviewers and what to do with their scores. Also, Education rarely assesses
information from the peer review process to ensure that controls are
properly applied.

Based on a framework for judging the adequacy of the peer review process
that we adapted from our earlier work, a fair peer review process must have
controls in several key areas, which include selecting reviewers with
relevant expertise, assessing peer reviewers' conflicts of interest,
training reviewers to appropriately apply program criteria when evaluating
applications, and exercising appropriate oversight to ensure that
applications are properly evaluated. We found that Education has management
controls in place to help ensure fair reviews of discretionary grant
applications. In addition to key controls, government auditing standards
establish the need for management controls to measure, report, and monitor
program performance. For peer review, we believe this means assessing
existing controls to ensure that the peer review process is conducted
fairly. Further, Education's own guidance instructs program officials to
annually conduct a review of their process for selecting and using peer
reviewers to ensure that qualified reviewers are used and that the review of
all applications is conducted efficiently, competitively, and fairly.
However, Education has not applied its controls as prescribed by the
auditing standards or by its own internal guidance.

We found that the nine programs selected outside reviewers with expertise
relevant to the grant award competition they served. In our review of
selected peer reviewer resumes, we found that reviewers had backgrounds and
training relevant to the competition. For example, we found that reviewers
with backgrounds in drug and social violence prevention and youth
development activities reviewed applications for a program whose purpose is
to provide safe and drug-free environments for students.

To help ensure that applications are fairly reviewed, Education requires
that reviewers certify that they have no actual or perceived financial
conflicts of interests, in accordance with Education's Departmental
Directive, dated June 23, 1992. According to the directive, a reviewer has a
conflict of interest or the appearance of a conflict of interest if, for
example, the reviewer, his/her spouse, minor child, partner, or organization
would receive or potentially receive financial benefit from a discretionary
grant award. We reviewed the conflict-of-interest statements that reviewers
were required to sign for eight programs and found the reviewers' had
certified that they had no financial conflicts.6 Programs use a variety of
methods to avoid potential financial conflicts of interest, such as asking
reviewers' to identify conflicts when applications are assigned to them and
not allowing reviewers to read applications from states in which they live
or work. If conflicts of interest are discovered during the course of a
review, a program may choose not to use the reviewer, reassign the
application or applications with which a reviewer has a conflict to another
panel, or request a conflict-of-interest waiver. Conflict-of-interest
waivers may be granted on an individual or group basis, when it is
determined that reviewers' financial interests are not so substantial that
they would affect the integrity of reviewers' services. These waivers are
approved by senior principal office officials, with the concurrence of
Education's Office of General Counsel. Eight reviewers reported conflicts of
interest in two programs. For the three reviewers in one program, officials
reassigned the applications to other panels. In the other program, the five
reviewers served under an approved conflict-of-interest waiver.

To further provide for fair peer reviews, Education trains staff and peer
reviewers on various aspects of the peer review process. Staff receive
training on the criteria that peer reviewers use to score applications, how
to review peer reviewers' comments, and other topics. Peer reviewers receive
training during orientation sessions in various areas, such as how to score
applications against program criteria and how to document the justifications
for their scores.

Finally, Education has several ways to help ensure that peer reviewers
correctly apply scoring criteria. For example, program staff periodically
sit in on panel discussions to ensure that peer reviewers consider the
appropriate criteria when evaluating applications. Also, Education staff
check peer reviewers' individual scoring sheets to ensure that their scores
have no math errors, adhere to program criteria, and sufficiently document
the strengths and weaknesses of the application. If the rationale for peer
reviewers' scores is not sufficiently documented, some program officials ask
reviewers to provide additional written comments on the strengths and
weaknesses of the applications. According to Education, unsuccessful
applicants and grantees have both stressed the importance of receiving
thoughtful, substantive reviewer comments that identify the strengths and
weaknesses of the proposed projects. According to Education, unsuccessful
applicants have said that weaknesses cited by reviewers help them strengthen
their applications for the next competition. Education also reported that
grantees have said that receiving substantive reviewer comments before the
start of their grants provides them with useful information to improve the
implementation of the projects.

We identified weaknesses in several key controls that may raise concerns
about the fairness or perceived fairness of the process. These weaknesses
include limitations in Education's efforts to fully assess reviewers'
potential nonfinancial conflicts of interest, absence of guidance for
replacing poorly performing peer reviewers, and insufficient efforts to
assess the effectiveness of existing management controls to ensure fairness
in the process. We found that Education's guidance for making
conflict-of-interest determinations does not clearly require that program
officials assess for nonfinancial conflicts--for example, personal and
professional relationships--which could also impair their objectivity, and
that conflict-of-interest certifications reviewers are required to sign, in
most cases, only attest to the absence of reviewers' financial conflicts.
Nonfinancial conflict's can include such situations as a close friendship
between the reviewer and applicant. A February 1999 Education study of one
principal office's peer review process recommended that the office's
conflict-of-interest determinations go beyond financial interests because of
findings that some peer reviewers had prior professional relationships with
applicants whose applications they were reviewing.7 The study reported that
such relationships could predispose reviewers to judge applicants based on
those prior relationships, rather than solely on the merit of the
application. To preserve the integrity of the process, it is important that
Education require reviewers to fully disclose all potential conflicts that
could affect their reviews.

Additionally, Education has no guidance to assist program officials in
determining situations in which a peer reviewer and his/her scores should be
dismissed due to poor performance or for documenting such situations and the
reasons for any actions taken. The two programs we reviewed that encountered
this situation used different approaches to address the problem. One program
threw out the reviewer's scores for one application and allowed another
panel to score it. The other program used a replacement reviewer to read and
rescore the applications, which changed some applicants' scores as well as
who received awards. In both situations, there was no documentation to
support the actions taken. Without clear guidance, program officials may not
be able to apply a consistent approach to deal with and document these
situations, which can leave Education vulnerable to criticisms that
applicants are not treated fairly.

While many programs collected information from the peer review process, they
rarely analyzed or tracked the information to identify weaknesses in the
process. For example, seven of the nine programs we reviewed collected peer
reviewers' feedback on their review experiences, but only two analyzed the
information to determine areas where improvement might be needed. Analyzing
peer reviewers' feedback on their experiences could provide programs with
valuable information to identify weaknesses in training and other areas that
could potentially affect reviewers' ability to perform adequate reviews. For
example, some reviewers for one program that did not analyze peer reviewers'
feedback expressed concern that not enough time was allowed to complete
their reviews. In addition, five of the nine programs documented information
on peer reviewer performance, but only two programs tracked the information
to flag peer reviewers whose performance was not satisfactory.

Finally, Education recognizes that some reviewers and panels may have a
tendency to score applications higher or lower than others. However, it has
done little to assess the extent to which this occurs and how such scoring
tendencies could affect applicants' chances to receive funding and affect
the actual or perceived fairness of the process. In the absence of these
assessments, we analyzed peer review scoring to determine if there were
patterns of variation among panels. We found differences in the level of
scores received by applications that had been reviewed by different panels.
Applications reviewed by some panels consistently received scores that would
probably be considered high, while applications reviewed by other panels
consistently received much lower scores. The scores received by applications
in one of the eight programs we reviewed illustrate this. The review process
for that program involved 16 panels, with 15 panels each reviewing 8
applications and 1 panel reviewing 7 applications; the maximum score an
application could receive was 145 points. One panel gave each of its eight
applications a score of 138 or higher. By comparison, three other panels
each scored only one of their applications that high and one panel did not
score any of its applications that high (see table 4 in app. I). This
suggests that some panels might have been more generous in their scoring
than others. Because applications are randomly assigned to reviewers, it
would be expected that if all reviewers had applied scoring criteria in the
same way, the applications reviewed by all panels would have had similar
distributions of high, medium, and low scores.

An application's score is a major determinant of its ranking, which serves
as the basis for funding decisions. Therefore, application scores play a
major role in determining which applications will be funded. Table 4 in app.
I illustrates how the score an application receives affects its chances to
receive funding. For example, in the program discussed here (identified as
program #1 in the table), all eight of the application's reviewed by the
panel that assigned the highest scores were at or above the funding cutoff
score; of the eight applications reviewed by each of the three panels that
assigned no more than one score of 138 or higher, only one was at or above
the funding cutoff score. We define the funding cutoff score as the score of
the lowest-scoring application selected for an award. It should be noted,
however, that a score at or above the cutoff level does not necessarily
guarantee funding. For example, one of the programs we reviewed selected
only a certain number of its highest-scoring applications from each panel
for grants.

To compensate for the tendency of some reviewers to score applications
higher or lower than others, Education encourages program officials to use
statistical standardization. Statistical standardization is a statistical
procedure used to correct the effect of differing reviewer approaches to
scoring. However, this procedure can only be used when certain criteria are
met, such as a large competition. While Education reported that this
procedure has been validated as sound by its expert statistician, program
officials told us they often do not have enough applications to meet the
requirements for standardization and, when they have used the procedure in
the past, it has resulted in the funding of mediocre applications over
higher-quality ones. However, Education officials said that standardization
is generally not used alone and may be used in conjunction with other
devices to further eliminate scoring variances across applications.

Program officials for the nine programs we reviewed reported that they spent
$2.4 million to conduct these peer review activities, or about 41 percent of
the $5.8 million authorized in fiscal year 1998 for peer review. Eight of
the programs are authorized to use up to 1 percent of program appropriations
to pay the fees and expenses for outside review activities, while one
program is limited to one-half percent of program appropriations for such
expenses. The programs used their funds to pay for allowable expenditures
such as honorariums ranging from $100 to $125 per day,8 and reviewers'
travel expenses. These funds were typically not set aside in distinct
accounts, but remained in the general program appropriation for programs to
draw from as needed. Some programs reported that they used remaining funds
to provide additional awards and to fund outreach efforts to increase the
visibility of programs, such as state directors' meetings and project
directors' meetings.

Reflect Legislative Objectives

Based on the programs we reviewed, Education typically made grant awards
that were consistent with the results of peer review scoring. Moreover, the
grant awards generally reflected the legislative objectives for the nine
programs we reviewed. Also, in those cases where the programs' legislation
specified that grantees have certain characteristics, we found that the
grantees generally possessed the characteristic specified.

Results

For the programs we reviewed, Education's funding decisions for grant awards
generally were consistent with the results of peer review scoring, with
awards predominantly given to applicants who received the highest peer
review scores. In four of the nine programs, peer review scores alone
determined who would receive funding. In these programs, officials used the
average score for each application to rank order the applications from high
to low and developed a list of those applicants recommended for funding
(including the size of the award). This meant that, in one program with more
than 300 applicants, the highest-scoring 68 applicants were recommended to
receive funding. Senior management officials usually reviewed these lists
and, in all instances, approved the recommendations.

In the remaining five programs we reviewed, applications that met certain
program priorities were awarded additional points. For example, one program
added points to the scores of applicants who had prior experience with the
program. Then applications were ranked from the highest-to lowest-scoring
and a list of applications was recommended for funding (including size of
the award) and approved, usually by senior management officials. While the
addition of priority points may have changed the rank order of applications,
peer review scores still played a key role. We found that at least 70
percent of the award recipients in three of the four programs would have
received awards even without the additional points based on their average
peer review scores. In the fourth program, applicants were not only awarded
priority points, but program officials were also allowed to select certain
applicants over others in order to provide geographic distribution (in
accordance with program objectives) in the awarding of grants. Even in this
case, about 98 percent of the applicants who received the highest peer
review scores received funding. In the fifth program, grant awards were made
in two allocations. In the initial allocation, 98 percent of applicants
receiving awards had the highest peer review scores. In the second
allocation, the program added 10 points to the score of applications that
were not recommended for funding in the initial allocation and that had not
received a grant in the previous award cycle and then reranked the
applications. The program made awards using criteria specified in their
guidance, which resulted in 85 percent of applicants with the highest
adjusted peer review scores receiving awards. Because peer review scores are
a major determinant in awards decisions, the need for strong and effective
management controls and assessment efforts to ensure that applications
receive a fair review is evident.

The grant awards we reviewed were generally consistent with the programs'
overall legislative objectives. We reviewed how Education translated the
legislative objectives into program requirements in (1) the application
announcement, which outlines the type of projects being solicited and who is
eligible to apply; and (2) scoring criteria, which explain how peer
reviewers should evaluate each application. We found that Education
adequately translated the legislative objectives into requirements in the
grant announcement and scoring criteria for each of the nine programs we
reviewed. For example, one program's objectives were to enhance the nation's
efforts to prevent the illegal use of drugs and violence among, and promote
safety and discipline for, students at all educational levels. The
application materials for this grant stated that applications should
"comprehensively address multiple factors that predispose youth to drug use
and violent behavior," and the scoring criteria emphasized the quality and
significance of proposed projects. If Education requires applicants to
adhere to the guidance provided to them in the application package, and peer
reviewers apply the scoring criteria, peer review scores will likely reflect
legislative objectives. If awards are based on these scores and priority
points, they too will reflect legislative objectives.

We found that grantees for the three programs for which the legislation
specified that recipients have certain characteristics generally possessed
the characteristic specified. The legislation governing two programs
specified that grantees develop or upgrade programs for limited English
proficient students for schools serving these students. About half the
awardees in one program and about one-third in the other had 5 to 20 times
the national average of students with limited English proficiency. The
legislation governing the other program specified that applicants were to be
a consortium composed of at least one local education agency (LEA) with a
high percentage of children living below the poverty line. We found that
nearly half the recipients had a greater percentage of children living in
poverty than the national average.9 App. II provides detailed information on
the characteristics of grantees and, to a lesser extent, about the
characteristics of applicants.

The peer review process is a crucial component in determining which
applications are awarded grants. Peer reviewers' scoring of grant
applications plays a major role in helping Education officials decide which
applications are most worthy of funding. While Education has many of the
management controls needed to provide for a fair peer review process,
weaknesses exist that could affect the actual or perceived fairness of the
process. Finally, Education rarely analyzes peer reviewers' feedback, tracks
peer reviewers' performance, or assesses the implications of variations in
average scores among review panels, all of which could improve the peer
review process. Greater efforts in these areas could help Education to
identify problems and improve the peer review process.

To improve the peer review process, we recommend that the Secretary of
Education take the following actions:

ï¿½ amend Education's directive dealing with conflict of interest and
applicable peer reviewer certifications to include a statement certifying
that peer reviewers do not have nonfinancial conflicts that could impair
their objectivity;

ï¿½ develop written policy on the protocol that program officials must follow
to identify and document actions taken when a peer reviewer must be
dismissed due to poor performance; and

ï¿½ amend Education's directive concerning discretionary grant planning,
review, and award procedures to specify that peer reviewer feedback be
collected and analyzed, peer reviewers' performance be documented and
tracked, and scoring variations among panels be assessed to determine the
effectiveness of peer review management controls.

Education provided comments to this report, which are reproduced in app.
III. Technical comments and clarifications were incorporated as appropriate.
In general, Education reported that it appreciated the recognition of the
improvements that it has made in the discretionary grant process and that
our report provides a good overview of those improvements. Education agreed
with all of the recommendations we made to improve the peer review process
and stated that it has already developed new policy guidance, which will be
issued soon. For example, it has amended the peer review certification forms
to include examples of conflicts of interest that arise from personal,
professional, and other nonfinancial relationships. It also plans to require
program offices to establish specific procedures for replacing reviewers and
documenting why such actions were taken. Finally, it will issue interim
guidance to ensure that effective management controls are used in the review
process, including a process for assessing and documenting instances
involving wide differences in panels' scoring of applications.

Copies of this report are being sent to the Honorable Richard W. Riley,
Secretary of Education, and other interested congressional committees. We
will also make copies available to others upon request.

If you have questions about this report, please call me on (202) 512-7215.
Other contacts and staff acknowledgments are listed in app. IV.

Sincerely yours,

Marnie S. Shaul
Associate Director, Education, Workforce,
and Income Security Issues

Scope and Methodology

This appendix discusses in detail our scope and methodology for determining
the progress that the Department of Education (Education) has made in
implementing its redesigned discretionary grants award process, the
management controls used to ensure fairness in the peer review process and
the costs of peer review, and the extent to which grant awards were made
according to peer review scores and legislative objectives.

In examining Education's progress in implementing the redesigned awards
process, we focused on department-wide progress in accomplishing specific
redesign goals. We also reviewed nine discretionary grant programs that
awarded funds in fiscal year 1998, to examine management controls for the
peer review process, the cost to conduct peer review activities, and the
extent to which grant awards were consistent with peer review scores and
legislative objectives. We selected these programs to cover a range of
intended recipients, such as local education agencies (LEAs), state
education agencies, institutions of higher education, and nonprofit
organizations. We included programs that administer larger competitions as
evidenced by the number of applicants, number of recipients, and dollar
amounts awarded. And we included smaller competitions with fewer applicants
and recipients and smaller dollar amounts awarded to see if there were
differences in how the peer review process was administered in programs of
varying sizes.

The nine programs represented 15 percent of the agency's 61 discretionary
grant programs making new awards in fiscal year 1998 and almost 43 percent
of the approximate $696 million in new awards made in fiscal year 1998. In
conducting our review, we did not independently verify agency data; however,
much of the data we received came from the Grants Administration and Payment
System, which Education uses to administer its grants program. Table 3
contains information regarding the legislative objectives of each of the
nine programs, the total amount awarded, average award size, number of
applicants, and number of grantees.

Table 3: Fiscal Year 1998 Discretionary Grant Data for Selected Programs

Continued

                                                                Number (%)
        Program         Total amount   Average     Number of     receiving
                          awarded      award     applications
                                                                  grants
 Technology Innovation
 Challenge Grant        $30,764,165  $1,538,208 328            20 (6)
 Program CFDA 84.303A
 Objective: Primarily
 to promote the use of
 technology to support
 school reform, network
 and telecommunication
 connections to improve
 student learning, and
 professional
 development in the
 integration of
 high-quality
 technology into the
 school curriculum
 21st Century Community
 Learning Centers
                        $39,550,001  $399,495   1,998          99 (5)
 CFDA 84.287A
 Objective: To enable
 rural and inner-city
 public elementary and
 secondary schools or a
 consortium of such
 schools to plan,
 implement, or expand
 projects that benefit
 the educational,
 health, social
 service, cultural, and
 recreational needs of
 their community
 Magnet Schools
 Assistance CFDA        $96,500,000  $1,663,793 129            58 (45)
 84.165A
 Objective: To provide
 grants to eligible
 LEAs or consortia of
 LEAs for use in magnet
 schools that are part
 of approved
 desegregation plans
 and that are designed
 to bring together
 students from
 different social,
 economic, racial, and
 ethnic backgrounds
 Bilingual Education
 Comprehensive School
 Grants                 $16,504,250  $261,972   401            63 (16)

 CFDA 84.290U
 Objective: To develop
 schoolwide programs
 for limited English
 proficient (LEP)
 students that reform,
 restructure, and
 upgrade all relevant
 programs and
 operations within an
 individual school that
 has a concentration of
 LEP students
 Program Development
 Implementation Grants  $5,790,825   $160,856   255            36 (14)
 CFDA 84.288S
 Objective: To develop
 and implement new
 comprehensive,
 coherent, and
 successful bilingual
 education or special
 alternative
 instructional programs
 for LEP students,
 including programs of
 early childhood
 education,
 kindergarten through
 12th-grade education,
 gifted and talented
 education, and
 vocational and applied
 technology education;
 to improve the
 education of LEP
 students and their
 families by
 implementing family
 education programs and
 parent outreach and
 training activities
 designed to assist
 parents to become
 active participants in
 the education of their
 children; to improve
 the instructional
 program by
 identifying,
 acquiring, and
 updating curriculum,
 instructional and
 educational software,
 and assessment
 procedures; to
 compensate personnel,
 including teacher
 aides who have been
 specifically trained,
 or are being trained,
 to provide services;
 and to provide
 tutorials and academic
 career counseling for
 LEP children and youth
 Fund for the
 Improvement of
 Education: State
 Partnerships for       $2,705,475   $270,547   35             10 (29)
 Character Education

 CFDA 84.215V
 Objective: To conduct
 nationally significant
 programs to improve
 the quality of
 education, assist all
 students to meet
 challenging state
 content standards, and
 contribute to the
 achievement of the
 National Education
 Goals
 Safe and Drug Free
 Schools and
 Communities: Model
 Demonstration Programs
 to Create Safe &
 Orderly Learning       $4,156,452   $692,724   98             6 (6)
 Environments in
 Schools

 CFDA 84.184J
 Objective: To enhance
 the nation's efforts
 to prevent the illegal
 use of drugs and
 violence among, and
 promote safety and
 discipline for,
 students at all
 educational levels
 Talent Search Program
                        $86,978,248  $265,989   763            327 (43)
 CFDA 84.044A
 Objective: To identify
 disadvantaged youths
 with potential for
 postsecondary
 education; to
 encourage them in
 continuing in and
 graduating from
 secondary school and
 in enrolling in
 programs of
 postsecondary
 education; to
 publicize the
 availability of
 student financial aid;
 and to increase the
 number of secondary
 and postsecondary
 school dropouts who
 reenter an educational
 program
 Strengthening
 Institutions Program
                        $19,174,811  $281,983   366            68 (19)
 CFDA 84.031A
 Objective: To help
 eligible colleges and
 universities to
 strengthen their
 management and fiscal
 operations and to
 assist such
 institutions to plan,
 develop, or implement
 activities including
 endowment building
 that promise to
 strengthen the
 academic quality of
 their institution

Note: Data are for new grants only.

Source: U.S. General Services Administration, Catalog of Federal Domestic
Assistance and Department of Education-Grant Administration and Payment
System, http://www.cfda.gov (cited Oct. 28, 1999).

To determine Education's progress in implementing its redesigned
discretionary grant awards process, we interviewed department officials and
reviewed agency documentation on the redesigned awards process.
Specifically, we reviewed the agency's progress in streamlining the process
and reducing the time to award grants, as well as the agency's progress in
achieving other redesign goals. In addition, we reviewed and discussed with
agency officials further efforts to change the process, such as expanding
options for reviewing and selecting applications for funding.

To determine the extent to which peer review management controls were in
place to help ensure fair grant awards decisions and the costs to conduct
outside peer review activities, we interviewed department officials and
reviewed department documentation on the use of peer review in making grant
awards decisions in the nine discretionary grant programs. One program was
unable to locate copies of the conflict-of-interest statements signed by
peer reviewers. In our previous work, we found that when certain elements
were missing, the fairness of a peer review process could be compromised.10
Based on that, we developed a framework of elements that are needed to
ensure a fair process: specifically, that reviewers (1) have relevant
expertise, (2) have no conflicts of interest, and (3) apply the appropriate
criteria when making assessments. In addition to the framework, we used the
Government Auditing Standards to assess whether Education's management
controls included a system for measuring, reporting, and monitoring the
performance of the peer review process. We also reviewed Education's
guidance for assessing the peer review process. We specifically discussed
with agency officials the process for selecting reviewers, the training
provided to peer reviewers and staff, the criteria used to assess grant
applications, and how peer reviewer scores are used in making awards
decisions.

To determine if there were patterns of variation among panels for each grant
competition we analyzed the results of peer review scoring. We began by
calculating the average application score for each panel. Then we calculated
the funding cutoff score for each program, which we defined as the
application score received by the lowest-scoring applicant chosen for
funding. Table 4 displays the detailed findings from one of the programs we
analyzed, numbers in bold representing scores of applications that scored at
or above the funding cutoff level. The funding cutoff score, in all cases,
does not specifically separate funded from nonfunded applications, as some
programs subsequently added points to the panel score because the
application met a program priority. We then grouped the panels into thirds
based on application scores, as shown in table 5. The three groups were (1)
panels with the highest scores, (2) panels with the middle scores, (3) and
panels with the lowest scores. Finally, we calculated the percentage of
applications in each of the three groups that scored at or above the funding
cutoff level. These percentages, along with similar findings from the other
programs, are shown in table 5.

Table 4: Results of Panel Score Analysis on One Program

 Panel                                      Score of applications assigned to each panel

      Top-scoring Second-highest Third-highest Fourth-highestFifth-highest Sixth-highest Seventh-highest Eighth-highest
                  scoring        scoring       scoring       scoring       scoring       scoring         scoring       Average
      application
                  application    application   application   application   application   application     application
 Highest one-third scoring panels
 A    145         145            144           144           141           141           139             138           142
 B    145         144            142           141           139           137           135             130           139
 C    145         143            143           142           139           136           101             none          136
 D    145         144            144           141           141           140           136             64            132
 E    143         139            137           137           135           128           118             117           132
 Middle one-third scoring panels
 F    143         141            135           131           128           119           111             106           127
 G    145         145            145           144           144           143           120             22            126
 H    144         142            139           135           129           118           100             91            125
 I    143         134            133           133           129           122           106             89            124
 J    142         137            134           129           118           111           110             102           123
 K    145         144            141           138           138           122           82              52            120
 Lowest one-third scoring panels
 L    144         137            131           127           119           118           102             48            116
 M    142         134            133           132           128           106           74              72            115
 N    143         136            134           108           97            93            86              76            109
 O    142         139            139           132           126           111           51              25            108
 P    137         136            136           134           121           99            51              24            105

Note: Score is in bold if it is at or above funding cutoff score.

Table 5: Results of Panel Score Analysis Across Programs

                                         Percentage of applications scored
                                           at or above the funding cutoff
                                                    score among
                     Range in
          Number     panels'    Funding   Highest      Middle      Lowest
 Programa   of       average     cutoff  one-third   one-third   one-third
          panels   application   score    scoring     scoring     scoring
                      score             panels (%)  panels (%)  panels (%)
 1        16      105-142       136.7   74          40          20
 2        7       39-62         87.7    23          12          5
 3        80b     71-99         84c     92          79          59
 4        59      68-98         95.3    37          14          11
 5        3       78-80         84.3    40          30          33
 6        44      29-89         94.67d  20          6           5
 7e       56      0-67          N/A     43          33          17
 8        26      29-83         89.0f   25          18          9

aWe did not analyze the one program in our review that standardized peer
review scores.

bWe excluded 11 panels that scored 12 or fewer applications.

cThis score does not contain the priority points that were awarded by
program officials after peer reviewers completed scoring. The score of 84
was the lowest panel score received by the applicant with the lowest score
that was awarded a grant.

dThis score does not contain the priority points that were awarded by
program officials after peer reviewers completed scoring. The score of 94.67
was the lowest panel score received by the applicant with the lowest score
who was awarded a grant.

eData presented are based on the results from the first round of scoring in
a multitiered selection process. Range scores represent panels with the
lowest and highest percentage of applications selected for a second round of
scoring. The "Percentage of applications scored..." column represents the
portion of applications in each group that were forwarded to the second
round of scoring.

fThis score does not contain the priority points that were awarded by
program officials after peer reviewers completed scoring. The score of 89.0
was the lowest panel score received by the applicant with the lowest score
who was awarded a grant.

We obtained information on the cost of conducting peer review activities
through discussions with program officials and a review of program
documentation. We discussed with program officials how they used the money
allowed for peer review and reviewed applicable legislation guiding the uses
of peer review funds to determine if funds were spent appropriately.

To determine the extent to which grant awards were consistent with peer
review scores, we compared the scores with awards decisions. We discussed
any discrepancies with program officials. We also reviewed the legislative
objectives for each program in authorizing legislation and relevant
regulations. We compared the legislative objectives to information contained
in each program's grant competition announcement proposal and the Technical
Review Plans. To provide information on the urban status of grantees and
applicants, we used GAPS data provided to us by Education and matched it
with Education Common Core of Data files to get applicants' metropolitan
statistical area (MSA) or consolidated metropolitan statistical area (CMSA)
codes. If the matching did not produce an MSA or CMSA code, we used the
National Public School District Locator available on the National Center for
Education Statistics website (http:www.ed.gov/ccdweb/school/index.asp) to
find the missing information. We classified applicants and grantees as urban
if they were located in an MSA or CMSA. Applicants who did not fall into
either category were classified as nonurban. Although we provide information
on the urban status of grantees and applicants, not all programs include a
legislative objective that emphasizes urban applicants.

We conducted our work between May 1999 and February 2000 in accordance with
generally accepted government auditing standards.

Analysis of Grantee Characteristics for Specific Programs

To provide general information about grantees, we focused on those five
programs which targeted LEAs as recipients. Doing so allowed us to provide
information about the grantees, such as per-pupil expenditure and percentage
of children in poverty, using readily available information.11 In the other
four programs, the grants were targeted primarily to state education
agencies, institutions of higher education, or nonprofit organizations for
which such information was not available.

Table 6 provides information about the grantees' LEP population for two
programs in the Office of Bilingual Education and Minority Language Affairs.
Tables 7 and 8 provide information regarding grantees' percentage of
children in poverty by program and urban status. Tables 9 and 10 detail
grantees' per-pupil expenditures by program and urban status. Finally, table
11 provides information regarding the urban status of all grant applicants
for each of the five programs.12

Table 6: Number and Percentage of Grantees in Office of Bilingual Education
and Minority Languages Affairs Programs, by Percentage of LEP Students

 Percentages Shown in Parentheses
                          Less
 Program        Total     than  2-4.99% 5-9.99%10-19.99% 20-49% 50-100% Missing
                grantees                                                data
                          2%
 Bilingual
 Education                6
 Comprehensive  63        (10)  13 (21) 22 (35)9 (14)    2 (3)  0 (0)   11 (17)
 School Grants
 Program
 Development              11
 Implementation 36        (31)  6 (17)  9 (25) 2 (6)     0 (0)  0 (0)   8 (22)
 Grants

Note: National average of LEP students is 1.1%. Information on LEP status is
derived from 1990 Census data. Percentages may not equal 100 due to
rounding.

Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).

Table 7: Number and Percentage of Grantees by Percentage of Children in
Poverty

 Percentages Shown in Parentheses
                                                                       Overall
 Program        Total     0-10% 11-20% 21-30%31-40% 41-50% 51%+ Missingaverage
                grantees                                        data   poverty
                                                                       rate
 Magnet Schools           7     10     31                  1
 Assistance     58        (12)  (17)   (53)  8 (14) 0 (0)  (2)  1 (2)  23.8
 Program
 Development              6            14                  1
 Implementation 36        (17)  3 (8)  (39)  4 (11) 0 (0)  (3)  8 (22) 24.2
 Grants
 Bilingual
 Education                      16     18    10            4
 Comprehensive  63        4 (6) (25)   (29)  (16)   0 (0)  (6)  11(17) 26.5
 School Grants
 21st Century
 Community                      27     30    11            5
 Learning       99        6 (6) (27)   (30)  (11)   4 (4)  (5)  16 (16)25.6
 Centers
 Technology
 Innovation               3                                0
 Challenge Grant20        (15)  1 (5)  5 (25)2 (10) 2 (10) (0)  7 35)  26.8
 Program

Percentages may not equal 100 due to rounding.

Note: National average of children in poverty is 16.7%. Information
regarding percentage of children in poverty is derived from 1990 Census
data.

Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).

Table 8: Distribution of Grantees by Percentage of Children in Poverty by
Urban Status

 Percentages Shown in Parentheses
                                                                        Average
 Program        Total     0-10% 11-20% 21-30%31-40% 41-50% 51% + Missingpoverty
                grantees                                         data   rate
                                                                        percent
 Magnet Schools
 Assistance

 Urban          57        7     10     30    8 (14) 0 (0)  1 (2) 1 (2)  23.7
                          (12)  (18)   (53)

 Nonurban       1         0 (0) 0 (0)  1     0 (0)  0 (0)  0 (0) 0 (0)  28.5
                                       (100)
 Program
 Development
 Implementation
 Grants

 Urban          28        6     1 (4)  12    2 (7)  0 (0)  1 (4) 6 (21) 23.8
                          (21)         (43)
 Nonurban       8         0 (0) 2 (25) 2 (25)2 (25) 0 (0)  0 (0) 2 (25) 25.8
 Bilingual
 Education
 Comprehensive
 School Grantsa

 Urban          53        4 (8) 16     17    9 (17) 0 (0)  2 (4) 5 (9)  24.9
                                (30)   (32)

 Nonurban       9         0 (0) 0 (0)  1 (11)1 (11) 0 (0)  2     5 (56) 45.1
                                                           (22)
 21st Century
 Community
 Learning
 Centers

 Urban          71        5 (7) 20     28    6 (8)  3 (4)  3 (4) 6 (8)  24.6
                                (28)   (39)
 Nonurban       28        1 (4) 7 (25) 2 (7) 5 (18) 1 (4)  2 (7) 10 (36)29.0
 Technology
 Innovation
 Challenge Grant
 Program

 Urban          12        2     0 (0)  3 (25)1 (8)  1 (8)  0 (0) 5 (42) 26.6
                          (17)

 Nonurban       8         1     1 (13) 2 (25)1 (13) 1 (13) 0 (0) 2 (25) 27.0
                          (13)

Note: National average of children in poverty is 16.7%. We classified
applicants as urban if they were located in a metropolitan statistical area
or a consolidated metropolitan statistical area. Applicants who did not fall
into either area were classified as nonurban. Information regarding
percentage of children in poverty is derived from 1990 Census data.
Percentages may not equal 100 due to rounding.

aOne grantee in the Bilingual Education Comprehensive Schools program could
not be identified as either urban or nonurban.

Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).

Table 9: Number and Percentage of Grantees by Per-Pupil Expenditure

 Percentages Shown in Parentheses

 Program        Total     $1-3,999 $4,000-5,999 $6,000-7,999$8,000-11,999 $12,000+ Missing Average
                grantees                                                           data
 Magnet Schools
 Assistance     58        3 (5)    30 (52)      18 (31)     6 (10)        0 (0)    1 (2)   5,982
 Program
 Development
 Implementation 36        2 (6)    15 (42)      7 (19)      5 (14)        1 (3)    6 (17)  7,022
 Grants
 Bilingual
 Education
 Comprehensive  63        2 (3)    37 (59)      15 (24)     2 (3)         1 (2)    6 (10)  5,609
 Schools Grants
 21st Century
 Community
 Learning       99        8 (8)    56 (57)      20 (20)     8 (8)         0 (0)    7 (7)   5,651
 Centers
 Technology
 Innovation
 Challenge Grant20        3 (15)   5 (25)       4 (20)      0 (0)         1 (5)    7 (35)  5,975
 Program

Note: National average per−pupil expenditure is $5,529. Information is
for 1995-96 school year. Percentages may not equal 100 due to rounding.

Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www/nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).

Table 10: Distribution of Grantees by Per-Pupil Expenditure Controlling for
Urban Status

 Program        Total     $1-3,999 $4,000-5,999 $6,000-7,999$8,000- $12,000 Missing Average
                grantees                                    11,999  +       data
 Magnet Schools
 Assistance
 Urban          57        3 (5)    29 (51)      18 (32)     6 (11)  0 (0)   1 (2)   5,997
 Nonurban       1         0 (0)    1 (100)      0 (0)       0 (0)   0 (0)   0 (0)   5,099
 Program
 Development
 Implementation
 Grants
 Urban          28        1 (4)    12 (43)      7 (25)      3 (11)  1 (4)   4 (14)  6,899
 Nonurban       8         1 (13)   3 (38)       0 (0)       2 (25)  0 (0)   2 (25)  6,343
 Bilingual
 Education
 Comprehensive
 School Grantsa
 Urban          53        2 (4)    31 (58)      15 (28)     2 (4)   1 (2)   2 (4)   5,668
 Nonurban       9         0 (0)    6 (67)       0 (0)       0 (0)   0 (0)   3 (33)  5,108
 21st Century
 Community
 Learning
 Centers
 Urban          71        6 (8)    41 (58)      14 (20)     7 (10)  0 (0)   3 (4)   5,685
 Nonurban       28        2 (7)    15 (54)      6 (21)      1 (4)   0 (0)   4 (14)  5,556
 Technology
 Innovation
 Challenge Grant
 Program
 Urban          12        2 (17)   1 (8)        4 (33)      0 (0)   0 (0)   5 (42)  5,879
 Nonurban       8         1 (13)   4 (50)       0 (0)       0 (0)   1 (13)  2 (25)  6,088

Note: National average per-pupil expenditure is $5,529. Information is for
1995-96 school year. Percentages may not equal 100 due to rounding. We
classified applicants as urban if they were located in a metropolitan
statistical area or a consolidated metropolitan statistical area. Applicants
who did not fall into either area were classified as nonurban.

aOne grantee in the Bilingual Education Comprehensive School Grants program
could not be identified as either urban or nonurban.

Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://wwwnces.ed.gov/edfin/search (cited Oct. 25, 1999).

Table 11: Percentage of Grant Applicants by Urban Status

 Program                                         Urban  Nonurban
 Magnet Schools Assistance
 Successful                                      98     2
 Unsuccessfula                                   93     6
 Program Development Implementation Grants
 Successful                                      78     22
 Unsuccessfulb                                   65     28
 Bilingual Education Comprehensive School Grants
 Successfulc                                     84     14
 Unsuccessfuld                                   70     26
 21st Century Community Learning Centers
 Successful                                      72     28
 Unsuccessfule                                   55     43
 Technology Innovation Challenge Grant Program
 Successful                                      60     40
 Unsuccessfulf                                   66     33

Note: We classified applicants as urban if they were located in a
metropolitan statistical area or consolidated metropolitan statistical area.
Applicants who did not fall into either area were classified as nonurban.

aOne percent of unsuccessful applicants cannot be classified as urban or
nonurban.

bSeven percent of unsuccessful applicants cannot be classified as urban or
nonurban.

cTwo percent of successful applicants cannot be classified as urban or
nonurban.

dFour percent of unsuccessful applicants cannot be classified as urban or
nonurban.

eTwo percent of unsuccessful applicants cannot be classified as urban or
nonurban.

fLess than 1 percent of unsuccessful applicants cannot be classified as
urban or nonurban.

Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data; http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center
website,http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).

Comments From the Department of Education

GAO Contacts and Staff Acknowledgements

Lori Rectanus, Assistant Director, (202) 512-9847
Mary A. Crenshaw, Project Manager, (202) 512-7053

In addition to those name above, George Erhart, Brett Fallavollita, and
Gillian Martin, Senior Evaluators, and Jonathan Barker, Senior Attorney,
Office of General Counsel, made key contributions to this report.

(104970)

Table 1: Discretionary Programs Reviewed 7

Table 2: Goals of Education's Redesign Process and Actions Taken 9

Table 3: Fiscal Year 1998 Discretionary Grant Data for Selected
Programs 23

Table 4: Results of Panel Score Analysis on One Program 27

Table 5: Results of Panel Score Analysis Across Programs 28

Table 6: Number and Percentage of Grantees in Office of Bilingual Education
and Minority Languages Affairs Programs, by
Percentage of LEP Students 30

Table 7: Number and Percentage of Grantees by Percentage of
Children in Poverty 31

Table 8: Distribution of Grantees by Percentage of Children in
Poverty by Urban Status 32

Table 9: Number and Percentage of Grantees by Per-Pupil
Expenditure 33

Table 10: Distribution of Grantees by Per-Pupil Expenditure
Controlling for Urban Status 34

Table 11: Percentage of Grant Applicants by Urban Status 35
  

1. From Red Tape to Results: Creating a Government That Works Better and
Costs Less, report of the National Performance Review, Vice President Al
Gore (Sept. 7, 1993).

2. See Peer Review: Reforms Needed to Ensure Fairness in Federal Agency
Grant Selection (GAO/PEMD-94-1, June 24,1994 ). Potential weaknesses
include, among other things, the fact that reviewers lack relevant expertise
and have a conflict of interest with the applications they review. The
framework and its adaptation for this report are further discussed in app.
I.

3. Discretionary grants also include cooperative agreements which Education
awards when it anticipates having substantial involvement in working with a
grantee to achieve project outcomes.

4. Schulman, Ronca, and Bucuvalas, Inc., Survey of Customer Satisfaction
with the Department of Education Discretionary Grant Process (Draft) (Sept.
27, 1999).

5. As of March 2000, Education had five competitions under way that allowed
applications to be electronically initiated, developed, and submitted.

6. One program was unable to locate reviewer conflict-of-interest
statements.

7. August and Associates, a subcontractor of the American Institutes for
Research, Strengthening the Standards: Recommendations for OERI Peer Review,
Final Report (Feb. 3, 1999).

8. One program paid reviewers a flat rate of $600 for the entire review.

9. Information was available only for the LEA that received the grant.
Because that LEA might not be the one with a high percentage of children
living below the poverty line, we may be underreporting the number of
grantees with high percentages of children in poverty.

10. GAO/PEMD-94-1 .

11. National Center for Education Statistics on the Department of Education
website.

12. In all five programs, if grantees are identified as specific schools
rather than LEAs, we determined what district a school was located in and
then provided information on that district.
*** End of document. ***