Education Discretionary Grants: Awards Process Could Benefit From
Additional Improvements (Letter Report, 03/30/2000, GAO/HEHS-00-55).
Pursuant to a congressional request, GAO reviewed the Department of
Education's redesigned discretionary grants process, focusing on: (1)
Education's progress in implementing the redesigned awards process; (2)
the management controls used to help ensure a fair peer review process
and the costs of peer review; and (3) the extent to which grant awards
are consistent with the results of peer review and the legislation
governing individual grant programs.
GAO noted that: (1) Education's redesigned grant awards process has
helped the agency improve the timeliness of awards and provide better
service to applicants and grantees; (2) as a result of the redesign,
Education provides a greater percentage of new awards earlier in the
year, which allows grantees more time to prepare for implementing their
projects; (3) also, Education has made grant information more available
and accessible to applicants and increased grantees' flexibility to make
certain administrative changes to their projects without prior approval;
(4) Education's peer review process, which in fiscal year 1998 cost
approximately $2.4 million for the nine programs GAO reviewed, has many
of the management controls necessary to help ensure that reviewers
fairly assess the merit of applications; (5) for example, Education
seeks to select peer reviewers who have relevant experience, requires
them to certify they are free of actual or perceived financial conflicts
of interest, and trains them to apply evaluation criteria to score the
merit of grant applications; (6) however, weaknesses in some management
controls could raise concerns about the fairness or perceived fairness
of the process; (7) for example, Education generally does not require
reviewers to certify that they have no nonfinancial conflicts of
interest (that is, personal and professional conflicts) and it lacks
guidance to assist officials in dismissing poorly performing peer
reviewers; (8) moreover, few of the programs assessed the effectiveness
of management controls to ensure that the process was conducted fairly;
(9) for example, although several programs collected information about
the peer review process, few analyzed peer reviewers' feedback on their
review experiences or tracked the performance of reviewers; (10) also,
none of the programs assessed the effect that some peer reviewers'
scoring variations had on applicants' chances to receive funding; (11)
grant awards for the programs GAO reviewed were generally consistent
with peer review scoring, with funds typically awarded to applicants
with the highest peer review scores; (12) awards also were consistent
with legislative objectives for each program and specific measurable
objectives, where specified; (13) about half the grantees in one of
these programs and about one-third in the other had 5 to 20 times the
national average number of students with limited English proficiency.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-00-55
TITLE: Education Discretionary Grants: Awards Process Could
Benefit From Additional Improvements
DATE: 03/30/2000
SUBJECT: Internal controls
Grant award procedures
Research grants
Conflict of interests
Discretionary grants
Educational grants
IDENTIFIER: National Performance Review
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO/HEHS-00-55
Appendix I: Scope and Methodology
22
Appendix II: Analysis of Grantee Characteristics for Specific
Programs
30
Appendix III: Comments From the Department of Education
36
Appendix IV: GAO Contacts and Staff Acknowledgements
42
Table 1: Discretionary Programs Reviewed 7
Table 2: Goals of Education's Redesign Process and Actions Taken 9
Table 3: Fiscal Year 1998 Discretionary Grant Data for Selected
Programs 23
Table 4: Results of Panel Score Analysis on One Program 27
Table 5: Results of Panel Score Analysis Across Programs 28
Table 6: Number and Percentage of Grantees in Office of Bilingual Education
and Minority Languages Affairs Programs, by
Percentage of LEP Students 30
Table 7: Number and Percentage of Grantees by Percentage of
Children in Poverty 31
Table 8: Distribution of Grantees by Percentage of Children in
Poverty by Urban Status 32
Table 9: Number and Percentage of Grantees by Per-Pupil
Expenditure 33
Table 10: Distribution of Grantees by Per-Pupil Expenditure
Controlling for Urban Status 34
Table 11: Percentage of Grant Applicants by Urban Status 35
CMSA consolidated metropolitan statistical area
GAPS Grant Administration and Payment System
LEA local education agencies
LEP limited English proficient
MSA metropolitan statistical area
OBEMLA Office of Bilingual Education and Minority Languages Affairs
OMB Office of Management and Budget
Health, Education, and
Human Services Division
B-282710
March 30, 2000
The Honorable Peter Hoekstra
Chairman, Subcommittee on
Oversight and Investigations
Committee on Education and
the Workforce
House of Representatives
Dear Mr. Chairman:
The Department of Education (Education) annually provides billions of
dollars in grants to state and local education agencies, school districts,
colleges and universities, and other organizations to conduct various
program and research activities. These funds are distributed
noncompetitively, through formula grants based on certain applicant
characteristics; and competitively, through discretionary grants based on
the Secretary's judgment. In fiscal year 1999, the Congress appropriated
about $1.7 billion for use in 88 discretionary grant programs that made new
awards during the fiscal year. Education uses the results of peer reviews,
in which a set of experts evaluates the merit of applications, to assist in
making awards decisions.
In the past, applicants expressed concern that the awards process was
time-consuming and burdensome. According to a September 1993 National
Performance Review report, Education estimated that the awards process
required 26 weeks, with only 6 to 8 weeks of that time available for
applicants to prepare applications.1 Applicants expressed frustration with
the limited time allowed to file applications, and grant recipients
(grantees) complained that they had limited time to prepare plans to
implement their projects after receiving their awards. To respond to these
concerns, in December 1995, Education undertook a major effort to totally
redesign its grant awards process. Among other actions, Education
decentralized its grant-making activities, giving authority for this
function to its six principal offices, and streamlined the process. In April
1999, because of your concern that Education's grant awards process was
still too time-consuming and burdensome, you asked that we provide
information on (1) Education's progress in implementing the redesigned
awards process, (2) the management controls used to help ensure a fair peer
review process and the costs of peer review, and (3) the extent to which
grant awards are consistent with the results of peer review and the
legislation governing individual grant programs.
To address each objective, we interviewed Education officials and reviewed
agency documents. We reviewed management controls for peer review using a
framework based on potentially vulnerable areas identified in our previous
work.2 We also reviewed Education's management controls for the peer review
process against government standards that establish criteria for assessing
the effectiveness of such controls. We selected nine discretionary grant
programs that made awards in fiscal year 1998 in four of Education's six
principal offices to obtain a general sense of how the peer review process
is conducted. These programs accounted for about 15 percent of the agency's
61 discretionary grant programs that made awards in fiscal year 1998 and
almost 43 percent of the approximately $696 million in new awards in fiscal
year 1998. We also compared agency awards data with peer review scores and
the legislative objectives of each program. App. I provides a more detailed
discussion of our scope and methodology and information about the nine
programs; app. II provides an analysis of grantee characteristics for
specific programs. We conducted our work between May 1999 and February 2000
in accordance with generally accepted government auditing standards.
Education's redesigned grant awards process has helped the agency improve
the timeliness of awards and provide better service to applicants and
grantees. Prior to the redesign, the process included several repetitive
steps and review layers, which impaired the agency's ability to award funds
to grantees in a timely manner. As a result of the redesign, Education
provides a greater percentage of new awards earlier in the year, which
allows grantees more time to prepare for implementing their projects. For
example, in fiscal year 1999, Education awarded 57 percent of its new awards
by the end of May, compared to 12 percent for the same time period prior to
the redesign. Also, Education has made grant information more available and
accessible to applicants and increased grantees' flexibility to make certain
administrative changes to their projects without prior approval. Education
is considering making additional changes to the process, which could
increase efficiency and reduce costs. However, these changes could present
tradeoffs. For example, one proposal that would allow programs to hold
separate competitions for inexperienced applicants could be viewed by other
applicants as unfair because it could reduce the amount of funding that
would otherwise be available for the general competition.
Education's peer review process, which in fiscal year 1998 cost
approximately $2.4 million (or about 41 percent of the funds authorized for
peer review) for the nine programs we reviewed, has many of the management
controls necessary to help ensure that reviewers fairly assess the merit of
applications. For example, Education seeks to select peer reviewers who have
relevant experience, requires them to certify they are free of actual or
perceived financial conflicts of interest, and trains them to apply
evaluation criteria to score the merit of grant applications. However,
weaknesses in some management controls could raise concerns about the
fairness or perceived fairness of the process. For example, Education
generally does not require reviewers to certify that they have no
nonfinancial conflicts of interest (that is, personal and professional
conflicts) and it lacks guidance to assist officials in dismissing poorly
performing peer reviewers. Moreover, few of the programs assessed the
effectiveness of management controls to ensure that the process was
conducted fairly. For example, although several programs collected
information about the peer review process, few analyzed peer reviewers'
feedback on their review experiences or tracked the performance of
reviewers. Also, none of the programs assessed the effect that some peer
reviewers' scoring variations had on applicants' chances to receive funding.
We make recommendations to Education to enhance and better assess the
effectiveness of its controls.
Grant awards for the programs we reviewed were generally consistent with
peer review scoring, with funds typically awarded to applicants with the
highest peer review scores. Awards also were consistent with legislative
objectives for each program and specific measurable objectives, where
specified. For example, the objective of two programs was to improve
schoolwide programs for limited English-proficient students at schools
serving these students. About half the grantees in one of these programs and
about one-third in the other had 5 to 20 times the national average number
of students with limited English proficiency.
The majority of Education's $38.4 billion fiscal year 1999 total budget
funded formula grants, which award funds noncompetitively to applicants who
meet certain program requirements. For example, one such program provides
grants to states to fund local adult education programs and literacy
services. Education also distributed approximately $1.7 billion in
competitive financial assistance, in the form of discretionary grants to
eligible grantees, which included local education agencies, colleges and
universities, and other applicants.3 Education's six principal offices are
responsible for administering these discretionary grant programs. These
offices are the Office of Bilingual Education and Minority Languages
Affairs, the Office of Educational Research and Improvement, the Office of
Elementary and Secondary Education, the Office of Postsecondary Education,
the Office of Special Education and Rehabilitative Services, and the Office
of Vocational and Adult Education. Table 1 lists the nine programs we
reviewed, by principal office. Within each principal office, there are
multiple program offices which administer the agency's discretionary grant
programs on a daily basis. The program office is responsible for the entire
grant awards process, which consists of publishing the grant awards
announcement, coordinating peer reviews to assess the merit of applications,
making funding recommendations, and awarding grants to successful
applicants. To fund these peer review activities, Education generally can
use up to 1 percent of program appropriations.
Table 1: Discretionary Programs Reviewed
Principal office Programs
Bilingual Education Comprehensive
Office of Bilingual Education and School Grants
Minority Languages Affairs
Program Development Implementation
Grants
21st Century Community Learning Centers
Fund for the Improvement of Education:
Office of Educational Research and State Partnerships for Character
Improvement Education
Technology Innovation Challenge Grant
Program
Safe and Drug Free Schools and
Communities: Model Demonstration
Office of Elementary and Secondary Programs to Create Safe and Orderly
Education Learning Environments in Schools
Magnet Schools Assistance
Strengthening Institutions Program
Office of Postsecondary Education
Talent Search Program
Peer review, the foundation of Education's discretionary awards process,
seeks to identify high-quality projects most worthy of funding. Each grant
program has a separate peer review competition in which individuals, largely
from outside the government, review and score applications, usually at a
central review location. The number of individuals selected to serve as peer
reviewers depends on the number of applications received for a specific
grant awards competition. These peer reviewers are usually grouped together
in panels of three or more members to review applications. Therefore, the
number of peer reviewers selected determines the number of panels that will
be used for each program competition. Each peer reviewer independently reads
and scores a group of applications randomly assigned to the panel, generally
using a numerical scoring system, against program criteria based on
legislative and regulatory requirements. After this, the reviewers convene
in their respective panels to discuss each reviewer's scores and the
strengths and weaknesses of each application. Reviewers may modify their
original scores based on the panel discussion, but are not required to agree
on a common score.
During the peer review process, program officials review all material for
accuracy and completeness and, when the review is completed, conduct a final
review of all materials. Program officials develop a single score for each
application--usually by averaging the scores of all peer reviewers on
the panel that reviewed the application or, less frequently, using a
statistical technique to equalize unusual scoring variances among reviewers.
Program officials may add points to the score if an applicant meets certain
program priorities, such as being in an empowerment zone or enterprise
community. The program office uses the final score to develop a rank-ordered
listing of applications reviewed by all panels, from the highest-scoring to
the lowest-scoring application. Once the rank order list is completed, each
office develops a cutoff point on the list, based on the appropriations for
the program. Next, each program office performs a cost analysis on those
applications that scored high enough to be considered for funding to ensure
that the costs of an applicant's budget are justified. After the cost
analysis is completed, the program office creates a list of applications
that it recommends for funding and includes recommended funding levels. This
list is forwarded for approval, usually to the senior management level. Once
it is decided who will receive funding, awards are made.
Education has achieved many of the goals of its redesigned discretionary
grant awards process, which were to increase the timeliness of awards,
streamline the process, and increase access to and availability of grant
information, among other improvements. In addition, Education has proposed
to make further changes to the process that could offer several advantages,
but could also present tradeoffs. For example, one proposal that would allow
programs to hold separate competitions just for inexperienced applicants
could reduce the amount of funding for the general competition.
Before October 1996, Education administered its discretionary grant awards
process through a 487-step process that took about 6 months from the time
Education publicized the awards to the time grantees received the funding.
The process involved numerous duplicative administrative steps and review
layers, which left Education staff little time to address applicant and
grantee concerns. In December 1995, Education began to redesign the process
to improve the timeliness of awards and improve Education staffs',
applicants', and grantees' satisfaction with the process. Some of the major
changes resulting from the redesign include eliminating unnecessary and
time-consuming processes, decentralizing the grants-making function and
providing a single point of contact for grants administration and
programmatic information, and establishing partnerships with grantees to
help ensure successful project outcomes. In order to accomplish these goals,
Education reported in 1995 that it would reduce the 487-step process to 221
steps. During our review, however, Education officials reported that,
because they had moved to a more results-oriented approach, the focus on
individual steps was no longer relevant. Ultimately, however, Education
reported that it has to date reduced the number of steps to 192. Education
officials stated that many of the steps in the redesigned process are
different from those in the original process and some are actually optional.
Table 2 summarizes the six specific goals of the redesign, the actions
taken, and the resulting changes.
Table 2: Goals of Education's Redesign Process and Actions Taken
Redesign goal Action taken and results
Reduced the number of steps in the
awards process to the current total
of 192,
Reduced the time needed for Office of
Management and Budget (OMB) clearance
Reduce unnecessary, burdensome, and and approval of application packages
time-consuming administrative steps used by applicants when submitting
and approval layers, which included grant proposals from 120 days to 60
a commitment to reduce the 487-step days or 10 days, depending on the
process to 221 steps criteria used for the discretionary
grant competition,a and
Trimmed about 30 days from the
approval process for Technical Review
Plans, which describe the procedures
used to review applications.
Reduce the time needed to make new Awarded 57 percent of the new fiscal
discretionary grant awards, year 1999 awards by the end of May,
including a specific target of compared to 12 percent for the same
making 48 percent of new fiscal year time period before the redesign;
1999 awards by the end of May earlier awards provide grantees with
more time to implement their projects
Education decentralized the
Establish a single point of contact grants-making function from its
("one-stop shopping") for both Grants Division to its six principal
programmatic and grants management offices.
information to better serve
potential applicants and grantees Grant teams in each office serve as a
single point of contact for
applicants and grantees.
Through a website, grantees can have
immediate access to grant
information, and applicants can
identify funding opportunities and
retrieve grant applications.
Increase the availability of and Education automated the payment
access to grant information process for discretionary grants
through its Grants Administration and
Payment System (GAPS) to provide
grantees with timely access to
information about their grants and
payments and to help program offices
to track grantee expenditure
activity.
Through its expanded authorities for
discretionary grants, Education
Provide grantees with more allows grantees a one-time project
flexibility to manage their projects extension of up to 1 year, without
prior approval, among other
flexibilities.
Establish partnerships with grantees Education now holds postaward
to facilitate successful project conferences with some grantees on
outcomes complex or large projects to agree on
expected project outcomes.
aThe OMB clearance process has been reduced to 60 days for competitions in
which programs use their own program-specific criteria to run the
competition for discretionary grants. The process for programs that use
existing Education criteria has been reduced to 10 days.
A preliminary independent survey of applicants and grantees, after these
changes were implemented, indicates that the majority of them are more
satisfied with various phases of the grant process, such as the availability
of information about grants and the application process.4 Education staff
report that the increased timeliness in making grant awards has also allowed
more time to address applicant and grantee concerns.
Education is considering proposals to amend its regulations that govern
discretionary grant programs to provide greater opportunity for
inexperienced applicants to compete for funding and increase the options
available for reviewing and selecting grants. According to Education
officials, these proposed changes offer additional options for reviewing and
selecting grants and are not mandatory for use in any Education program.
Before implementing these proposals, Education officials stated that they
will publish a Notice of Proposed Rulemaking in the Federal Register and
request public comment. Prompted by recent statutory requirements to provide
for the optional use and acceptance of electronic documents, such as the
Government Paperwork Elimination Act of 1998 (P.L. 105-227), Education is
also considering initiatives to automate the application and review process.
Implementing these changes could offer certain benefits, but could also have
drawbacks. Education is considering the following changes:
� Giving special consideration to grant applicants who have never received a
grant or who have not had an active discretionary grant from the federal
government in the 5-year period prior to the application deadline. Education
had previously considered allowing programs to set aside 10 percent of their
funds for inexperienced applicants but determined that such a threshold
would not serve the best interests of programs. It is now considering
allowing programs to hold a separate competition just for inexperienced
applicants or giving preference to these applicants in the general
competition by adding points to their scores. This would address
inexperienced applicants' perception that they are at a disadvantage in
competing with experienced applicants for funding. On the other hand,
allowing programs to hold separate competitions for inexperienced applicants
could reduce the amount of funding that would otherwise be available for
experienced applicants to receive.
� Offering reviewers the option of rating the merits of applications using a
qualitative description ranging from excellent to unacceptable, rather than
a numerical score. Under this proposal, applications of comparable merit
would be grouped together in quality bands, thereby allowing Education to
select any of the applications in the band. Selecting projects in this
manner, however, may lead to a less defensible justification for
applications that were not selected because of the increased discretion in
drawing distinctions between applications. While Education officials believe
that using a qualitative rating that relies on detailed reviewer comments
would have more than adequate justification, the greater degree of
discretion could make it more difficult to explain why some applications in
the same quality band were selected over others.
� Providing applicants with the opportunity to file applications on-line to
Education's database via the agency's website, which could expedite the
applications process by eliminating the need for Education to manually log
applications and send acknowledgement letters to applicants through the
mail.5 However, applicants are concerned that data could be lost or changed
during transmission and that the transmission of the application could be
delayed, causing them to miss the application deadline.
� Providing programs with the option of conducting peer reviews on-line,
which could eliminate staff time spent on peer review logistics, such as
renting hotel space and the costs of physically convening reviewers in a
central location to review applications. Under this initiative, Education
would send applications to peer reviewers electronically, which would allow
reviewers to review applications at their homes or offices. Education
officials believe that electronic reviews would give Education staff greater
capability to monitor reviewers performance. However, conducting reviews
on-line could make the process vulnerable to system failures and delays.
Additionally, on-line reviews could change the nature of panel deliberations
that are key to this process. Further, for competitions with large numbers
of reviewers, it could be a challenge for Education to effectively monitor
the activities and discussions of reviewers, given the large volume of
information exchanged.
Could Be Improved
Education's peer review process includes many of the management controls
needed to help ensure that applications are reviewed fairly. The nine
programs we reviewed conducted their peer reviews similarly and program
officials reported that they spent $2.4 million to conduct their peer
reviews in fiscal year 1998. To be fair, a peer review process must have
several management controls, such as measures to ensure that reviewers
possess relevant expertise and are unbiased in their reviews of grant
applications. In addition, government standards require measures to be in
place to assess the effectiveness of existing controls. However, weaknesses
in controls in some areas raise concerns about the actual or perceived
fairness of the process. For example, Education generally does not require
that peer reviewers certify that they are free of nonfinancial conflicts of
interest and does not have guidance on how to handle poorly performing peer
reviewers and what to do with their scores. Also, Education rarely assesses
information from the peer review process to ensure that controls are
properly applied.
Based on a framework for judging the adequacy of the peer review process
that we adapted from our earlier work, a fair peer review process must have
controls in several key areas, which include selecting reviewers with
relevant expertise, assessing peer reviewers' conflicts of interest,
training reviewers to appropriately apply program criteria when evaluating
applications, and exercising appropriate oversight to ensure that
applications are properly evaluated. We found that Education has management
controls in place to help ensure fair reviews of discretionary grant
applications. In addition to key controls, government auditing standards
establish the need for management controls to measure, report, and monitor
program performance. For peer review, we believe this means assessing
existing controls to ensure that the peer review process is conducted
fairly. Further, Education's own guidance instructs program officials to
annually conduct a review of their process for selecting and using peer
reviewers to ensure that qualified reviewers are used and that the review of
all applications is conducted efficiently, competitively, and fairly.
However, Education has not applied its controls as prescribed by the
auditing standards or by its own internal guidance.
We found that the nine programs selected outside reviewers with expertise
relevant to the grant award competition they served. In our review of
selected peer reviewer resumes, we found that reviewers had backgrounds and
training relevant to the competition. For example, we found that reviewers
with backgrounds in drug and social violence prevention and youth
development activities reviewed applications for a program whose purpose is
to provide safe and drug-free environments for students.
To help ensure that applications are fairly reviewed, Education requires
that reviewers certify that they have no actual or perceived financial
conflicts of interests, in accordance with Education's Departmental
Directive, dated June 23, 1992. According to the directive, a reviewer has a
conflict of interest or the appearance of a conflict of interest if, for
example, the reviewer, his/her spouse, minor child, partner, or organization
would receive or potentially receive financial benefit from a discretionary
grant award. We reviewed the conflict-of-interest statements that reviewers
were required to sign for eight programs and found the reviewers' had
certified that they had no financial conflicts.6 Programs use a variety of
methods to avoid potential financial conflicts of interest, such as asking
reviewers' to identify conflicts when applications are assigned to them and
not allowing reviewers to read applications from states in which they live
or work. If conflicts of interest are discovered during the course of a
review, a program may choose not to use the reviewer, reassign the
application or applications with which a reviewer has a conflict to another
panel, or request a conflict-of-interest waiver. Conflict-of-interest
waivers may be granted on an individual or group basis, when it is
determined that reviewers' financial interests are not so substantial that
they would affect the integrity of reviewers' services. These waivers are
approved by senior principal office officials, with the concurrence of
Education's Office of General Counsel. Eight reviewers reported conflicts of
interest in two programs. For the three reviewers in one program, officials
reassigned the applications to other panels. In the other program, the five
reviewers served under an approved conflict-of-interest waiver.
To further provide for fair peer reviews, Education trains staff and peer
reviewers on various aspects of the peer review process. Staff receive
training on the criteria that peer reviewers use to score applications, how
to review peer reviewers' comments, and other topics. Peer reviewers receive
training during orientation sessions in various areas, such as how to score
applications against program criteria and how to document the justifications
for their scores.
Finally, Education has several ways to help ensure that peer reviewers
correctly apply scoring criteria. For example, program staff periodically
sit in on panel discussions to ensure that peer reviewers consider the
appropriate criteria when evaluating applications. Also, Education staff
check peer reviewers' individual scoring sheets to ensure that their scores
have no math errors, adhere to program criteria, and sufficiently document
the strengths and weaknesses of the application. If the rationale for peer
reviewers' scores is not sufficiently documented, some program officials ask
reviewers to provide additional written comments on the strengths and
weaknesses of the applications. According to Education, unsuccessful
applicants and grantees have both stressed the importance of receiving
thoughtful, substantive reviewer comments that identify the strengths and
weaknesses of the proposed projects. According to Education, unsuccessful
applicants have said that weaknesses cited by reviewers help them strengthen
their applications for the next competition. Education also reported that
grantees have said that receiving substantive reviewer comments before the
start of their grants provides them with useful information to improve the
implementation of the projects.
We identified weaknesses in several key controls that may raise concerns
about the fairness or perceived fairness of the process. These weaknesses
include limitations in Education's efforts to fully assess reviewers'
potential nonfinancial conflicts of interest, absence of guidance for
replacing poorly performing peer reviewers, and insufficient efforts to
assess the effectiveness of existing management controls to ensure fairness
in the process. We found that Education's guidance for making
conflict-of-interest determinations does not clearly require that program
officials assess for nonfinancial conflicts--for example, personal and
professional relationships--which could also impair their objectivity, and
that conflict-of-interest certifications reviewers are required to sign, in
most cases, only attest to the absence of reviewers' financial conflicts.
Nonfinancial conflict's can include such situations as a close friendship
between the reviewer and applicant. A February 1999 Education study of one
principal office's peer review process recommended that the office's
conflict-of-interest determinations go beyond financial interests because of
findings that some peer reviewers had prior professional relationships with
applicants whose applications they were reviewing.7 The study reported that
such relationships could predispose reviewers to judge applicants based on
those prior relationships, rather than solely on the merit of the
application. To preserve the integrity of the process, it is important that
Education require reviewers to fully disclose all potential conflicts that
could affect their reviews.
Additionally, Education has no guidance to assist program officials in
determining situations in which a peer reviewer and his/her scores should be
dismissed due to poor performance or for documenting such situations and the
reasons for any actions taken. The two programs we reviewed that encountered
this situation used different approaches to address the problem. One program
threw out the reviewer's scores for one application and allowed another
panel to score it. The other program used a replacement reviewer to read and
rescore the applications, which changed some applicants' scores as well as
who received awards. In both situations, there was no documentation to
support the actions taken. Without clear guidance, program officials may not
be able to apply a consistent approach to deal with and document these
situations, which can leave Education vulnerable to criticisms that
applicants are not treated fairly.
While many programs collected information from the peer review process, they
rarely analyzed or tracked the information to identify weaknesses in the
process. For example, seven of the nine programs we reviewed collected peer
reviewers' feedback on their review experiences, but only two analyzed the
information to determine areas where improvement might be needed. Analyzing
peer reviewers' feedback on their experiences could provide programs with
valuable information to identify weaknesses in training and other areas that
could potentially affect reviewers' ability to perform adequate reviews. For
example, some reviewers for one program that did not analyze peer reviewers'
feedback expressed concern that not enough time was allowed to complete
their reviews. In addition, five of the nine programs documented information
on peer reviewer performance, but only two programs tracked the information
to flag peer reviewers whose performance was not satisfactory.
Finally, Education recognizes that some reviewers and panels may have a
tendency to score applications higher or lower than others. However, it has
done little to assess the extent to which this occurs and how such scoring
tendencies could affect applicants' chances to receive funding and affect
the actual or perceived fairness of the process. In the absence of these
assessments, we analyzed peer review scoring to determine if there were
patterns of variation among panels. We found differences in the level of
scores received by applications that had been reviewed by different panels.
Applications reviewed by some panels consistently received scores that would
probably be considered high, while applications reviewed by other panels
consistently received much lower scores. The scores received by applications
in one of the eight programs we reviewed illustrate this. The review process
for that program involved 16 panels, with 15 panels each reviewing 8
applications and 1 panel reviewing 7 applications; the maximum score an
application could receive was 145 points. One panel gave each of its eight
applications a score of 138 or higher. By comparison, three other panels
each scored only one of their applications that high and one panel did not
score any of its applications that high (see table 4 in app. I). This
suggests that some panels might have been more generous in their scoring
than others. Because applications are randomly assigned to reviewers, it
would be expected that if all reviewers had applied scoring criteria in the
same way, the applications reviewed by all panels would have had similar
distributions of high, medium, and low scores.
An application's score is a major determinant of its ranking, which serves
as the basis for funding decisions. Therefore, application scores play a
major role in determining which applications will be funded. Table 4 in app.
I illustrates how the score an application receives affects its chances to
receive funding. For example, in the program discussed here (identified as
program #1 in the table), all eight of the application's reviewed by the
panel that assigned the highest scores were at or above the funding cutoff
score; of the eight applications reviewed by each of the three panels that
assigned no more than one score of 138 or higher, only one was at or above
the funding cutoff score. We define the funding cutoff score as the score of
the lowest-scoring application selected for an award. It should be noted,
however, that a score at or above the cutoff level does not necessarily
guarantee funding. For example, one of the programs we reviewed selected
only a certain number of its highest-scoring applications from each panel
for grants.
To compensate for the tendency of some reviewers to score applications
higher or lower than others, Education encourages program officials to use
statistical standardization. Statistical standardization is a statistical
procedure used to correct the effect of differing reviewer approaches to
scoring. However, this procedure can only be used when certain criteria are
met, such as a large competition. While Education reported that this
procedure has been validated as sound by its expert statistician, program
officials told us they often do not have enough applications to meet the
requirements for standardization and, when they have used the procedure in
the past, it has resulted in the funding of mediocre applications over
higher-quality ones. However, Education officials said that standardization
is generally not used alone and may be used in conjunction with other
devices to further eliminate scoring variances across applications.
Program officials for the nine programs we reviewed reported that they spent
$2.4 million to conduct these peer review activities, or about 41 percent of
the $5.8 million authorized in fiscal year 1998 for peer review. Eight of
the programs are authorized to use up to 1 percent of program appropriations
to pay the fees and expenses for outside review activities, while one
program is limited to one-half percent of program appropriations for such
expenses. The programs used their funds to pay for allowable expenditures
such as honorariums ranging from $100 to $125 per day,8 and reviewers'
travel expenses. These funds were typically not set aside in distinct
accounts, but remained in the general program appropriation for programs to
draw from as needed. Some programs reported that they used remaining funds
to provide additional awards and to fund outreach efforts to increase the
visibility of programs, such as state directors' meetings and project
directors' meetings.
Reflect Legislative Objectives
Based on the programs we reviewed, Education typically made grant awards
that were consistent with the results of peer review scoring. Moreover, the
grant awards generally reflected the legislative objectives for the nine
programs we reviewed. Also, in those cases where the programs' legislation
specified that grantees have certain characteristics, we found that the
grantees generally possessed the characteristic specified.
Results
For the programs we reviewed, Education's funding decisions for grant awards
generally were consistent with the results of peer review scoring, with
awards predominantly given to applicants who received the highest peer
review scores. In four of the nine programs, peer review scores alone
determined who would receive funding. In these programs, officials used the
average score for each application to rank order the applications from high
to low and developed a list of those applicants recommended for funding
(including the size of the award). This meant that, in one program with more
than 300 applicants, the highest-scoring 68 applicants were recommended to
receive funding. Senior management officials usually reviewed these lists
and, in all instances, approved the recommendations.
In the remaining five programs we reviewed, applications that met certain
program priorities were awarded additional points. For example, one program
added points to the scores of applicants who had prior experience with the
program. Then applications were ranked from the highest-to lowest-scoring
and a list of applications was recommended for funding (including size of
the award) and approved, usually by senior management officials. While the
addition of priority points may have changed the rank order of applications,
peer review scores still played a key role. We found that at least 70
percent of the award recipients in three of the four programs would have
received awards even without the additional points based on their average
peer review scores. In the fourth program, applicants were not only awarded
priority points, but program officials were also allowed to select certain
applicants over others in order to provide geographic distribution (in
accordance with program objectives) in the awarding of grants. Even in this
case, about 98 percent of the applicants who received the highest peer
review scores received funding. In the fifth program, grant awards were made
in two allocations. In the initial allocation, 98 percent of applicants
receiving awards had the highest peer review scores. In the second
allocation, the program added 10 points to the score of applications that
were not recommended for funding in the initial allocation and that had not
received a grant in the previous award cycle and then reranked the
applications. The program made awards using criteria specified in their
guidance, which resulted in 85 percent of applicants with the highest
adjusted peer review scores receiving awards. Because peer review scores are
a major determinant in awards decisions, the need for strong and effective
management controls and assessment efforts to ensure that applications
receive a fair review is evident.
The grant awards we reviewed were generally consistent with the programs'
overall legislative objectives. We reviewed how Education translated the
legislative objectives into program requirements in (1) the application
announcement, which outlines the type of projects being solicited and who is
eligible to apply; and (2) scoring criteria, which explain how peer
reviewers should evaluate each application. We found that Education
adequately translated the legislative objectives into requirements in the
grant announcement and scoring criteria for each of the nine programs we
reviewed. For example, one program's objectives were to enhance the nation's
efforts to prevent the illegal use of drugs and violence among, and promote
safety and discipline for, students at all educational levels. The
application materials for this grant stated that applications should
"comprehensively address multiple factors that predispose youth to drug use
and violent behavior," and the scoring criteria emphasized the quality and
significance of proposed projects. If Education requires applicants to
adhere to the guidance provided to them in the application package, and peer
reviewers apply the scoring criteria, peer review scores will likely reflect
legislative objectives. If awards are based on these scores and priority
points, they too will reflect legislative objectives.
We found that grantees for the three programs for which the legislation
specified that recipients have certain characteristics generally possessed
the characteristic specified. The legislation governing two programs
specified that grantees develop or upgrade programs for limited English
proficient students for schools serving these students. About half the
awardees in one program and about one-third in the other had 5 to 20 times
the national average of students with limited English proficiency. The
legislation governing the other program specified that applicants were to be
a consortium composed of at least one local education agency (LEA) with a
high percentage of children living below the poverty line. We found that
nearly half the recipients had a greater percentage of children living in
poverty than the national average.9 App. II provides detailed information on
the characteristics of grantees and, to a lesser extent, about the
characteristics of applicants.
The peer review process is a crucial component in determining which
applications are awarded grants. Peer reviewers' scoring of grant
applications plays a major role in helping Education officials decide which
applications are most worthy of funding. While Education has many of the
management controls needed to provide for a fair peer review process,
weaknesses exist that could affect the actual or perceived fairness of the
process. Finally, Education rarely analyzes peer reviewers' feedback, tracks
peer reviewers' performance, or assesses the implications of variations in
average scores among review panels, all of which could improve the peer
review process. Greater efforts in these areas could help Education to
identify problems and improve the peer review process.
To improve the peer review process, we recommend that the Secretary of
Education take the following actions:
� amend Education's directive dealing with conflict of interest and
applicable peer reviewer certifications to include a statement certifying
that peer reviewers do not have nonfinancial conflicts that could impair
their objectivity;
� develop written policy on the protocol that program officials must follow
to identify and document actions taken when a peer reviewer must be
dismissed due to poor performance; and
� amend Education's directive concerning discretionary grant planning,
review, and award procedures to specify that peer reviewer feedback be
collected and analyzed, peer reviewers' performance be documented and
tracked, and scoring variations among panels be assessed to determine the
effectiveness of peer review management controls.
Education provided comments to this report, which are reproduced in app.
III. Technical comments and clarifications were incorporated as appropriate.
In general, Education reported that it appreciated the recognition of the
improvements that it has made in the discretionary grant process and that
our report provides a good overview of those improvements. Education agreed
with all of the recommendations we made to improve the peer review process
and stated that it has already developed new policy guidance, which will be
issued soon. For example, it has amended the peer review certification forms
to include examples of conflicts of interest that arise from personal,
professional, and other nonfinancial relationships. It also plans to require
program offices to establish specific procedures for replacing reviewers and
documenting why such actions were taken. Finally, it will issue interim
guidance to ensure that effective management controls are used in the review
process, including a process for assessing and documenting instances
involving wide differences in panels' scoring of applications.
Copies of this report are being sent to the Honorable Richard W. Riley,
Secretary of Education, and other interested congressional committees. We
will also make copies available to others upon request.
If you have questions about this report, please call me on (202) 512-7215.
Other contacts and staff acknowledgments are listed in app. IV.
Sincerely yours,
Marnie S. Shaul
Associate Director, Education, Workforce,
and Income Security Issues
Scope and Methodology
This appendix discusses in detail our scope and methodology for determining
the progress that the Department of Education (Education) has made in
implementing its redesigned discretionary grants award process, the
management controls used to ensure fairness in the peer review process and
the costs of peer review, and the extent to which grant awards were made
according to peer review scores and legislative objectives.
In examining Education's progress in implementing the redesigned awards
process, we focused on department-wide progress in accomplishing specific
redesign goals. We also reviewed nine discretionary grant programs that
awarded funds in fiscal year 1998, to examine management controls for the
peer review process, the cost to conduct peer review activities, and the
extent to which grant awards were consistent with peer review scores and
legislative objectives. We selected these programs to cover a range of
intended recipients, such as local education agencies (LEAs), state
education agencies, institutions of higher education, and nonprofit
organizations. We included programs that administer larger competitions as
evidenced by the number of applicants, number of recipients, and dollar
amounts awarded. And we included smaller competitions with fewer applicants
and recipients and smaller dollar amounts awarded to see if there were
differences in how the peer review process was administered in programs of
varying sizes.
The nine programs represented 15 percent of the agency's 61 discretionary
grant programs making new awards in fiscal year 1998 and almost 43 percent
of the approximate $696 million in new awards made in fiscal year 1998. In
conducting our review, we did not independently verify agency data; however,
much of the data we received came from the Grants Administration and Payment
System, which Education uses to administer its grants program. Table 3
contains information regarding the legislative objectives of each of the
nine programs, the total amount awarded, average award size, number of
applicants, and number of grantees.
Table 3: Fiscal Year 1998 Discretionary Grant Data for Selected Programs
Continued
Number (%)
Program Total amount Average Number of receiving
awarded award applications
grants
Technology Innovation
Challenge Grant $30,764,165 $1,538,208 328 20 (6)
Program CFDA 84.303A
Objective: Primarily
to promote the use of
technology to support
school reform, network
and telecommunication
connections to improve
student learning, and
professional
development in the
integration of
high-quality
technology into the
school curriculum
21st Century Community
Learning Centers
$39,550,001 $399,495 1,998 99 (5)
CFDA 84.287A
Objective: To enable
rural and inner-city
public elementary and
secondary schools or a
consortium of such
schools to plan,
implement, or expand
projects that benefit
the educational,
health, social
service, cultural, and
recreational needs of
their community
Magnet Schools
Assistance CFDA $96,500,000 $1,663,793 129 58 (45)
84.165A
Objective: To provide
grants to eligible
LEAs or consortia of
LEAs for use in magnet
schools that are part
of approved
desegregation plans
and that are designed
to bring together
students from
different social,
economic, racial, and
ethnic backgrounds
Bilingual Education
Comprehensive School
Grants $16,504,250 $261,972 401 63 (16)
CFDA 84.290U
Objective: To develop
schoolwide programs
for limited English
proficient (LEP)
students that reform,
restructure, and
upgrade all relevant
programs and
operations within an
individual school that
has a concentration of
LEP students
Program Development
Implementation Grants $5,790,825 $160,856 255 36 (14)
CFDA 84.288S
Objective: To develop
and implement new
comprehensive,
coherent, and
successful bilingual
education or special
alternative
instructional programs
for LEP students,
including programs of
early childhood
education,
kindergarten through
12th-grade education,
gifted and talented
education, and
vocational and applied
technology education;
to improve the
education of LEP
students and their
families by
implementing family
education programs and
parent outreach and
training activities
designed to assist
parents to become
active participants in
the education of their
children; to improve
the instructional
program by
identifying,
acquiring, and
updating curriculum,
instructional and
educational software,
and assessment
procedures; to
compensate personnel,
including teacher
aides who have been
specifically trained,
or are being trained,
to provide services;
and to provide
tutorials and academic
career counseling for
LEP children and youth
Fund for the
Improvement of
Education: State
Partnerships for $2,705,475 $270,547 35 10 (29)
Character Education
CFDA 84.215V
Objective: To conduct
nationally significant
programs to improve
the quality of
education, assist all
students to meet
challenging state
content standards, and
contribute to the
achievement of the
National Education
Goals
Safe and Drug Free
Schools and
Communities: Model
Demonstration Programs
to Create Safe &
Orderly Learning $4,156,452 $692,724 98 6 (6)
Environments in
Schools
CFDA 84.184J
Objective: To enhance
the nation's efforts
to prevent the illegal
use of drugs and
violence among, and
promote safety and
discipline for,
students at all
educational levels
Talent Search Program
$86,978,248 $265,989 763 327 (43)
CFDA 84.044A
Objective: To identify
disadvantaged youths
with potential for
postsecondary
education; to
encourage them in
continuing in and
graduating from
secondary school and
in enrolling in
programs of
postsecondary
education; to
publicize the
availability of
student financial aid;
and to increase the
number of secondary
and postsecondary
school dropouts who
reenter an educational
program
Strengthening
Institutions Program
$19,174,811 $281,983 366 68 (19)
CFDA 84.031A
Objective: To help
eligible colleges and
universities to
strengthen their
management and fiscal
operations and to
assist such
institutions to plan,
develop, or implement
activities including
endowment building
that promise to
strengthen the
academic quality of
their institution
Note: Data are for new grants only.
Source: U.S. General Services Administration, Catalog of Federal Domestic
Assistance and Department of Education-Grant Administration and Payment
System, http://www.cfda.gov (cited Oct. 28, 1999).
To determine Education's progress in implementing its redesigned
discretionary grant awards process, we interviewed department officials and
reviewed agency documentation on the redesigned awards process.
Specifically, we reviewed the agency's progress in streamlining the process
and reducing the time to award grants, as well as the agency's progress in
achieving other redesign goals. In addition, we reviewed and discussed with
agency officials further efforts to change the process, such as expanding
options for reviewing and selecting applications for funding.
To determine the extent to which peer review management controls were in
place to help ensure fair grant awards decisions and the costs to conduct
outside peer review activities, we interviewed department officials and
reviewed department documentation on the use of peer review in making grant
awards decisions in the nine discretionary grant programs. One program was
unable to locate copies of the conflict-of-interest statements signed by
peer reviewers. In our previous work, we found that when certain elements
were missing, the fairness of a peer review process could be compromised.10
Based on that, we developed a framework of elements that are needed to
ensure a fair process: specifically, that reviewers (1) have relevant
expertise, (2) have no conflicts of interest, and (3) apply the appropriate
criteria when making assessments. In addition to the framework, we used the
Government Auditing Standards to assess whether Education's management
controls included a system for measuring, reporting, and monitoring the
performance of the peer review process. We also reviewed Education's
guidance for assessing the peer review process. We specifically discussed
with agency officials the process for selecting reviewers, the training
provided to peer reviewers and staff, the criteria used to assess grant
applications, and how peer reviewer scores are used in making awards
decisions.
To determine if there were patterns of variation among panels for each grant
competition we analyzed the results of peer review scoring. We began by
calculating the average application score for each panel. Then we calculated
the funding cutoff score for each program, which we defined as the
application score received by the lowest-scoring applicant chosen for
funding. Table 4 displays the detailed findings from one of the programs we
analyzed, numbers in bold representing scores of applications that scored at
or above the funding cutoff level. The funding cutoff score, in all cases,
does not specifically separate funded from nonfunded applications, as some
programs subsequently added points to the panel score because the
application met a program priority. We then grouped the panels into thirds
based on application scores, as shown in table 5. The three groups were (1)
panels with the highest scores, (2) panels with the middle scores, (3) and
panels with the lowest scores. Finally, we calculated the percentage of
applications in each of the three groups that scored at or above the funding
cutoff level. These percentages, along with similar findings from the other
programs, are shown in table 5.
Table 4: Results of Panel Score Analysis on One Program
Panel Score of applications assigned to each panel
Top-scoring Second-highest Third-highest Fourth-highestFifth-highest Sixth-highest Seventh-highest Eighth-highest
scoring scoring scoring scoring scoring scoring scoring Average
application
application application application application application application application
Highest one-third scoring panels
A 145 145 144 144 141 141 139 138 142
B 145 144 142 141 139 137 135 130 139
C 145 143 143 142 139 136 101 none 136
D 145 144 144 141 141 140 136 64 132
E 143 139 137 137 135 128 118 117 132
Middle one-third scoring panels
F 143 141 135 131 128 119 111 106 127
G 145 145 145 144 144 143 120 22 126
H 144 142 139 135 129 118 100 91 125
I 143 134 133 133 129 122 106 89 124
J 142 137 134 129 118 111 110 102 123
K 145 144 141 138 138 122 82 52 120
Lowest one-third scoring panels
L 144 137 131 127 119 118 102 48 116
M 142 134 133 132 128 106 74 72 115
N 143 136 134 108 97 93 86 76 109
O 142 139 139 132 126 111 51 25 108
P 137 136 136 134 121 99 51 24 105
Note: Score is in bold if it is at or above funding cutoff score.
Table 5: Results of Panel Score Analysis Across Programs
Percentage of applications scored
at or above the funding cutoff
score among
Range in
Number panels' Funding Highest Middle Lowest
Programa of average cutoff one-third one-third one-third
panels application score scoring scoring scoring
score panels (%) panels (%) panels (%)
1 16 105-142 136.7 74 40 20
2 7 39-62 87.7 23 12 5
3 80b 71-99 84c 92 79 59
4 59 68-98 95.3 37 14 11
5 3 78-80 84.3 40 30 33
6 44 29-89 94.67d 20 6 5
7e 56 0-67 N/A 43 33 17
8 26 29-83 89.0f 25 18 9
aWe did not analyze the one program in our review that standardized peer
review scores.
bWe excluded 11 panels that scored 12 or fewer applications.
cThis score does not contain the priority points that were awarded by
program officials after peer reviewers completed scoring. The score of 84
was the lowest panel score received by the applicant with the lowest score
that was awarded a grant.
dThis score does not contain the priority points that were awarded by
program officials after peer reviewers completed scoring. The score of 94.67
was the lowest panel score received by the applicant with the lowest score
who was awarded a grant.
eData presented are based on the results from the first round of scoring in
a multitiered selection process. Range scores represent panels with the
lowest and highest percentage of applications selected for a second round of
scoring. The "Percentage of applications scored..." column represents the
portion of applications in each group that were forwarded to the second
round of scoring.
fThis score does not contain the priority points that were awarded by
program officials after peer reviewers completed scoring. The score of 89.0
was the lowest panel score received by the applicant with the lowest score
who was awarded a grant.
We obtained information on the cost of conducting peer review activities
through discussions with program officials and a review of program
documentation. We discussed with program officials how they used the money
allowed for peer review and reviewed applicable legislation guiding the uses
of peer review funds to determine if funds were spent appropriately.
To determine the extent to which grant awards were consistent with peer
review scores, we compared the scores with awards decisions. We discussed
any discrepancies with program officials. We also reviewed the legislative
objectives for each program in authorizing legislation and relevant
regulations. We compared the legislative objectives to information contained
in each program's grant competition announcement proposal and the Technical
Review Plans. To provide information on the urban status of grantees and
applicants, we used GAPS data provided to us by Education and matched it
with Education Common Core of Data files to get applicants' metropolitan
statistical area (MSA) or consolidated metropolitan statistical area (CMSA)
codes. If the matching did not produce an MSA or CMSA code, we used the
National Public School District Locator available on the National Center for
Education Statistics website (http:www.ed.gov/ccdweb/school/index.asp) to
find the missing information. We classified applicants and grantees as urban
if they were located in an MSA or CMSA. Applicants who did not fall into
either category were classified as nonurban. Although we provide information
on the urban status of grantees and applicants, not all programs include a
legislative objective that emphasizes urban applicants.
We conducted our work between May 1999 and February 2000 in accordance with
generally accepted government auditing standards.
Analysis of Grantee Characteristics for Specific Programs
To provide general information about grantees, we focused on those five
programs which targeted LEAs as recipients. Doing so allowed us to provide
information about the grantees, such as per-pupil expenditure and percentage
of children in poverty, using readily available information.11 In the other
four programs, the grants were targeted primarily to state education
agencies, institutions of higher education, or nonprofit organizations for
which such information was not available.
Table 6 provides information about the grantees' LEP population for two
programs in the Office of Bilingual Education and Minority Language Affairs.
Tables 7 and 8 provide information regarding grantees' percentage of
children in poverty by program and urban status. Tables 9 and 10 detail
grantees' per-pupil expenditures by program and urban status. Finally, table
11 provides information regarding the urban status of all grant applicants
for each of the five programs.12
Table 6: Number and Percentage of Grantees in Office of Bilingual Education
and Minority Languages Affairs Programs, by Percentage of LEP Students
Percentages Shown in Parentheses
Less
Program Total than 2-4.99% 5-9.99%10-19.99% 20-49% 50-100% Missing
grantees data
2%
Bilingual
Education 6
Comprehensive 63 (10) 13 (21) 22 (35)9 (14) 2 (3) 0 (0) 11 (17)
School Grants
Program
Development 11
Implementation 36 (31) 6 (17) 9 (25) 2 (6) 0 (0) 0 (0) 8 (22)
Grants
Note: National average of LEP students is 1.1%. Information on LEP status is
derived from 1990 Census data. Percentages may not equal 100 due to
rounding.
Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).
Table 7: Number and Percentage of Grantees by Percentage of Children in
Poverty
Percentages Shown in Parentheses
Overall
Program Total 0-10% 11-20% 21-30%31-40% 41-50% 51%+ Missingaverage
grantees data poverty
rate
Magnet Schools 7 10 31 1
Assistance 58 (12) (17) (53) 8 (14) 0 (0) (2) 1 (2) 23.8
Program
Development 6 14 1
Implementation 36 (17) 3 (8) (39) 4 (11) 0 (0) (3) 8 (22) 24.2
Grants
Bilingual
Education 16 18 10 4
Comprehensive 63 4 (6) (25) (29) (16) 0 (0) (6) 11(17) 26.5
School Grants
21st Century
Community 27 30 11 5
Learning 99 6 (6) (27) (30) (11) 4 (4) (5) 16 (16)25.6
Centers
Technology
Innovation 3 0
Challenge Grant20 (15) 1 (5) 5 (25)2 (10) 2 (10) (0) 7 35) 26.8
Program
Percentages may not equal 100 due to rounding.
Note: National average of children in poverty is 16.7%. Information
regarding percentage of children in poverty is derived from 1990 Census
data.
Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).
Table 8: Distribution of Grantees by Percentage of Children in Poverty by
Urban Status
Percentages Shown in Parentheses
Average
Program Total 0-10% 11-20% 21-30%31-40% 41-50% 51% + Missingpoverty
grantees data rate
percent
Magnet Schools
Assistance
Urban 57 7 10 30 8 (14) 0 (0) 1 (2) 1 (2) 23.7
(12) (18) (53)
Nonurban 1 0 (0) 0 (0) 1 0 (0) 0 (0) 0 (0) 0 (0) 28.5
(100)
Program
Development
Implementation
Grants
Urban 28 6 1 (4) 12 2 (7) 0 (0) 1 (4) 6 (21) 23.8
(21) (43)
Nonurban 8 0 (0) 2 (25) 2 (25)2 (25) 0 (0) 0 (0) 2 (25) 25.8
Bilingual
Education
Comprehensive
School Grantsa
Urban 53 4 (8) 16 17 9 (17) 0 (0) 2 (4) 5 (9) 24.9
(30) (32)
Nonurban 9 0 (0) 0 (0) 1 (11)1 (11) 0 (0) 2 5 (56) 45.1
(22)
21st Century
Community
Learning
Centers
Urban 71 5 (7) 20 28 6 (8) 3 (4) 3 (4) 6 (8) 24.6
(28) (39)
Nonurban 28 1 (4) 7 (25) 2 (7) 5 (18) 1 (4) 2 (7) 10 (36)29.0
Technology
Innovation
Challenge Grant
Program
Urban 12 2 0 (0) 3 (25)1 (8) 1 (8) 0 (0) 5 (42) 26.6
(17)
Nonurban 8 1 1 (13) 2 (25)1 (13) 1 (13) 0 (0) 2 (25) 27.0
(13)
Note: National average of children in poverty is 16.7%. We classified
applicants as urban if they were located in a metropolitan statistical area
or a consolidated metropolitan statistical area. Applicants who did not fall
into either area were classified as nonurban. Information regarding
percentage of children in poverty is derived from 1990 Census data.
Percentages may not equal 100 due to rounding.
aOne grantee in the Bilingual Education Comprehensive Schools program could
not be identified as either urban or nonurban.
Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).
Table 9: Number and Percentage of Grantees by Per-Pupil Expenditure
Percentages Shown in Parentheses
Program Total $1-3,999 $4,000-5,999 $6,000-7,999$8,000-11,999 $12,000+ Missing Average
grantees data
Magnet Schools
Assistance 58 3 (5) 30 (52) 18 (31) 6 (10) 0 (0) 1 (2) 5,982
Program
Development
Implementation 36 2 (6) 15 (42) 7 (19) 5 (14) 1 (3) 6 (17) 7,022
Grants
Bilingual
Education
Comprehensive 63 2 (3) 37 (59) 15 (24) 2 (3) 1 (2) 6 (10) 5,609
Schools Grants
21st Century
Community
Learning 99 8 (8) 56 (57) 20 (20) 8 (8) 0 (0) 7 (7) 5,651
Centers
Technology
Innovation
Challenge Grant20 3 (15) 5 (25) 4 (20) 0 (0) 1 (5) 7 (35) 5,975
Program
Note: National average per−pupil expenditure is $5,529. Information is
for 1995-96 school year. Percentages may not equal 100 due to rounding.
Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www/nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).
Table 10: Distribution of Grantees by Per-Pupil Expenditure Controlling for
Urban Status
Program Total $1-3,999 $4,000-5,999 $6,000-7,999$8,000- $12,000 Missing Average
grantees 11,999 + data
Magnet Schools
Assistance
Urban 57 3 (5) 29 (51) 18 (32) 6 (11) 0 (0) 1 (2) 5,997
Nonurban 1 0 (0) 1 (100) 0 (0) 0 (0) 0 (0) 0 (0) 5,099
Program
Development
Implementation
Grants
Urban 28 1 (4) 12 (43) 7 (25) 3 (11) 1 (4) 4 (14) 6,899
Nonurban 8 1 (13) 3 (38) 0 (0) 2 (25) 0 (0) 2 (25) 6,343
Bilingual
Education
Comprehensive
School Grantsa
Urban 53 2 (4) 31 (58) 15 (28) 2 (4) 1 (2) 2 (4) 5,668
Nonurban 9 0 (0) 6 (67) 0 (0) 0 (0) 0 (0) 3 (33) 5,108
21st Century
Community
Learning
Centers
Urban 71 6 (8) 41 (58) 14 (20) 7 (10) 0 (0) 3 (4) 5,685
Nonurban 28 2 (7) 15 (54) 6 (21) 1 (4) 0 (0) 4 (14) 5,556
Technology
Innovation
Challenge Grant
Program
Urban 12 2 (17) 1 (8) 4 (33) 0 (0) 0 (0) 5 (42) 5,879
Nonurban 8 1 (13) 4 (50) 0 (0) 0 (0) 1 (13) 2 (25) 6,088
Note: National average per-pupil expenditure is $5,529. Information is for
1995-96 school year. Percentages may not equal 100 due to rounding. We
classified applicants as urban if they were located in a metropolitan
statistical area or a consolidated metropolitan statistical area. Applicants
who did not fall into either area were classified as nonurban.
aOne grantee in the Bilingual Education Comprehensive School Grants program
could not be identified as either urban or nonurban.
Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data, http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center website,
http://wwwnces.ed.gov/edfin/search (cited Oct. 25, 1999).
Table 11: Percentage of Grant Applicants by Urban Status
Program Urban Nonurban
Magnet Schools Assistance
Successful 98 2
Unsuccessfula 93 6
Program Development Implementation Grants
Successful 78 22
Unsuccessfulb 65 28
Bilingual Education Comprehensive School Grants
Successfulc 84 14
Unsuccessfuld 70 26
21st Century Community Learning Centers
Successful 72 28
Unsuccessfule 55 43
Technology Innovation Challenge Grant Program
Successful 60 40
Unsuccessfulf 66 33
Note: We classified applicants as urban if they were located in a
metropolitan statistical area or consolidated metropolitan statistical area.
Applicants who did not fall into either area were classified as nonurban.
aOne percent of unsuccessful applicants cannot be classified as urban or
nonurban.
bSeven percent of unsuccessful applicants cannot be classified as urban or
nonurban.
cTwo percent of successful applicants cannot be classified as urban or
nonurban.
dFour percent of unsuccessful applicants cannot be classified as urban or
nonurban.
eTwo percent of unsuccessful applicants cannot be classified as urban or
nonurban.
fLess than 1 percent of unsuccessful applicants cannot be classified as
urban or nonurban.
Source: U.S. Department of Education, Grant Administration and Payment
System, and Common Core of Data; http://www.nces.ed.gov/ccd; and National
Center for Education Statistics, Education Finance Center
website,http://www.nces.ed.gov/edfin/search (cited Oct. 25, 1999).
Comments From the Department of Education
GAO Contacts and Staff Acknowledgements
Lori Rectanus, Assistant Director, (202) 512-9847
Mary A. Crenshaw, Project Manager, (202) 512-7053
In addition to those name above, George Erhart, Brett Fallavollita, and
Gillian Martin, Senior Evaluators, and Jonathan Barker, Senior Attorney,
Office of General Counsel, made key contributions to this report.
(104970)
Table 1: Discretionary Programs Reviewed 7
Table 2: Goals of Education's Redesign Process and Actions Taken 9
Table 3: Fiscal Year 1998 Discretionary Grant Data for Selected
Programs 23
Table 4: Results of Panel Score Analysis on One Program 27
Table 5: Results of Panel Score Analysis Across Programs 28
Table 6: Number and Percentage of Grantees in Office of Bilingual Education
and Minority Languages Affairs Programs, by
Percentage of LEP Students 30
Table 7: Number and Percentage of Grantees by Percentage of
Children in Poverty 31
Table 8: Distribution of Grantees by Percentage of Children in
Poverty by Urban Status 32
Table 9: Number and Percentage of Grantees by Per-Pupil
Expenditure 33
Table 10: Distribution of Grantees by Per-Pupil Expenditure
Controlling for Urban Status 34
Table 11: Percentage of Grant Applicants by Urban Status 35
1. From Red Tape to Results: Creating a Government That Works Better and
Costs Less, report of the National Performance Review, Vice President Al
Gore (Sept. 7, 1993).
2. See Peer Review: Reforms Needed to Ensure Fairness in Federal Agency
Grant Selection (GAO/PEMD-94-1, June 24,1994 ). Potential weaknesses
include, among other things, the fact that reviewers lack relevant expertise
and have a conflict of interest with the applications they review. The
framework and its adaptation for this report are further discussed in app.
I.
3. Discretionary grants also include cooperative agreements which Education
awards when it anticipates having substantial involvement in working with a
grantee to achieve project outcomes.
4. Schulman, Ronca, and Bucuvalas, Inc., Survey of Customer Satisfaction
with the Department of Education Discretionary Grant Process (Draft) (Sept.
27, 1999).
5. As of March 2000, Education had five competitions under way that allowed
applications to be electronically initiated, developed, and submitted.
6. One program was unable to locate reviewer conflict-of-interest
statements.
7. August and Associates, a subcontractor of the American Institutes for
Research, Strengthening the Standards: Recommendations for OERI Peer Review,
Final Report (Feb. 3, 1999).
8. One program paid reviewers a flat rate of $600 for the entire review.
9. Information was available only for the LEA that received the grant.
Because that LEA might not be the one with a high percentage of children
living below the poverty line, we may be underreporting the number of
grantees with high percentages of children in poverty.
10. GAO/PEMD-94-1 .
11. National Center for Education Statistics on the Department of Education
website.
12. In all five programs, if grantees are identified as specific schools
rather than LEAs, we determined what district a school was located in and
then provided information on that district.
*** End of document. ***