Environmental Research: STAR Grants Focus on Agency Priorities, but
Management Enhancements Are Possible (Letter Report, 09/11/2000,
GAO/RCED-00-170).
Pursuant to a congressional request, GAO provided information on the
Environmental Protection Agency's (EPA) Science to Achieve Results
(STAR) grants program, focusing on: (1) whether funding amounts awarded
for the grants align with EPA's strategic goals, EPA's Office of
Research and Development's (ORD) research priorities, and program office
priorities; (2) the extent to which the completed focused grants have
provided research that is being used by EPA's program offices; and (3)
ways in which ORD could enhance its management of the program to help
ensure that it meets its objectives.
GAO noted that: (1) STAR grant funding has generally been aligned with
EPA's ORD's, and the program offices' broadly defined priorities; (2)
since the STAR program began in 1995 through March 2000, about
two-thirds of the approximately $415 million in funding for STAR grants
has addressed EPA's multipurpose strategic goal of ensuring that EPA
uses sound science in addressing environmental hazards and improving
environmental protection; (3) although STAR grants were generally
aligned with the agency's strategic priorities and those of ORD and the
program offices, EPA's program officials varied in the extent to which
they believe the grants' results are useful to them; (4) according to
the officials, the grants varied in their usefulness to the program
offices largely because the STAR program's goals--meeting program office
needs specifically and advancing environmental science generally--may be
in conflict; (5) ORD has not consistently tracked the STAR grants to
ensure that they are completed on time and that the results are
available for use by EPA's program office; (6) according to several
program officials, ORD could improve its communication with the program
offices during the grant process; (7) ORD has not consistently obtained
information from cognizant program officials when designing the requests
for grant applications or reviewing grant proposals for relevancy during
the grant selection process, nor has ORD adequately communicated the
results of the completed grants; (8) ORD recognizes these concerns and
has several initiatives under way; (9) these include establishing a Web
site to improve communication and involving more program officials in
grant application design and review; (10) because the STAR program is
relatively new, ORD's primary focus has been on establishing a framework
and processes for the program and not on setting program criteria to
measure its overall effectiveness; and (11) recent reports by two
independent scientific organizations agree that while the program is
well structured to achieve its goals, communications between ORD and
EPA's program offices should be improved.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-00-170
TITLE: Environmental Research: STAR Grants Focus on Agency
Priorities, but Management Enhancements Are Possible
DATE: 09/11/2000
SUBJECT: Strategic planning
Research grants
Agency missions
Grant administration
Program evaluation
Environmental research
IDENTIFIER: EPA Science to Achiever Results
Grant Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO/RCED-00-170
A
Report to the Chairman, Subcommittee on VA, HUD, and Independent Agencies,
Committee on Appropriations, House of
Representatives
September 2000 ENVIRONMENTAL RESEARCH
STAR Grants Focus on Agency Priorities, but Management Enhancements Are
Possible
GAO/ RCED- 00- 170
Letter 3 Appendixes Appendix I: Scope and Methodology 22
Appendix II: EPA Funding for Environmental Research 24 Appendix III: STAR
Program Requests for Grant Applications
by ORD's Priority Topics, Fiscal Years 1995 Through 2000 26
Appendix IV: Comments From the Environmental Protection Agency 28
Tables Table 1: STAR Grants by EPA Strategic Goals, Fiscal Years 1995- 99 11
Figures Figure 1: EPA's Process for Awarding STAR Grants 9
Figure 2: Usefulness of 25 Completed Focused Grants to EPA Program Officials
15 Figure 3: Percentage of STAR Funding, by Type of Grant, Fiscal
Years 1995 and 1998 25
Abbreviations
EPA Environmental Protection Agency ORD Office of Research and Development
STAR Science to Achieve Results
Resources, Community, and Economic Development Division
Lett er
B- 142370 September 11, 2000 The Honorable James T. Walsh Chairman,
Subcommittee on VA, HUD, and Independent Agencies Committee on
Appropriations House of Representatives
Dear Mr. Chairman: The Environmental Protection Agency (EPA) has
significantly changed the way it conducts scientific research to fulfill its
mission of protecting human health and the environment. Responding to
criticisms of its research quality and focus, EPA established the Science to
Achieve Results (STAR) grant program in 1995, which is administered by EPA's
Office of Research and Development (ORD). The three objectives of the
program are to (1)
ensure that the agency involves the best non- EPA scientists in its research
efforts, (2) provide useful research support to the agency's program
offices, and (3) train a cadre of environmental scientists for the future.
The STAR program accounted for about 20 percent of the funds ORD obligated
for research for fiscal year 1999. ORD awards STAR grants to scientists on
the basis of 10 broad strategic goals as set out by EPA's annual performance
plan, 1 ORD's six priority topics for scientific research at the agency, and
the specific research priorities of each program office. STAR grants are
available for (1) basic exploratory environmental research, (2) research
focused on specific environmental topics (STAR focused grants), and (3)
fellowships to graduate students to develop the nation's capacity for
addressing future environmental concerns. As of January 2000, about 800
exploratory and focused grants had been awarded and 53 of them
completed; in addition, about 600 fellowships had been awarded and about 200
completed. You asked us to review the status of the STAR grants program.
Specifically, we are providing information on (1) whether funding amounts
awarded for the grants align with EPA's strategic goals, ORD's research
priorities, and
1 The Government Performance and Results Act of 1993 requires each federal
agency to prepare an annual plan that includes the performance goals to be
achieved. See table 1 for a complete listing of EPA's strategic goals.
program office priorities; (2) the extent to which the completed focused
grants have provided research that is being used by EPA's program offices;
and (3) ways in which ORD could enhance its management of the program to
help ensure that it meets its objectives. Because the STAR program is new
and relatively few grants have been completed, it is too early to
definitively judge the program's long- term effectiveness. To address your
questions, we analyzed program data; interviewed EPA
officials knowledgeable about STAR grants; and reviewed studies by
independent scientific groups- specifically, a March 2000 report on the STAR
program by a joint subcommittee of EPA's Science Advisory Board and ORD's
Board of Scientific Counselors and a draft report on EPA's research
management by the National Research Council. 2 To further examine the extent
to which STAR grants were being used by EPA's program offices, we compiled a
list of the 53 focused and exploratory grants completed as of December 6,
1999, from the National Center for
Environmental Research's Web site, as identified by STAR program managers.
We discussed the usefulness of 25 completed grants that were focused on
specific environmental topics with potential users of the
research in EPA, as identified by the STAR program managers. We limited our
review to focused grants because they were more likely to be of more
immediate use to the program offices than those grants related to
exploratory research. Because providing useful research to the program
offices is only one of the program's three objectives, our observations on
whether the research support is being used by the program offices should not
be the only measure of judging the program's overall effectiveness. We also
recognize that scientific research can have long- term benefits but may
not be immediately useful. Our detailed scope and methodology are discussed
in appendix I. Results in Brief STAR grant funding has generally been
aligned with EPA's, ORD's, and the
program offices' broadly defined priorities. Since the STAR program began in
1995 through March 2000, about two- thirds of the approximately $415 million
in funding for STAR grants has addressed EPA's multipurpose 2 A Joint SAB/
BOSC Report: Review of the Science to Achieve Results ( STAR) Program,
Joint Subcommittee of the Science Advisory Board and the Board of Scientific
Counselors, March 2000; and Strengthening Science at the U. S. Environmental
Protection Agency: Research Management and Peer Review Practices, National
Research Council (draft report).
strategic goal of ensuring that EPA uses sound science in addressing
environmental hazards and improving environmental protection. Another 21
percent has addressed the strategic goal aimed at achieving clean air, and
the remaining grant funding has addressed six of EPA's other strategic
goals. STAR grant funds also generally have been aligned with one or more
of ORD's six research priorities, such as achieving safe drinking water, and
with the program offices' main priorities. Some program office research
needs, such as reducing acid rain, were not addressed by STAR grants because
EPA determined they had a lower research priority and because STAR funding
was limited. Although STAR grants were generally aligned with the agency's
strategic
priorities and those of ORD and the program offices, EPA's program officials
varied in the extent to which they believe the grants' results are useful to
them. Of the 25 completed focused grants we examined, 6 were being used and
4 were not expected to be used at all, according to program officials. For
the remaining 15 completed grants, the officials were
uncertain whether the grants would prove useful or not to their programs,
but they stated they might be useful in the future- by serving as building
blocks for research, for example. According to the officials, the grants
varied in their usefulness to the program offices largely because the STAR
program's goals- meeting program office needs specifically and advancing
environmental science generally- may be in conflict. ORD could enhance its
management of the program to help ensure it meets its objectives in several
areas. First, ORD has not consistently tracked the STAR grants to ensure
that they are completed on time and that the results are available for use
by EPA's program offices. Second, according to several EPA program
officials, ORD could improve its communication with the
program offices during the grant process. ORD has not consistently obtained
information from cognizant program officials when designing the requests for
grant applications or reviewing grant proposals for relevancy during the
grant selection process, nor has ORD adequately communicated the results of
the completed grants. ORD recognizes these concerns and has several
initiatives under way. These include establishing a Web site to
improve communication and involving more program officials in grant
application design and review. Because the STAR program is relatively new,
ORD's primary focus has been on establishing a framework and processes for
the program and not on setting program criteria to measure its overall
effectiveness. Recent reports by two independent scientific organizations
agree that while the program is well structured to achieve its goals,
communications between ORD and EPA's program offices should be
improved. Accordingly, we are making several recommendations to EPA for
enhancing its management of the STAR program. We believe these will help
improve the program's potential usefulness and effectiveness.
We provided a draft of this report to EPA for its review and comment. EPA
generally agreed with the report's conclusions that the grants aligned with
EPA's, ORD's, and the program offices' broadly defined priorities and that
funding for STAR grants falls within priority research areas. EPA also
agreed with two of the three recommendations in this report and said it is
taking steps to implement them. EPA did not agree with the basis for the
third recommendation regarding the need for improved tracking and
availability of final grant reports to program offices. Specifically, EPA
stated that its review of grant files showed that more final reports were
completed than indicated by our analysis. However, at the time of our
review, a STAR program official charged with posting reports to the Web
site represented that our list of final reports, compiled from those posted
on EPA's Web site and available to program officials, was complete. Because
it now appears that a significant number of completed final
reports had not been posted to EPA's Web site, which was not represented to
us at the time of our review, we believe the recommendation is still
warranted. Finally, EPA identified several aspects of the draft report that
it believed needed modification; for example, it stated that we negatively
characterized the data on the usefulness of STAR grants to the program
offices. We clarified and presented additional information to address EPA's
comments and made technical changes as appropriate.
Background Credible scientific research is critical to EPA's decisions on
environmental regulations. EPA relies on agency- funded research as well as
on research
funded by others, including government agencies, such as the National
Science Foundation and National Institutes of Health, and private sources.
Prior to the STAR program, EPA mainly funded research through
cooperative agreements with outside entities, issued through its
laboratories, to supplement the research EPA scientists performed. In
addition, ORD funded a relatively small number of grants, managed by ORD
headquarters staff, for longer- term exploratory research.
During the early 1990s, EPA's independent Science Advisory Board, the
National Academy of Sciences, and other groups reviewed the status of
scientific research at EPA. The groups recommended that EPA make changes to
strengthen the quality of its scientific research, such as better balancing
its short- and long- term research and taking steps to train the
next generation of scientists. EPA responded, in part, by establishing the
STAR program in 1995. In doing so, EPA shifted much of its funding for
external research away from noncompetitively awarded cooperative agreements
administered by ORD's laboratories and placed more emphasis on competitively
awarded, peer- reviewed grants. EPA believed that this realignment would
allow ORD scientists to spend less time administering contracts and more
time conducting research, while the STAR program
would involve top- quality scientists from outside EPA in helping to meet
the agency's research needs. The STAR program has three principal
objectives. First, through the
program, EPA attempts to make the nation's academic research community an
integral part of its research program, thereby ensuring that the agency has
the highest quality science for its use. Second, EPA seeks to supplement and
complement its internal research program, thereby supporting its mission by
meeting the scientific and technical needs of the program offices. Finally,
EPA attempts to support higher education in selected scientific fields,
thereby developing a stronger scientific community for the future.
To achieve these objectives, the STAR program relies on three types of
grants: exploratory grants, focused grants, and graduate fellowships.
Exploratory grants are for fairly broad areas of environmental science, such
as environmental chemistry and physics. Focused grants are targeted
more specifically at particular research needs, such as developing standards
for safe drinking water, that have been identified in ORD's strategic plan.
Focused grants can be awarded to environmental research
“centers,” such as the Airborne Particulate Matter Centers.
These centers comprise multiple scientists- usually from different
disciplines- who are concerned with longer- term and cross- issue research.
Finally, fellowships are given to graduate students in specified
environmental disciplines, such
as toxicology, chemistry, and economics. Fellowships for master's and
doctoral students are for 2 and 3 years, respectively. For fiscal year 1999,
ORD obligated about $91 million in STAR grants for externally conducted
research. This represented about 20 percent of ORD's overall obligations of
about $458 million for research. Other resources within and outside of EPA
also provide funding for environmental research. Additional information on
overall environmental research funding and funding levels for the three
types of STAR grants is provided in appendix II. EPA solicits STAR grants
through annual requests for grant applications.
Proposals for exploratory and focused grants are reviewed for their
scientific and technical merit by panels of non- EPA scientists, which
reject those that are not considered scientifically sound. Next, panels of
EPA officials from ORD and other offices review and rank the proposals for
relevance to their research needs. ORD then selects proposals to be awarded,
taking into consideration both scientific merit and relevance to research
needs. ORD follows a similar process for fellowship proposals.
EPA's process for awarding STAR grants is illustrated in figure 1.
Figure 1: EPA's Process for Awarding STAR Grants
EPA's strategic plan (10 strategic goals) ORD's strategic plan (6 priority
topics)
Topic- specific research plan (developed by research coordinating teams)
Extramural Intramural research needs
research needs RFAs forwarded via ORD Federal Register, Internet,
Requests for implementation
and university and plans scientific organizations Applications written
(developed by laboratories) External peer Scientific
review and community
ranking of proposals (based on technical merit)
Internal relevancy review (performed by agencywide panel) Proposal selection
(based on research needs and to complement intramural research)
STAR Grant Funding Is Because EPA's, ORD's, and the program offices'
priorities are broadly
Aligned With EPA's, defined, most STAR grant funding has generally been
aligned with them. Since the STAR program began in 1995, ORD has funded
grants aligned
ORD's, and Program with 8 of EPA's 10 strategic goals. Most of the grant
funds have been aligned
Offices' Broad Goals with EPA's strategic goal of ensuring that the agency
uses sound science in addressing environmental issues, followed by the goal
aimed at achieving
clean air. Furthermore, all of the funding for STAR grants awarded falls
within one or more of ORD's six priority research topics, and grants have
generally been awarded for the highest priority needs of each program
office. However, some lower- priority research topics, such as reducing acid
rain, are not being funded by STAR grants or other EPA research because of
resource constraints.
STAR Grant Funding Falls Since 1995, when the STAR program began, and
through March 2000, ORD Within EPA's Strategic Goals
has awarded about $415 million in exploratory and focused grants. Once all
of the awards are made for fiscal year 1999, total exploratory and focused
grant awards will amount to about $430 million. EPA has awarded these grants
in keeping with eight program- related strategic goals that could benefit
from STAR grant research. Table 1 shows how the $415 million in grant awards
corresponds to EPA's strategic goals. The table does not include fellowship
grant awards, which amounted to about $39 million for fiscal years 1995
through 1999.
Table 1: STAR Grants by EPA Strategic Goals, Fiscal Years 1995- -99 --
Dollars in millions
EPA's strategic goal Award amounts
Clean air $87.0
Clean and safe water 16. 5
Safe food 2.5
Preventing pollution and reducing risk in communities, homes, 3.0
workplaces and ecosystems Better waste management, restoration of
contaminated waste 9.7
sites, and emergency response Reduction of global and cross- border
environmental risks 15. 6
Expansion of Americans' right to know about their environment 3.6
Sound science, improved understanding of environmental risk,
277.1
and greater innovation to address environmental problems Total $415. 0
Note: Individual grant amounts are associated with one strategic goal. STAR
grants are not applicable to 2 of EPA's 10 strategic goals, “a
credible deterrent to pollution and greater compliance with the law”
and “effective management.”
Source: GAO's analysis of EPA's data.
STAR grants applicable to the sound science and clean air goals represented
about 67 and 21 percent of the funds awarded, respectively. Since sound
science is a multipurpose goal, grants funded under it frequently apply to
another goal as well.
STAR Grants Align With In addition to EPA's strategic goals, ORD has
established six broad priority ORD and Program Office research topics: safe
drinking water, high- priority air pollutants, emerging Priorities issues,
improving ecological risk assessment, improving human health risk
assessment, and pollution prevention and new technology. These are
generally consistent with EPA's strategic goals. Three of ORD's priority
topics- improving the assessment of risks to human health, emerging issues,
and improving ecological risk assessment- have been the focus of the
majority of STAR grants awarded. Appendix III shows how the research areas
of the awarded grants are aligned with the six research topics. ORD takes
the needs of program and regional offices into consideration
when writing the requests for grant applications and awarding STAR grants.
ORD's research coordination teams include program office and regional
representatives who identify EPA's research needs and determine
the research that will be conducted internally or externally. According to
the science advisers in the offices of Air and Radiation and Solid Waste and
Emergency Response who have served on the teams, ORD considers program
offices' research priorities and ensures that the offices' top research
needs are covered by the STAR grants. For example, because EPA considers
particulate matter in the air to be a costly potential health risk, the
Office of Air and Radiation has particulate matter research as a top
priority, and ORD has awarded STAR grants for research on particulate matter
under its topic of high- priority air pollutants. A program official in the
Office of Air and Radiation, who is responsible for developing the
standards, is incorporating STAR grant results into the new standards.
Funding for STAR grants has generally been consistent with the highest
research priorities of the program offices. Some lower- priority research
needs are not benefiting from STAR grants, however, because of resource
constraints. According to program officials, the likelihood that the grants
will address a research need depends on where that need ranks in the
program offices' and ORD's priorities. For example, officials concerned with
acid rain issues, which is a lower priority for the Office of Air and
Radiation, told us that to date no STAR grants have included research that
can be used to meet their needs. Similarly, no STAR grants have targeted
research on stratospheric ozone or asthma mitigation. Officials responsible
for these areas believe they could benefit from STAR research, but their
research needs have a relatively low priority within the Office of Air and
Radiation. Although ORD recently decided to fund a combustion research
project that the Office of Solid Waste and Emergency Response will use, an
official from that office told us that the STAR program generally has
focused on few of its research needs because waste issues are currently a
low priority for ORD.
According to program officials, the approximately $100 million in annual
funding for STAR grants is not sufficient to cover all of the program
offices' research needs that are not otherwise met by other sources, such as
ORD's
laboratories. To compensate, the officials stated, they seek out research
being conducted by other organizations, some in other countries, on similar
topics that may meet their research needs. Recently, the Science Advisory
Board's Research Strategies Advisory Committee completed its annual review
of the agency's budget request for fiscal year 2001 science and technology
funds. The committee agreed that EPA is not adequately funding research on
certain environmental issues, such as hazardous waste cleanup. ORD officials
concur that budget constraints limit grant awards and that consequently some
research cannot be funded through the
program. They told us that they consider research being done by scientists
at EPA and other agencies, such as the National Science Foundation, when
awarding STAR grants to ensure that program offices' research needs get the
best coverage possible.
Usefulness of the Although STAR grants are generally aligned with EPA's
strategic priorities Focused Grants to
and those of ORD and the program offices, program officials believed that
the usefulness of the completed focused grants to their programs varied.
EPA's Program Offices
They stated that they were using 6 of the grants, did not expect to use 4 of
Varies, Largely
them, and were uncertain about the extent to which the remaining 15 Because
of Competing grants might be useful in the future. According to the
officials, the grants varied in their usefulness to the programs largely
because the STAR
Program Objectives program's goals- advancing environmental science
generally and meeting
and the Nature of program needs specifically- may be in conflict.
Grants Usefulness of the
For the 25 completed grants we examined, 6 were being used by EPA Completed
Focused Grants
program officials because they met a research need. For example, one to
EPA's Program Offices
STAR grant created an animal model emulating the development of autism
Varies
in humans. The goal of the research was to assess how a woman's exposure to
certain chemicals during pregnancy could affect the fetus. According to
officials in ORD and the Office of Prevention, Pesticides, and Toxic
Substances, the STAR research was the first proving a relationship between
chemicals and the development of autism. The officials are using the results
as they develop guidelines for assessing risks to children's health.
In contrast, program officials did not expect to use the research results
from four grants at all. For example, a grant researching the development of
methods to prevent wastes by auto suppliers did not provide useful results,
according to officials in the Office of Prevention, Pesticides, and Toxic
Substances. The officials stated that they had begun similar research 2 to 3
years before the grant was issued and that they had researched the issue
more extensively than the grantees did. They stated, however, that the
grantees benefited by increasing their technical expertise in conducting
this type of research.
Program officials believed that the extent to which the remaining 15
completed focused grants might prove useful in the future varied. For
example, six grants, totaling over $750,000, that researched methods for
estimating the benefit of environmental policies were not currently useful
but were expected to be useful to the agency in the long term. Program
officials told us that the research topic- estimating values for lives,
ecosystems, and other entities affected by environmental policies- is an
important one for the agency. These officials generally agreed that the
grants contributed to an improved understanding of the overall issue of
environmental valuation and would probably provide some basis for future
work on the issue. One of the program officials also stated that it was
probably prudent to fund these grants in order to advance knowledge and
understanding of this issue, even though the grants are not providing
research results that can be used immediately. On the other hand, the future
usefulness of a grant investigating wet cleaning processes- instead of the
standard dry cleaning process- is less certain, according to officials in
the Office of Prevention, Pesticides, and Toxic Substances. The officials
stated that the research was well done and illustrated the environmental
benefits of wet cleaning over dry cleaning. However, they believe that the
garment cleaning industry is unlikely to adopt the new technology because
cleaning companies have already made costly changes in the infrastructure of
their plants to comply with EPA regulations. The officials plan to keep
the grant results as a reference on the garment cleaning industry in the
event that future regulatory changes make alternative cleaning approaches
more viable.
Figure 2 shows how program officials perceived the usefulness of the 25
completed focused grants.
Figure 2: Usefulness of 25 Completed Focused Grants to EPA Program Officials
Number of grants 6 Useful
15 4 Not useful Usefulness uncertain
Note: These focused grants were completed as of December 1999.
The STAR Program's The STAR program's objectives may be contradictory in
that the program is
Competing Goals and expected to focus on achieving long- term gains in
environmental science as
Reliance on Grants Limit the well as being responsive to the more immediate
needs of the program
Usefulness of the Research offices. According to some program officials,
they understood that the to EPA Program Officials
foremost goal of the STAR program was to advance environmental science
rather than to meet the programs' needs. As a result, STAR grants may be
limited in the extent to which they are useful to the program offices.
Furthermore, the STAR program's reliance on grants to accomplish
environmental research limits its ability to directly respond to the
research needs of EPA's program offices. The principal purpose of any grant
is to
assist grantees in furthering a public purpose rather than to obtain
services directly for the government's benefit. According to several EPA
program officials, program offices' needs do not always match up with grant
recipients' research objectives; therefore, some of the STAR grant topics
are not directly applicable. In addition, one program official told us that
STAR researchers are sometimes reluctant to answer questions posed by
program officials because the terms of the grant do not require them to be
responsive to the needs of EPA's program offices.
Research for STAR grants may also not be completed in a time frame that is
useful to the program offices. STAR grants range from 1 to 5 years. Grantees
can request a 1- year extension, which ORD officials must automatically
approve in accordance with OMB's Circular A- 110, according to ORD
officials, and ORD may authorize additional extensions. Thus, even
though it may be several years before STAR grants produce results, the
grantees are still in compliance with the terms of their grants. Program
officials also differed in whether grant timing fit their needs. For
example, an official in the Office of Air and Radiation told us that
projecting research needs that far in advance was possible, while another in
the Office of Solid Waste and Emergency Response stated that many of his
office's research
needs involve cleanup work at Superfund sites and require results in a
shorter amount of time than the grants can usefully provide.
ORD Could Enhance ORD could enhance its management of the STAR program to
help ensure
Its Management of the that the results of the grants are readily useful to
the EPA program offices
and that the program meets its objectives. Areas needing further STAR
Program to Help improvements include (1) tracking grants to ensure that they
are Ensure That It Meets
completed on time and have produced the research intended; (2) improving Its
Objectives
communications with EPA's program offices when designing and reviewing
grants for relevancy and disseminating grant results; and (3) establishing
criteria to measure program effectiveness. Reports by two independent
scientific organizations agree that while the program is generally well
structured for achieving its goals, communications between ORD and EPA's
program offices should be improved. ORD Could Further Improve
ORD has not been adequately tracking grants and has not ensured that Its
Tracking of Grant
exploratory and focused grants were completed on time. As of December
Results
1999, 144 of the nearly 200 grantees for exploratory and focused grants had
missed their deadlines, including extensions, for submitting final reports.
ORD officials recently noted that they plan to do a better job of tracking
grants. In that regard, in June 1999, ORD hired a contractor to track and
identify interim and final reports. ORD has identified a number of
additional final reports since December 1999. According to a STAR
program manager, however, ORD cannot ensure that research will produce
results in the agreed time or with the intended results. Similarly, ORD has
not adequately tracked the fellowship grants to determine whether they have
been effective. It had not analyzed whether any of the approximately
200 fellowships completed as of December 1999 had resulted in college
graduates taking jobs in environmental occupational fields. Recognizing
the need to determine whether fellowship recipients are continuing in the
environmental sciences, ORD began in early 2000 to track the recipients of
completed fellowships. In that effort, ORD has obtained follow- up
information for about 60 of the 200 completed fellowships. ORD Could Better
ORD could also improve its communication with the program offices
Communicate With Program
during the grant process, according to several EPA program officials. For
Offices About STAR Grants example, ORD has not consistently obtained
information from cognizant program office officials when designing requests
for grant applications or
reviewing grant proposals for relevancy during the grant selection process.
While a number of program officials have participated in these processes,
others told us that they have not been invited to do so. One official, who
had not been involved in the pre- award process, believes that, had she been
involved, she might have influenced ORD to select a grant that could have
met her and others' research needs. She acknowledged that ORD has recently
recognized that more participation from the program offices prior to
awarding the grants could increase their usefulness. The draft report by
the National Research Council agreed that some program officials had
expressed concern that, in practice, they have little influence on ORD's
research priorities. Furthermore, ORD had not adequately communicated grant
results to program officials. For 13 of the 25 grants we examined, program
officials learned about completed research as a result of ORD outreach and
notification. For the remaining grants, officials learned about 1 grant's
final results through a professional conference and were not aware of the
completed research for the remaining 11 grants until we contacted them for
this review. In addition, most of the program officials were not aware of
interim research results for the grants we reviewed. One program official
explained that, if he had been aware of the interim research results on air
pollution in large buildings for one STAR grant, he could have notified the
researchers of problems with their methodology and corrections could have
been made. Instead, the usefulness of the completed research is
questionable, and unless the methodology issue can be resolved, the
official will be unable to use the results. ORD has taken steps to
communicate the results of STAR program research, including posting STAR
research report summaries on its Web site, providing program offices with
lists of grants that might contain useful research, and sponsoring workshops
on grant topics. While the program officials acknowledged that ORD makes
STAR information available, the
format of this information makes it difficult for program officials to
identify research relevant to their needs. For example, program officials
stated that interim and summaries of final research reports are organized on
ORD's Web site by grant topic, a format that makes identifying specific
relevant research difficult because they have to sift through numerous
lengthy reports. For the ORD workshops on grant topics, not all grant topics
are covered. For example, the indoor air research topics were not covered in
any workshop, according to a program official. ORD managers acknowledge
ongoing difficulties in conveying research results to the program offices
and say they have been increasing their efforts to do so.
According to both the National Research Council's draft report and the March
2000 report by the joint subcommittee of EPA's Science Advisory Board and
ORD's Board of Scientific Counselors, now that the STAR
program is starting to produce results, EPA should place greater emphasis on
communicating those results to potential users in the agency. By
strengthening communication between STAR program managers and the program
offices, ORD could better ensure that STAR results are being rapidly and
effectively transferred to the officials who can use them. The
joint subcommittee report also recognizes that ORD has initiated several
efforts to improve communication, such as developing reports on the status
of research topics.
ORD Has Not Focused on Because the STAR program is relatively new, ORD's
primary focus has been Developing Measures to on establishing a framework
and processes for the program, not on setting Help Ensure the Program
criteria for measuring its overall effectiveness. For example, ORD has not
Meets Its Goals
set goals for measuring whether the grants are meeting the research needs of
EPA's program offices. Such measures would allow STAR managers to determine
if the program as a whole is achieving its expected results of involving the
best scientists in EPA research, providing useful research for
the program offices, and training a cadre of environmental scientists for
the future. According to ORD officials, most of their efforts to date have
focused on establishing a high- quality research program. They now believe
it is time to place a greater emphasis on evaluating the program's benefits.
Consequently, ORD asked the joint subcommittee of EPA's Science Advisory
Board and ORD's Board of Scientific Counselors for assistance in developing
measures for determining if the STAR program is meeting its overall goals.
The March 2000 joint subcommittee report focused on evaluating whether the
STAR program is structured appropriately to achieve its goals. Overall,
the report found that the program is structured and managed to generate
high- quality science, conducted by well- qualified scientists, on topics
that are relevant to the environmental problems identified in EPA's
strategic plan. The report concluded that the STAR program is generally well
planned, organized, and managed for achieving its goals. However, it
recommended that ORD seek expert assistance in developing a monitoring and
evaluation system for the program. Conclusions Initial results indicate that
the STAR program is yielding some research that is useful to EPA's program
offices and that it is well- structured and
managed to generate high- quality science. However, for the STAR program to
better meet one of its major goals- that of supporting the research needs of
EPA's program offices- ORD must improve its tracking of grants to ensure
that the research is being completed as agreed and that it is communicating
the research results throughout EPA. Reports by two
independent scientific groups agree that the STAR program's research results
should be better communicated. EPA recognizes this concern and has taken
several actions to improve communications, but more could be done.
ORD has focused the majority of its efforts to date on the near- term
objectives of establishing the STAR program and on awarding the grants.
However, ORD has not established criteria to help ensure that the program is
meeting its major objectives. Now that the program has begun to
produce results, it is appropriate for EPA to focus greater attention on
managing on- going research, disseminating research results, and evaluating
program results to date. Recommendations To enhance the effectiveness of the
STAR program, we recommend that the
Administrator of EPA direct the Assistant Administrator of ORD to take the
following actions:
Track and monitor the grants to ensure that interim and final research
results are delivered on time and are made available as soon as possible for
use by the program offices. Take the additional steps needed to better
disseminate and communicate STAR research results to the appropriate program
officials. This would require continuing and expanding the efforts
already under way to consult with program offices in determining the most
effective communication methods. Develop program criteria to evaluate the
effectiveness of each type of grant- exploratory grants, focused grants, and
fellowships. In addition,
the criteria should assist EPA in drawing an overall conclusion on whether
these grants satisfy the program's overall objectives. Agency Comments We
provided a draft of this report to EPA for its review and comment. EPA's
comments and our detailed responses are in appendix IV. EPA agreed with the
report's conclusions that the grants generally align with EPA's, ORD's, and
the program offices' broadly defined priorities and the funding for STAR
grants falls within priority research areas. However, EPA suggested changes
in five other areas. First, EPA stated that our report negatively
characterized the data on the usefulness of STAR grants to the program
offices, noting that no benchmarks exist for an appropriate level of
usefulness. We disagree. Rather, our report simply portrays the views of EPA
program officials, one of the intended beneficiaries of the program, on
the grants' usefulness. We agree that no benchmarks exist for the usefulness
of grants, although benchmarks would be helpful to evaluate the program.
Second, EPA stated that there is no contradiction in using grants to support
the program offices and to further advance environmental research. We
recognize that the STAR program can contribute to both of these goals.
However, as the draft report stated, program officials noted
that these goals may conflict and that the primary purpose of grants is to
assist grantees in furthering a public purpose rather than in obtaining
services directly for the government's benefit. Third, EPA stated that the
report misrepresented the relative amount of STAR grant funding to overall
ORD funding. We changed the report to more accurately reflect the proportion
of ORD's funding accounted for by the STAR grant program. Fourth, EPA stated
that it did not agree with our conclusions as to the number of grants that
had been completed and made available to program offices. Specifically, EPA
stated that, by counting only those final reports posted to its Web site, we
did not include a substantial number of additional completed grant reports
that were in its files. Therefore, EPA disagreed with our conclusion and
recommendation that tracking of STAR
grants should be improved. While EPA states that additional research reports
were in its files, at the time of our review, a STAR program official
confirmed that all final reports were posted to the Web site. If they were
not posted, they were not readily available to the program offices.
Therefore, we believe our conclusions and recommendation are warranted.
Finally, EPA stated that it has been taking steps to implement two of the
report's recommendations to better disseminate and communicate STAR research
results and to evaluate the program. We acknowledged these efforts in the
report.
We are sending copies of this report to appropriate congressional
committees; interested Members of Congress; the Honorable Carol M. Browner,
Administrator, Environmental Protection Agency; and other interested
parties. We will also make copies available to others on request.
Please call me at (202) 512- 6111 if you have any questions about this
report. Key contributors to this report were John Wanska, Karla Springer,
John C. Johnson, and Roger Bothun.
Sincerely yours, Peter F. Guerrero Director, Environmental
Protection Issues
Appendi Appendi xes x I
Scope and Methodology To obtain information on the Environmental Protection
Agency's (EPA) Science To Achieve Results (STAR) grant program, we
interviewed officials in, and collected and analyzed information from, the
Office of Research and Development's (ORD) National Center for Environmental
Research and the Grants Administration Division; the Office of Air and
Radiation; the Office of Solid Waste and Emergency Response; and the Science
Advisory Board's Research Strategies Advisory Committee. We also reviewed
two recent reports by independent scientific groups. The March 2000 report
by a joint
subcommittee of EPA's Science Advisory Board and ORD's Board of Scientific
Counselors objectives were to examine whether (1) the STAR program is
structured appropriately to achieve its stated purpose, (2) the program is
integrated effectively with the agency's strategic plans and programs, and
(3) efforts to communicate with the external scientific and regulatory
communities regarding STAR research opportunities and
outputs were adequate. The draft report by the National Research Council
assessed the overall structure and management of EPA's research program and
evaluated scientific peer review procedures used by the agency. To determine
whether the grants awarded met EPA's strategic goals and ORD's priorities,
we interviewed ORD officials to obtain an understanding of how grants are
categorized by subject area and how these subjects were aligned with EPA's
strategic goals and priorities. We also obtained information on the grant
award process and reviewed documentation on the process. The documents
reviewed included EPA's and ORD's strategic
plans and various listings of the grants awarded. The listings identified
the year of the award, the research topic, and the dollars awarded. Using
this information, we compared the grants awarded to EPA's goals and ORD's
research priorities to determine how the grants were aligned with specific
goals and priorities. To assess the extent to which the completed focused
grants have provided research that is being used by EPA's program offices,
we obtained information on those grants completed as of December 6, 1999,
from the National Center for Environmental Research's Web site. At the
suggestion of STAR program managers, we considered a grant to be completed
if its final report was posted on the Web site and the December 6, 1999,
date provided STAR program staff time to ensure that all available final
research
reports had been posted on the site. We excluded completed exploratory
grants, since those grants were less likely, according to STAR managers,
than focused grants to yield results that the programs could use in the
nearterm. We also obtained from a STAR manager a list of staff in the
program offices and regions who were most likely to benefit from the results
of the
completed grants. We contacted those officials, and selected others who we
learned were also potential users of the results, to discuss their views on
the research from the completed grants. Officials we interviewed were from
ORD; the offices of Prevention, Pesticides, and Toxic Substances, Water, Air
and Radiation, Solid Waste and Emergency Response, International Activities,
and Policy and Reinvention; and regions IX and X.
In addition, we used data from EPA's automated Grants Information Control
System to compile lists of grants that should have been completed as of
December 6, 1999. Exploratory and focused grant recipients are required by
their grants to provide a final report to ORD within 90 days of the
expiration of the grant's project period. Therefore, to determine which
exploratory and focused grants should have been completed and had final
reports submitted to STAR managers, we considered only those with project
periods ending at least 90 days prior to December 6, 1999. As a condition of
the grant, fellowship recipients must submit their theses or other
publications to ORD after the end of the grant period, as soon as those
papers are available. Because that deadline is less definitive, to determine
which fellowships should have been completed, we used
December 31, 1999, as the cutoff date and asked STAR managers about any
actions they were taking to ensure that the fellowships were effective. We
did not independently verify the accuracy of the grants data. However, we
discussed the issue of data accuracy with an official in the Grants
Administration Division, who told us that the data fields we used were
likely to be accurate because they are relatively easy for the grant project
managers to keep updated. We conducted our work between September 1999 and
August 2000 in accordance with generally accepted government auditing
standards.
Appendi x II
EPA Funding for Environmental Research For fiscal year 1999, EPA obligated
about $458 million for ORD, whose primary purpose is to provide research for
the agency. About $218 million of this amount was for internal research ORD
conducted, about $91 million was for the external STAR program, and about
$149 million was for other external research, such as contracts. In addition
to the ORD funding, EPA's program offices fund research projects. In fiscal
year 1999, about $118 million was available for these projects, but the
agency does not know how much of that was spent on research- related
efforts. ORD is currently trying to identify all the research that EPA is
conducting; however, ORD does not
plan to identify the amount of funding for research conducted outside of ORD
because of difficulties in defining and accounting for research. Estimates
of the total federal funding for environmental research range from about $2
billion to $3.2 billion for fiscal year 2000. 1 The relative funding levels
for the three types of STAR grants- exploratory, focused, and fellowships-
has changed from fiscal year 1995 to fiscal year 1998. In fiscal year 1995,
the year the STAR program began, EPA awarded approximately $58 million, with
focused grants accounting for the largest share of funding, about $29
million, or 50 percent, of the total. In fiscal year
1998, ORD awarded approximately $148 million in STAR grants, and focused
grants constituted 83 percent of the total amount as shown in figure 3. 2 1
In April 2000, the American Association for the Advancement of Science
estimated about
$2 billion in total federal environmental research for fiscal year 1999; in
1999, the National Science Foundation estimated about $3. 2 billion.
2 Fiscal year 1998 was the last year for which all of the STAR grants were
awarded. ORD is still in the process of awarding grants for requests for
proposals from fiscal year 1999 and has not yet awarded any grants for the
fiscal year 2000 applications.
Figure 3: Percentage of STAR Funding, by Type of Grant, Fiscal Years 1995
and 1998
Fiscal year 1995 Fiscal year 1998 50%
7%
Focused
Fellowships
10% 9%
Exploratory Fellowships
Noncenter
41%
Exploratory
46%
Center Noncenter
37% 83%
Focused
Source: GAO's analysis of EPA's data.
Over half of the focused grants awarded for fiscal year 1998 were for
research centers at universities, a practice that ORD has recently begun to
emphasize. There were no center grants for fiscal year 1995. According to a
STAR program manager, many of the research issues that the agency faces
currently, and expects to face in the future, require multidisciplinary
work, and grants to centers are more suited to this type of research.
Fellowship grants accounted for about $10 million in fiscal year 1998 and
supported approximately 300 fellowships. ORD expects to continue funding
fellowships at this level. Awards for exploratory grants have decreased
significantly. According a to STAR program manager, exploratory grants are
proving to be the least useful of the STAR grants. While the manager
believed some of the broad research done with exploratory grants is
beneficial to the agency, ORD has gradually diminished the exploratory
share in favor of more focused grants.
STAR Program Requests for Grant Applications by ORD's Priority Topics,
Fiscal
Appendi x I II
Years 1995 Through 2000 ORD's priority topics Research to
Research to improve
improve Pollution
Safe Highpriority ecological
human health prevention
STAR Requests for Application by drinking air
Emerging risk
risk and new
subject matter water
pollutants issues assessment
assessment technology
Air
Indoor air quality X X
Health effects of particulate matter X X X X
Air pollution chemistry and physics X X X X
Air toxics X X X
Mercury fate and transport X X X X Water
Drinking water X X X
Risk- based decisions for contaminated sediments X X X
Water and watersheds X X
Health effects of arsenic X X Ecology
Ecological assessment and indicators X X
Global climate change X X
Regional scale assessment and analysis X X
Ecology and oceanography of harmful algal blooms X X X
Ecosystem restoration X Health
Exposure of children to pesticides X X
Endocrine disrupters X X X
Children's environmental health and disease prevention research centers X X
Human health risk assessment X X X
Role of inter- individual variability in human susceptibility X X X
Children's vulnerability to toxic substances in environment X X X X X
Exposure to waste combustion products X X X X
Chemical mixtures in environmental health X X X X X Other
Analytical and monitoring methods X X X X X
(Continued From Previous Page)
ORD's priority topics Research to
Research to improve
improve Pollution
Safe Highpriority ecological
human health prevention
STAR Requests for Application by drinking air
Emerging risk
risk and new
subject matter water
pollutants issues assessment
assessment technology
Environmental fate and treatment of toxics and hazardous wastes X X X
Environmental statistics X X X X
High- performance computing X X X X
Technology for a sustainable environment X X X
Decision- making and valuation for environmental policy X X
General solicitation - exploratory research X X X X X X
Socioeconomic projects related to pollution prevention X X
Program on bio- remediation X X X
Futures: detecting the early signals X Total 10 12 22 18 24 11
Source: EPA's classification of grants awarded by research priorities.
Comments From the Environmental
Appendi x V I Protection Agency Note: GAO's comments supplementing those in
the report text appear at the end of this appendix.
See comment 1.
See comment 2.
See comment 3. See comment 4.
See comment 5.
GAO's Comments. 1. We do not agree that our report negatively characterizes
the number of grants useful to program officials. Our report states the
number of completed focused grants that are useful, not useful, or of
uncertain usefulness based on our discussions with EPA program officials. We
recognize that there is no benchmark for the number of grants that should
be useful to program officials, although one would be helpful to determine
the effectiveness of the STAR program. Furthermore, while we agree that STAR
research may be useful to parties outside of EPA, we did not go
beyond interviewing EPA program and regional officials about the usefulness
of STAR focused grants because it is a stated goal of these grants to
address the research needs of these officials.
2. We disagree that there may not be a contradiction in using a grants
program to simultaneously support the agency's mission and to advance the
understanding of complex environmental issues. We recognize that EPA has
made efforts to direct STAR grants into research areas that are relevant to
the program offices, and our review has shown that, consequently, some
research results have proven useful in meeting program needs. However, the
principal purpose of any grant is to assist grantees in furthering a public
purpose rather than obtaining services directly for the government's
benefit. Furthermore, program officials stated that the needs of the program
offices have not always matched the grant recipients' research objectives
and that the grant research has often been designed to contribute to broad
scientific issues. Therefore, as the program officials we
interviewed pointed out, the goals of the program may conflict. 3. We agree
that the amounts comparing STAR program obligations with total ORD research
obligations for fiscal year 1999 should be revised. We revised the report to
reflect that ORD obligated $91 million for the STAR
program out of a total of $458 million it obligated for all research. Thus,
the STAR program accounts for about 20 percent of all research obligated by
ORD. We do not agree that the STAR program's obligated amounts should be
compared to ORD's overall budget because that amount may include funding for
activities other than research.
4. We disagree with EPA's assertion that our conclusion and recommendation
regarding the need to improve the tracking process is incorrect and should
be revised accordingly. While EPA states that additional research reports
were available in the files at the time of our
review, these were not posted to the Web site and were therefore not readily
available to the program offices. We worked with STAR program
managers to develop the methodology for counting completed grants as those
with final reports posted to the Web site, as discussed in appendix I.
Furthermore, we provided the officials with additional time to identify all
completed reports in their files and to post them to the Web site. As a
result, we believe that no changes are needed to our report.
5. Our report acknowledges the efforts EPA has begun to develop criteria for
evaluating the STAR program's effectiveness and to better disseminate and
communicate research results.
(160505) Lett er
(160505) Lett er
Ordering Information The first copy of each GAO report is free. Additional
copies of reports are $2 each. A check or money order should be made out to
the Superintendent of Documents. VISA and MasterCard credit cards are
accepted, also. Orders for 100 or more copies to be mailed to a single
address are discounted 25 percent.
Orders by mail: U. S. General Accounting Office P. O. Box 37050 Washington,
DC 20013
Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW)
U. S. General Accounting Office Washington, DC
Orders by phone: (202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet: For information on how to access GAO reports on the
Internet, send an e- mail message with “info” in the body to:
info@ www. gao. gov or visit GAO's World Wide Web home page at: http:// www.
gao. gov
To Report Fraud,
Contact one:
Waste, or Abuse in Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
Federal Programs
e- mail: fraudnet@ gao. gov 1- 800- 424- 5454 (automated answering system)
GAO United States General Accounting Office
Page 1 GAO/ RCED- 00- 170 STAR Grant Program
Contents
Page 2 GAO/ RCED- 00- 170 STAR Grant Program
Page 3 GAO/ RCED- 00- 170 STAR Grant Program United States General
Accounting Office
Washington, D. C. 20548 Page 3 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 4 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 5 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 6 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 7 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 8 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 9 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 10 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 11 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 12 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 13 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 14 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 15 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 16 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 17 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 18 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 19 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 20 GAO/ RCED- 00- 170 STAR Grant Program
B- 142370 Page 21 GAO/ RCED- 00- 170 STAR Grant Program
Page 22 GAO/ RCED- 00- 170 STAR Grant Program
Appendix I
Appendix I Scope and Methodology
Page 23 GAO/ RCED- 00- 170 STAR Grant Program
Page 24 GAO/ RCED- 00- 170 STAR Grant Program
Appendix II
Appendix II EPA Funding for Environmental Research
Page 25 GAO/ RCED- 00- 170 STAR Grant Program
Page 26 GAO/ RCED- 00- 170 STAR Grant Program
Appendix III
Appendix III STAR Program Requests for Grant Applications by ORD's Priority
Topics, Fiscal Years 1995 Through 2000
Page 27 GAO/ RCED- 00- 170 STAR Grant Program
Page 28 GAO/ RCED- 00- 170 STAR Grant Program
Appendix IV
Appendix IV Comments From the Environmental Protection Agency
Page 29 GAO/ RCED- 00- 170 STAR Grant Program
Appendix IV Comments From the Environmental Protection Agency
Page 30 GAO/ RCED- 00- 170 STAR Grant Program
Appendix IV Comments From the Environmental Protection Agency
Page 31 GAO/ RCED- 00- 170 STAR Grant Program
Appendix IV Comments From the Environmental Protection Agency
Page 32 GAO/ RCED- 00- 170 STAR Grant Program
Appendix IV Comments From the Environmental Protection Agency
Page 33 GAO/ RCED- 00- 170 STAR Grant Program
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300
Address Correction Requested Bulk Rate
Postage & Fees Paid GAO Permit No. GI00
*** End of document. ***