National Science Foundation: Status of Achieving Key Outcomes and
Addressing Major Management Challenges (15-JUN-01, GAO-01-758).
This report reviews the National Science Foundation's (NSF)
fiscal year 2000 performance report and fiscal year 2002
performance report plan required by the Government Performance
and Results Act. Specifically, GAO discusses NSF's progress in
addressing several key outcomes that are important to NSF's
mission. NSF reported that it made substantial progress in
achieving its key outcomes. Although the planned strategies for
achieving these key outcomes generally are clear and reasonable,
some are vague and do not identify the specific steps for
achieving the goals. NSF's fiscal year 2000 performance report
and fiscal year 2002 performance plan reflect continued
improvement compared with the prior year's report and plan.
Although the 2002 performance plan does not substantially address
NSF's human capital management, NSF is developing a five-year
workforce strategic plan to address strategic human capital
management issues that must be submitted to the Office of
Management and Budget by July 20, 2001. NSF's performance report
did not explain its progress in resolving information security
challenges, but NSF indicated that it has internal management
controls that continually monitor data security.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-758
ACCNO: A01201
TITLE: National Science Foundation: Status of Achieving Key
Outcomes and Addressing Major Management Challenges
DATE: 06/15/2001
SUBJECT: Human resources utilization
Internal controls
Personnel management
Reporting requirements
Performance measures
Strategic planning
Information resources management
GPRA
Government Performance and Results Act
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-758
Report to the Ranking Minority Member, Committee on Governmental Affairs, U.
S. Senate
United States General Accounting Office
GAO
June 2001 NATIONAL SCIENCE FOUNDATION
Status of Achieving Key Outcomes and Addressing Major Management Challenges
GAO- 01- 758
Page 1 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
June 15, 2001 The Honorable Fred Thompson Ranking Minority Member Committee
on Governmental Affairs United States Senate
Dear Senator Thompson: As you requested, we reviewed the National Science
Foundation?s (NSF) fiscal year 2000 performance report and fiscal year 2002
performance plan required by the Government Performance and Results Act of
1993 (GPRA) to assess the agency?s progress in achieving selected key
outcomes that you identified as important mission areas for the agency. 1
These are the same outcomes we addressed in our June 2000 review of the
agency?s fiscal year 1999 performance report and fiscal year 2001
performance plan to provide a baseline by which to measure the agency?s
performance from year- to- year. 2 These selected key outcomes are:
NSF?s research funding awards lead to discoveries at and across the
frontier of science and engineering and NSF efficiently and effectively
administers research grants.
As agreed, using the selected key outcomes for NSF as a framework, we (1)
assessed the progress NSF has made in achieving these outcomes and the
strategies that NSF has in place to achieve them and (2) compared NSF?s
fiscal year 2000 performance report and fiscal year 2002 performance plan
with its prior year performance report and plan for these outcomes.
Additionally, we agreed to analyze how NSF addressed the major management
challenges, including the governmentwide highrisk areas of strategic human
capital management and information security, that we and NSF?s Inspector
General identified. Appendix I provides detailed information on how NSF
addressed these challenges.
1 This report is one of a series on the 24 Chief Financial Officers Act
agencies? fiscal year 2000 performance reports and fiscal year 2002
performance plans. 2 Observations on the National Science Foundation?s
Fiscal Year 1999 Performance Report and Fiscal Year 2001 Performance Plan
(GAO/ RCED- 00- 205R, June 30, 2000).
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
NSF reported that it made substantial progress in achieving its key
outcomes. While the planned strategies for achieving these key outcomes
generally are clear and reasonable, some are vague and do not identify the
specific steps for achieving the goals. Specifically:
Planned outcome: NSF?s research funding awards lead to discoveries at and
across the frontier of science and engineering. NSF judged itself successful
in achieving the scientific discoveries outcome goal on the basis of
assessments made by independent committees of scientific experts. The
performance report did not indicate how many committee reports NSF had used
in judging the success of this outcome or any of the other outcomes.
However, NSF officials told us that all of the committees? assessments were
used for the scientific discoveries outcome goal because the committees?
ratings were justified. NSF?s performance plan included a new ?means and
strategies for success? section for this outcome that includes strategies
that generally are clear and reasonable.
Planned outcome: NSF efficiently and effectively administers research
grants. NSF reported that it achieved most of its performance goals related
to the award and administration of research grants. For example, NSF met its
goals to electronically receive research funding proposals; however, it did
not achieve its goal to electronically process them because of technological
concerns about using electronic signatures. NSF?s performance plan generally
includes strategies for achieving its performance goals that appear to be
clear and reasonable. However, in some cases, NSF provides background
information rather than specific steps for effectively achieving the
performance goal, and it is unclear how the strategies will be used to
achieve the goal. The plan also does not provide information on the
strategic human capital management strategies to achieve this outcome.
NSF?s fiscal year 2000 performance report and fiscal year 2002 performance
plan reflect continued improvement compared with the prior year?s report and
plan. Specifically, both NSF?s Inspector General and we expressed concern
last year about the validity of assessments that independent scientific
committees made to judge NSF?s progress in achieving the scientific
discoveries outcome goal because NSF provided little information to support
the successful judgment it reported. In response, NSF improved the
evaluation form used by the scientific committees and contracted with
PricewaterhouseCoopers, LLP, to independently assess its performance results
by examining the committees? scores and justifications.
PricewaterhouseCoopers found the results to be valid and verifiable.
Furthermore, in response to our concern Results in Brief
Page 3 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
that last year?s plan did not clearly discuss the strategies for achieving
goals, NSF?s fiscal year 2002 performance plan includes a new section- means
and strategies for successful implementation- under each goal. However, NSF
does not identify the specific resources needed for achieving them.
Although the 2002 performance plan does not substantially address NSF?s
human capital management, NSF is developing a 5- year workforce strategic
plan to address strategic human capital management issues that must be
submitted to the Office of Management and Budget (OMB) by July 20, 2001.
NSF?s plan has also added a new goal to implement an agencywide information
security program in response to the Government Information Security Reform
Act. While NSF?s performance report did not explain its progress in
resolving information security challenges, NSF indicated that it has
internal management controls that continually monitor data security.
We provided NSF with a draft of this report for its review and comment. NSF
generally agreed with the draft and provided some specific comments, which
we have incorporated where appropriate.
GPRA is intended to shift the focus of government decisionmaking,
management, and accountability from activities and processes to the results
and outcomes achieved by federal programs. New and valuable information on
the plans, goals, and strategies of federal agencies has been provided since
federal agencies began implementing GPRA. Under GPRA, annual performance
plans are to clearly inform the Congress and the public of (1) the annual
performance goals for agencies? major programs and activities, (2) the
measures that will be used to gauge performance, (3) the strategies and
resources required to achieve the performance goals, and (4) the procedures
that will be used to verify and validate performance information. These
annual plans, issued soon after transmittal of the President?s budget,
provide a direct linkage between an agency?s longer- term goals and mission
and day- to- day activities. 3 Annual performance reports are to
subsequently report on the degree to which performance goals were met. The
issuance of the agencies? performance reports, due this year by March 31,
represents a new and potentially more substantive phase in the
implementation of GPRA- the opportunity to
3 The fiscal year 2002 performance plan is the fourth of these annual plans
under GPRA. Background
Page 4 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
assess federal agencies? actual performance for the prior fiscal year and to
consider what steps are needed to improve performance and reduce costs in
the future. 4
NSF?s mission is to promote the progress of science; to advance the national
health, prosperity, and welfare; and to secure the national defense. NSF
carries out its mission primarily by making merit- based grants and
cooperative agreements to individual researchers and groups in partnership
with colleges, universities, and other public and private institutions. For
fiscal year 2001, NSF has a budget of $4.4 billion and a staff of about
1,200 government employees to accomplish its mission.
Implementing GPRA has been a challenge for NSF, whose mission involves
funding research activities, because the substance and timing of research
outcomes are unpredictable and research results can be difficult to report
quantitatively. With OMB?s approval, NSF uses an alternative format- a
qualitative scale for the assessment of outcomes- for which it relies on
independent committees of scientific experts. These committees determine the
level of NSF?s success in achieving its goals. NSF uses quantitative goals
for its management and investment process goals.
This section discusses our analysis of NSF?s performance in achieving the
selected key outcomes, as well as the strategies it has in place-
particularly strategic human capital management 5 and information technology
strategies- for achieving these outcomes. In discussing these outcomes, we
have also provided information drawn from our prior work on the extent to
which NSF has provided assurance that the performance information it is
reporting is accurate and credible.
NSF, in its fiscal year 2000 performance report, states that it met its
discoveries outcome and cites numerous examples of its achievements in such
scientific fields as mapping the Arctic Ocean floor and extra- solar
4 The fiscal year 2000 performance report is the second of these annual
reports under GPRA. 5 Key elements of modern human capital management
include strategic human capital planning and organizational alignment;
leadership continuity and succession planning; acquiring and developing
staff whose size, skills, and deployment meet agency needs; and creating
results- oriented organizational cultures. Assessment of NSF?s
Progress and Strategies in Achieving Selected Key Outcomes
Discoveries at and Across the Frontier of Science and Engineering
Page 5 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
planetary discovery. NSF judged its performance as successful on the basis
of assessments by independent committees of scientific experts. In compiling
committee members? scores and aggregating their comments, NSF took into
account only those reports with substantive comments and ratings that were
clearly justified. NSF officials told us that, for the scientific
discoveries outcome goal, all of the committees judged NSF as successful in
achieving it and justified their assessments. However, the performance
report did not provide information on the specific numbers of reports it
included and excluded in reaching its judgments for this outcome or any of
the other outcomes. Furthermore, NSF discussed the independent scientific
committees? results for only one of the scientific discoveries five areas of
emphasis- namely, the balance of innovative, risky, and interdisciplinary
research area. Instead of providing a more complete analysis of the
scientific committees? assessments, NSF contracted with an external third
party- PricewaterhouseCoopers- to make an independent assessment of the
performance results. PricewaterhouseCoopers concluded that NSF?s fiscal year
2000 results were valid and verifiable.
NSF?s fiscal year 2002 performance plan included a new section on the means
and strategies for success related to this outcome that includes strategies
that generally are clear and reasonable. 6 To implement its outcome goal,
NSF has both (1) process strategies, such as supporting the most promising
ideas through merit- based grants and cooperative agreements, and (2)
program strategies, such as supporting programmatic themes identified as
areas of emphasis. However, NSF?s plan generally does not address key
components of strategic human capital management, although its ?people? and
?management? outcome goals include such human capital initiatives as
workforce diversity, an NSF Academy for workforce training, and a survey on
the work environment. NSF is in the process of developing a 5- year
strategic plan on its workforce needs that must be submitted to OMB by July
20, 2001. This strategic plan will guide NSF?s future effort in this area.
6 In NSF?s fiscal year 2002 performance plan, the discovery across the
frontier of science and engineering outcome goal has become one of six
performance goals under a broader
?ideas? outcome goal. NSF will consider itself successful if, in the
aggregate, reported results demonstrate significant achievement in the
majority of its performance indicators.
Page 6 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
NSF reported that it made substantial progress, achieving most of its
performance goals related to the award and administration of research
grants. While not listed as an outcome goal, the administration of grants
includes many of NSF?s management and investment process goals. 7 For
example, NSF exceeded by 21 percent one of its management performance goals-
to receive at least 60 percent of full grant proposal submissions
electronically through a new computer system called FastLane. NSF also
exceeded by 5 percent another management goal that at least 90 percent of
its funds will be allocated to projects reviewed by appropriate peers
external to NSF and selected through a merit- based competitive process. NSF
continued to miss one of its investment process goals- to process 70 percent
of proposals within 6 months of receipt- dropping from 58 percent to 54
percent in fiscal year 2000. As part of its review of NSF,
PricewaterhouseCoopers concluded that NSF?s fiscal year 2000 processes were
valid and verifiable and relied on sound business processes, system and
application controls, and manual checks of system queries to confirm the
accuracy of reported data.
NSF?s fiscal year 2002 performance plan generally includes strategies for
achieving NSF?s performance goals that appear to be clear and reasonable.
However, in some cases, the strategies are vague, and how NSF will use them
to achieve its performance goals is unclear. For example, one of NSF?s three
strategies for identifying best management practices for its large
infrastructure projects is to ensure input from members of the external
community who build, operate, and utilize research facilities. Furthermore,
while NSF has strategies for the process of funding awards, it does not
generally address the oversight needs to ensure that funding recipients meet
the awards? requirements. NSF?s 5- year workforce strategic plan is
addressing concerns regarding the management of a growing portfolio of
program activities with relatively flat personnel levels- a key issue for
developing strategic human capital management strategies.
7 These goals include the proposal and award processes, award portfolio,
award oversight and management, business practices, and human resources and
workplace. Efficiency and
Effectiveness of Administration of Research Grants
Page 7 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
For the selected key outcomes, this section describes major improvements or
remaining weaknesses in NSF?s (1) fiscal year 2000 performance report in
comparison with its fiscal year 1999 report and (2) fiscal year 2002
performance plan in comparison with its fiscal year 2001 plan. It also
discusses the degree to which the agency?s fiscal year 2000 report and
fiscal year 2002 plan address concerns and recommendations by NSF?s
Inspector General.
NSF improved its fiscal year 2000 performance report, making major changes
to address the weaknesses we reported in the prior year?s performance
report. Our prior year?s review noted that NSF did not discuss either its
reasons for falling short of a performance goal or its strategies for
attaining the goal in the future. NSF?s 2000 report corrected this weakness.
For example, regarding the technology- related goal to submit, review, and
process proposals electronically, the report states that the reason for not
achieving the goal was due to the technological, financial, and legal issues
related to electronic signatures. The strategy for addressing the
technological issue was to demonstrate the paperless review capability by
conducting 10 pilot paperless projects in 2001 that manage the review
process in an electronic environment. We also questioned the quality of the
information in the 1999 performance report, noting that it provided
virtually no assurance that the information was credible. As mentioned
earlier, NSF contracted with PricewaterhouseCoopers to review aspects of its
GPRA data collection efforts and its performance assessment results.
PricewaterhouseCoopers found no basis for questioning the integrity of the
results.
NSF can improve its future reports in several ways. The results of the
independent committees? reviews would benefit from more detailed
information, such as including all of the areas of emphasis and the results.
In addition, last year, we noted that the 1999 performance report did not
describe NSF?s financial role in the examples of scientific successes
presented. Such information, we said, would help to judge the extent of
NSF?s role in achieving these successes. NSF officials maintain that
determining NSF?s financial role in these successes would be extremely
difficult and would take a considerable effort. NSF officials told us that
the Comparison of NSF?s
Fiscal Year 2000 Performance Report and Fiscal Year 2002 Performance Plan
With the Prior Year?s Report and Plan for Selected Key Outcomes
Comparison of Performance Reports for Fiscal Years 1999 and 2000
Page 8 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
successes they identified for this outcome were primarily due to NSF awards.
That statement would have been useful in assessing the 2000 performance
report.
NSF made improvements to its fiscal year 2002 performance plan. For example,
last year, we reported that the performance plan contained little useful
information about NSF?s intended strategy to achieve its goals, including a
discussion of the problems. The 2002 plan includes a new section on the
means and strategies for success. For example, for its new goal of award
oversight and management, NSF will ensure that the internal committee
reviewing the oversight activities for large infrastructure projects has
broad disciplinary expertise and experience in managing facilities. As
previously mentioned, NSF is also addressing data quality concerns,
providing confidence that future performance information will be credible.
Furthermore, NSF revised its outcome goal such that it does not have to
succeed in demonstrating significant achievement in discoveries that advance
the frontiers of science, engineering, or technology. Rather, discoveries is
now one of six performance indicators for which NSF will consider itself
successful when a majority is achieved. Last year, we also reported that the
strategies for achieving the goals were not clearly discussed. NSF includes
a new section on the means and strategies for success under each goal.
NSF can improve its future performance plans by addressing its resource
needs. Last year, we noted that the plan did not clearly discuss the
resources for achieving the goals or the specific links between the
resources and the areas of emphasis. The 2002 performance plan still does
not do so. As discussed earlier, NSF?s 5- year workforce strategic plan is
expected to address human capital issues, providing a basis for addressing
this issue in next year?s performance plan.
GAO has identified two governmentwide high- risk areas: strategic human
capital management and information security. Regarding strategic human
capital management, we found that NSF?s performance plan generally did not
have goals and measures related to strategic human capital management, and
NSF?s performance report did not explain its progress in resolving strategic
human capital management challenges. However, as mentioned earlier, NSF is
developing a 5- year workforce strategic plan. With respect to information
security, we found that NSF?s performance plan had a goal and measures
related to information security. While NSF?s performance report did not
explain its progress in resolving information Comparison of
Performance Plans for Fiscal Years 2001 and 2002
NSF?s Efforts to Address Its Major Management Challenges Identified by GAO
Page 9 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
security challenges, it did indicate that NSF has internal management
controls that continually monitor data security.
We provided NSF and the Office of the Inspector General with a draft of this
report for their review and comment. We met with NSF officials, including
the Chief Information Officer and the Inspector General. The NSF officials
generally agreed with the report. However, they noted that the fiscal year
2000 performance report did not respond to some of the Inspector General?s
management challenges primarily because these challenges were identified in
a November 30, 2000, letter. The Inspector General agreed that some of these
management challenges were new. The NSF officials recognize that certain
challenges not in the current plan and report are important, and they noted
that these challenges are being addressed through internal management
controls and processes. They added that NSF will continue to consider these
challenges for incorporation in future performance plans. The NSF officials
also provided technical clarifications, which we incorporated as
appropriate.
Our evaluation was generally based on the requirements of GPRA, the Reports
Consolidation Act of 2000, guidance to agencies from OMB for developing
performance plans and reports (OMB Circular A- 11, Part 2), previous reports
and evaluations by us and others, our knowledge of NSF?s operations and
programs, GAO?s identification of best practices concerning performance
planning and reporting, and our observations on NSF?s other GPRA- related
efforts. We also discussed our review with NSF officials in the Office of
Information and Resource Management; the Office of Budget, Finance, and
Award Management; the Office of Integrative Activities; and the Office of
Inspector General.
The agency outcomes that were used as the basis for our review were
identified by the Ranking Minority Member of the Senate Committee on
Governmental Affairs as important mission areas for NSF and do not reflect
the outcomes for all of NSF?s programs or activities. The major management
challenges confronting NSF, including the governmentwide high- risk areas of
strategic human capital management and information security, were identified
by (1) our January 2001 high- risk update and (2) NSF?s Office of Inspector
General in November 2000. We did not independently verify the information
contained in the performance report and plan, although we did draw from
other GAO work in assessing the validity, reliability, and timeliness of
NSF?s performance data. We Agency Comments
Scope and Methodology
Page 10 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
conducted our review from April through June 2001 in accordance with
generally accepted government auditing standards.
As arranged with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days after
the date of this letter. At that time, we will send copies to appropriate
congressional committees; the Director, NSF; and the Director of OMB. Copies
will also be made available to others on request.
If you or your staff have any questions, please call me at (202) 512- 3841.
Key contributors to this report were Richard Cheston, Alan Stapleton,
Elizabeth Johnston, and Sandy Joseph.
Sincerely yours, Jim Wells Director, Natural Resources and
Environment
Appendix I: Observations on NSF?s Efforts to Address Its Management
Challenges
Page 11 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
The following table discusses the major management challenges confronting
the National Science Foundation (NSF), including the governmentwide high-
risk areas of strategic human capital management and information security,
identified by our January 2001 high- risk update and NSF?s Office of
Inspector General (IG) in November 2000. The first column of the table lists
the management challenges identified by our office and NSF?s IG. The second
column discusses NSF?s progress, as discussed in its fiscal year 2000
performance report, in resolving these challenges. The third column
discusses the extent to which NSF?s fiscal year 2002 performance plan
includes performance goals and measures to address each of these challenges.
We found that while the fiscal year 2000 performance report discussed NSF?s
progress in resolving most of its major challenges, it did not discuss NSF?s
progress in resolving the following challenges: (1) addressing strategic
human capital management issues regarding strategic human capital planning
and organizational alignment, leadership continuity and succession planning,
and creating results- oriented organizational cultures; (2) developing
appropriate data security controls to reduce the ever increasing risk of
unauthorized access; (3) developing a more coherent award administration
program that ensures that grantees comply with NSF?s award requirements; (4)
ensuring that NSF grantees meet their cost- sharing obligations; and (5)
providing the science, operations, and logistics support needed to manage
the U. S. Antarctic Program. Of NSF?s 10 major management challenges, its
fiscal year 2002 performance plan (1) had goals and measures that were
directly related to 5 of the challenges; (2) had goals and measures that
were indirectly applicable to 1 challenge; (3) had no goals and measures
related to 1 challenge but discussed strategies to address it; and (4) did
not have goals, measures, or strategies to address 3 challenges. Appendix I:
Observations on NSF?s Efforts to
Address Its Management Challenges
Appendix I: Observations on NSF?s Efforts to Address Its Management
Challenges
Page 12 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
Table 1: Major Management Challenges Major management challenge
Progress in resolving major management challenge as discussed in the fiscal
year 2000 performance report Applicable goals and measures in the
fiscal year 2002 performance plan GAO- designated governmentwide high -risk
area
Strategic Human Capital Management: We have identified shortcomings at many
agencies involving key elements of modern strategic human capital
management, including (1) strategic planning and organizational alignment;
(2) leadership continuity and succession planning; (3) acquiring and
developing staff whose size, skills, and deployment meet agency needs; and
(4) creating results- oriented organizational cultures.
(NSF?s IG identified workforce planning and training issues as a management
challenge primarily because NSF?s staff have not grown in relation to the
funds available for research awards. The IG believes that this challenge is
a significant problem that needs management?s attention.)
1. NSF did not address this issue in its 2000 report. 2. NSF did not address
this issue in its
2000 report. 3. NSF met its management goal to
improve staff diversity by increasing the number of new hires from
underrepresented groups in science and engineering. NSF also met its
management goal that at least 80 percent of its staff receive practice in
using key modules of FastLane. 4. NSF did not address this issue in its
2000 report. New IG challenge for 2001. As discussed above, at least 80
percent of NSF?s staff have received practice in using key modules of
FastLane, a new electronic system to facilitate receiving and evaluating
research funding proposals.
1. NSF did not address this issue in its 2002 plan. 2. NSF did not address
this issue in its
2002 plan. 3. Although NSF has not addressed this
element directly, it has extended its performance goal to improve staff
diversity by increasing the number of new hires from underrepresented groups
in science and engineering. NSF also added a new performance goal to
establish an internal NSF academy to promote continual learning for NSF
staff. 4. NSF has addressed aspects of this
issue as part of the GPRA process. NSF currently is preparing a 5- year
workforce strategic plan required by the Office of Management and Budget
(OMB) for submission by July 20, 2001. Information/ Data Security: Our
January 2001 high- risk update noted that the agencies? and governmentwide
efforts to strengthen information security have gained momentum and
expanded. However, recent audits continue to show that federal computer
systems are riddled with weaknesses that make them highly vulnerable to
computer- based attacks and place a broad range of critical operations and
assets at risk of fraud, misuse, and disruption.
(NSF?s IG similarly identified data security and controls as a management
challenge. The IG believes that NSF needs to monitor this issue because of
its importance to NSF?s mission.)
New IG challenge for 2001. NSF did not address the information security
management challenge in its performance report. However, NSF?s IG has
indicated that NSF is taking effective steps to respond to this management
challenge and that NSF has internal management controls that continually
monitor data security.
According to the IG, NSF needs to monitor this issue because NSF?s automated
computer systems (1) are critical for managing over $4 billion in funds and
for processing grant proposals and (2) have experienced several recent
computer intrusions that could have compromised users? accounts and
passwords and allowed unauthorized access to proprietary scientific data.
The 2002 plan includes a new goal to implement an NSF- wide security program
in response to the government information security reform provisions of the
fiscal year 2001 National Defense Authorization Act.
Means and strategies to accomplish this goal include (1) conducting risk
assessments of all mission- critical systems and certifying the systems for
operation, (2) publishing policies documenting the security program, (3)
establishing a security management structure and assigning security
responsibilities, and (4) incorporating security- related issues into
personnel policies and providing ongoing training of staff.
The proposed indicators for success are (1) documented risk assessments and
certifications, (2) policies disseminated and maintained on an internal
security Website, (3) organizational and position descriptions, and (4)
documented personnel policies and formal training.
Appendix I: Observations on NSF?s Efforts to Address Its Management
Challenges
Page 13 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
Major management challenge Progress in resolving major
management challenge as discussed in the fiscal year 2000 performance report
Applicable goals and measures in the
fiscal year 2002 performance plan IG- designated major management challenges
Assure that FastLane is as user- friendly and reliable as possible. (The IG
believes that NSF needs to monitor this issue because of its importance to
NSF?s mission.)
NSF did not directly address the IG?s management challenge that FastLane be
user- friendly and reliable. However, NSF identified two management goals
for increasing the use of FastLane in its grant award process: (1) NSF
exceeded its goal that at least 60 percent of full proposal submissions be
transmitted electronically through FastLane. (2) NSF did not achieve its
goal to have the technological capability for a paperless review process
because of electronic signatures issues.
NSF revised its performance goals regarding FastLane to state that it will
continue to advance the role of
?e- business? in review, award, and management processes. NSF plans to
extend its goal for a paperless review process to the establishment of an
?electronic jacket.? It also plans to double to 20 the number of paperless
projects in the competitive review process.
Operate a viable, credible, and efficient merit review system. (The IG
believes that NSF needs to monitor this issue because of its importance to
NSF?s mission.)
NSF identified two investment process goals for using merit review in its
proposal award process: (1) NSF exceeded its first goal that at least 90
percent of its funds be allocated to projects reviewed by appropriate peers
external to NSF and selected through a merit- based competitive process. (2)
NSF did not achieve its second goal that expert reviewers and NSF program
officers address the two merit review criteria- quality and impact- mainly
because reviewers did not fully address the proposals? impact. NSF was
judged successful in achieving this goal in 20 of 58 external evaluator
reports that rated programs on their use of both criteria. In addition, NSF
achieved two of five other investment process goals for identifying and
addressing customer dissatisfaction with its merit review and complaint
systems.
Although NSF?s performance on the second goal dropped this year, NSF noted
that the assessments included projects that had been reviewed before the two
criteria were implemented. In fiscal year 2000, the National Academy of
Public Administration studied the impact of the new merit review criteria
and concluded that it was too soon to make valid judgments about their
impact and effectiveness. However, the academy made several suggestions,
such as improving quantitative measures and performance indicators. NSF said
it would act upon the suggestions and make other computer- related
improvements beginning in fiscal year 2001.
NSF retained its two merit review goals with minor revisions to (1) conform
with OMB?s governmentwide definition of merit review for basic and applied
research funds and (2) split the second goal into separate goals for merit
reviewers and NSF program officers. In addition, NSF retained customer
service goals aimed at improving the time available for preparing proposals
and reducing the time taken for making award decisions.
Appendix I: Observations on NSF?s Efforts to Address Its Management
Challenges
Page 14 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
Major management challenge Progress in resolving major
management challenge as discussed in the fiscal year 2000 performance report
Applicable goals and measures in the
fiscal year 2002 performance plan
Ensure that NSF grantees meet their costsharing obligations. (The IG
believes that this challenge is a significant problem that needs management
attention.)
NSF did not address this issue in its 2000 report. NSF did not address this
issue in its 2002
plan. Develop a more coherent award administration program that ensures that
grantees comply with NSF?s award requirements. (The IG believes that this
challenge is a significant problem that needs management attention.)
New IG challenge for 2001. NSF identified one management goal for improving
its oversight of grant awards: NSF exceeded its goal that at least 85
percent of all project reports be submitted through its new electronic
project reporting system.
NSF?s efforts to improve award oversight focus on its large infrastructure
projects. NSF added a performance goal to identify best management practices
for constructing and operating its large infrastructure projects.
Improve management controls of large infrastructure projects, including
tighter cost and schedule oversight. (The IG believes that this challenge is
a significant problem that needs management?s attention.)
New IG challenge for 2001. NSF identified two investment process goals for
the construction and upgrade of large research facilities: (1) NSF achieved
its goal of keeping construction and upgrades within 110 percent of the
estimated annual expenditure plan. (2) NSF did not achieve its goal of
keeping construction and upgrades within 110 percent of the estimated annual
schedule time frames for major components. NSF also had an investment
process goal for minimizing operating time lost because of unscheduled
downtime.
The IG, in a December 2000 report on the twin telescopes Gemini Project,
recommended that NSF develop policies and procedures specifically focused at
managing large capital projects.
NSF revised its performance goals for large infrastructure projects keeping
construction and upgrades within 110 percent of the annual expenditure plan
and within 110 percent of all major annual schedule milestones for 90
percent of facilities. In addition, NSF submitted a preliminary plan to OMB
in April 2001 and expects to submit its final plan in September 2001. This
plan will document its approach for the costing, approval, and oversight of
major facility projects.
Provide the science, operations, and logistics support needed to manage the
U. S. Antarctic Program. (The IG believes that NSF needs to monitor this
issue because of its importance for scientists? health and safety.)
NSF did not address this issue in its 2000 report. NSF did not address this
issue in its 2002
plan. Foster a diverse scientific workforce nationwide. (The IG believes
that NSF needs to monitor this issue because of its importance to NSF?s
mission.)
New IG challenge for 2001. This is one of NSF?s five outcome goals for 2000.
NSF?s performance was judged successful in the aggregate with regard to
achieving a globally oriented workforce. However, NSF was not fully
successful with regard to achieving diversity or increasing participation of
underrepresented groups.
In addition, NSF identified two investment process goals for fostering a
diverse scientific workforce: (1) NSF did not
NSF?s 2002 plan has three strategic outcomes, including to develop ?a
diverse, internationally competitive and globally engaged workforce of
scientists, engineers, and well- prepared citizens.?
NSF dropped its performance goal of awarding a percentage of competitive
research grants to new investigators. NSF revised its performance goal of
increasing the participation of underrepresented groups in NSF?s proposal
review activities
Appendix I: Observations on NSF?s Efforts to Address Its Management
Challenges
Page 15 GAO- 01- 758 NSF?s Status of Achieving Key Outcomes
Major management challenge Progress in resolving major
management challenge as discussed in the fiscal year 2000 performance report
Applicable goals and measures in the
fiscal year 2002 performance plan
achieve its goal of awarding at least 30 percent of its competitive research
grants to new investigators- new investigators received only 28 percent of
the awards. (2) NSF achieved its goal of identifying mechanisms to increase
the number of women and underrepresented minorities in its proposal
applicant pool.
to state that participation levels of fiscal year 2002 will exceed those of
fiscal year 2001.
Continue to improve the quality of performance measurement data for GPRA.
(The IG believes that NSF needs to monitor this issue because of its
importance to NSF?s mission.)
Discussed under the verification and validation section of the report.
Discussed under the verification and
validation section of the plan.
(360067)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***