Results Act: Observations on the National Science Foundation's Draft
Strategic Plan (Correspondence, 07/11/97, GAO/RCED-97-203R).

Pursuant to a congressional request, GAO reviewed the draft strategic
plan for the National Science Foundation (NSF), focusing on: (1) whether
it fulfills the requirements of the Results Act and its overall quality;
(2) whether it reflects NSF's key statutory authorities; (3) whether it
reflects interagency coordination for crosscutting programs, activities,
or functions that are similar or complementary to other federal
agencies'; and (4) whether NSF's data and information systems are
providing adequate information for measuring results.

GAO noted that: (1) NSF's draft strategic plan is incomplete and not
specific enough to allow the Congress to evaluate whether the agency's
goals are achievable; (2) the draft strategic plan addresses aspects of
five of six required elements; (3) however, three of the five elements
are not yet complete, goals and objectives, strategies for achieving
goals, and how program evaluation was used; (4) furthermore, NSF did not
include one key element, external factors that could affect the
achievement of the plan's goals; (5) because the plan is incomplete, the
Congress is missing critical information for its consultations with NSF;
(6) NSF's draft strategic plan appears to reflect the consideration of
its key statutory authority, the National Science Foundation Act of
1950, as amended; (7) however, NSF is subject to other statutes related
to its core functions which broaden the scope of its responsibilities;
(8) NSF could provide useful information by describing its
responsibilities under these other statutes when its plan includes goals
and objectives based on them; (9) NSF's draft strategic plan
acknowledges the crosscutting nature of its work; (10) however, the
draft plan does not show evidence of interagency coordination; (11)
while the strategic plan emphasizes the importance of NSF's many
partners in the research and education enterprise, it does not identify
who these partners are or provide sufficient information to determine
the extent to which NSF's and its partners' functions are duplicative or
overlapping; (12) while GAO has not analyzed NSF's data and information
systems, inadequacies in both financial information and information
technology at NSF have been identified by NSF's Office of Inspector
General (OIG); (13) NSF's OIG and independent public accounting firm
completed the first audit of NSF's consolidated, agencywide fiscal year
1996 Statement of Financial Position (balance sheet); (14) except for
inadequate documentation to support the reported amounts for property,
plant, and equipment, the auditors concluded that NSF's assets,
liabilities, and net position are reliable; (15) in addition, the
auditors found that NSF had not yet met the requirements in the Chief
Financial Officers (CFO) Act to develop an integrated agency accounting
and financial management system that provides for reporting cost
information and the systematic measurement of performance; and (16)
furthermore, linkages between NSF's technology and its programs'
missions and goals were not included in its strategic plan.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  RCED-97-203R
     TITLE:  Results Act: Observations on the National Science 
             Foundation's Draft Strategic Plan
      DATE:  07/11/97
   SUBJECT:  Strategic planning
             Agency missions
             Research program management
             Program evaluation
             Interagency relations
             Congressional/executive relations
             Information technology
             Financial management systems
             Information systems

             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER



September 1997


GAO/RCED-97-203R

NSF's Draft Strategic Plan

(141077)


Abbreviations
=============================================================== ABBREV

  BOP - Federal Bureau of Prisons
  CFO - Chief Financial Officers
  DEA - Drug Enforcement Administration
  DOJ - Department of Justice
  FBI - Federal Bureau of Investigation
  INS - Immigration and Naturalization Service
  NPR - National Performance Review
  NSF - x
  OMB - Office of Management and Budget
  OIG - x
  DOE - x
  R&D - x

Letter
=============================================================== LETTER


B-277424

July 11, 1997

The Honorable Richard K.  Armey
Majority Leader
House of Representatives

The Honorable John Kasich
Chairman, Committee on the Budget
House of Representatives

The Honorable Dan Burton
Chairman, Committee on Government
 Reform and Oversight
House of Representatives

The Honorable Bob Livingston
Chairman, Committee on Appropriations
House of Representatives

Subject:  Results Act:  Observations on the National Science
Foundation's Draft Strategic Plan

On June 12, 1997, you asked us to review the draft strategic plans
submitted by the Cabinet departments and selected major agencies for
consultation with the Congress as required by the Government
Performance and Results Act of 1993 (the Results Act).  This report
is our response to that request concerning the draft strategic plan
for the National Science Foundation (NSF). 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
------------------------------------------------------------ Letter :1

We agreed to review NSF's draft plan and assess (1) whether it
fulfills the requirements of the Results Act and to provide our views
on its overall quality; (2) whether it reflects NSF's key statutory
authorities; (3) whether it reflects interagency coordination for
crosscutting programs, activities, or functions that are similar or
complementary to other federal agencies'; and (4) whether NSF's data
and information systems are providing adequate information for
measuring results.  You also asked us to assess whether the draft
plan addresses the management problems we have previously identified. 
Because we have not reported on NSF's management problems in the
past, this report will not address that issue. 

We reviewed the most recent draft strategic plan--dated June 9,
1997--provided to congressional committees.  Our overall assessment
of NSF's draft strategic plan was generally based on our knowledge of
NSF's programs and operations, our discussions with NSF's Assistant
to the Director for Science Policy and Planning and the Deputy
Director, Office of Legislative and Public Affairs, and other
existing information available at the time of our assessment. 

Specifically, the criteria we used to determine whether NSF's draft
strategic plan complies with the requirements of the Results Act were
the Results Act itself and the Office of Management and Budget's
(OMB) guidance on developing the plans (Circular A-11, Part 2).  To
make judgments about the overall quality of the plan and its
components, we used our May 1997 guidance for congressional review of
the plans (GAO/GGD-10.1.16) as a tool.  To determine whether the plan
contains information on interagency coordination, we relied on our
general knowledge of federal science agencies' operations and
programs and the results of our previous reports.  In determining
whether NSF's draft strategic plan reflects its major statutory
responsibilities, as you requested, we coordinated our review with
the Congressional Research Service.  To determine whether NSF has
adequate systems in place to provide reliable information on
performance, we relied on information provided by the Foundation's
Office of Inspector General (OIG). 

It is also important to recognize that NSF's final plan is not due to
the Congress and OMB until September 1997.  Furthermore, the Results
Act anticipated that it may take several planning cycles to perfect
the process and that the final plan will continue to be refined as
future planning cycles occur.  Thus, our comments are a snapshot of
the plan at this time.  We recognize that developing a strategic plan
is a dynamic process and that NSF is continuing to work to revise the
draft with input from OMB, congressional staff, and other
stakeholders. 

Our work was performed in June and July 1997.  We obtained comments
on a draft of this report from NSF.  Its comments are in enclosure I. 


   BACKGROUND
------------------------------------------------------------ Letter :2

NSF is an independent federal agency created by the National Science
Foundation Act of 1950 (P.L.  81-507).  Its aim is to promote and
advance scientific and engineering progress in the United States. 
The idea of such a foundation was an outgrowth of the important
contributions made by science and technology during World War II. 
Since that time, NSF has been responsible for the overall well-being
of science and engineering across all disciplines.  In contrast,
other federal agencies support research focused on specific missions,
such as health or defense.  NSF is also committed to help ensure the
nation's supply of scientists, engineers, and science educators
through the financial support of education and research. 

NSF funds research and education in the areas of science and
engineering through grants, contracts, and cooperative agreements to
more than 2,000 colleges, universities, and other research
institutions.  NSF receives about 53,000 requests for funding (both
new and renewal projects) each year and makes about 20,000 awards. 
The agency operates no laboratories itself but does support National
Research Centers, certain oceanographic vessels, and Antarctic
research stations.  It also supports cooperative research between
universities and industry and U.S.  participation in international
scientific efforts. 

NSF has been involved in strategic planning efforts since 1992, when
the National Science Board Commission on the Future of the NSF was
established.  The board established national science and technology
goals that later became the basis for NSF's 1994 strategic plan.  The
plan was developed with input and support from in-house staff and
advisory bodies.  NSF views the Results Act plan as its
implementation strategy for the goals set forth in its 1995 strategic
plan. 

The Results Act requires that an agency's strategic plan contain the
following six critical elements:  (1) a comprehensive mission
statement; (2) agencywide long-term goals and objectives for all
major functions and operations; (3) approaches (or strategies) and
the various resources needed to achieve the goals and objectives; (4)
the relationship between the long-term goals and objectives and the
annual performance goals; (5) an identification of key factors,
external to the agency and beyond its control, that could
significantly affect the achievement of the strategic goals; and (6)
a description of how program evaluations were used to establish or
revise strategic goals and a schedule for future program evaluations. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :3

NSF's draft strategic plan is incomplete and not specific enough to
allow the Congress to evaluate whether the agency's goals are
achievable.  The draft strategic plan addresses aspects of five of
six required elements.  However, three of the five elements are not
yet complete--goals and objectives, strategies for achieving goals,
and how program evaluation was used.  Furthermore, NSF did not
include one key element--external factors that could affect the
achievement of the plan's goals.  Because the plan is incomplete, the
Congress is missing critical information for its consultations with
NSF. 

NSF's draft strategic plan appears to reflect the consideration of
its key statutory authority, the National Science Foundation Act of
1950, as amended.  We note, however, that NSF is subject to other
statutes related to its core functions; these other statutes broaden
the scope of its responsibilities.  NSF could provide useful
information by describing its responsibilities under these other
statutes when its plan includes goals and objectives based on them. 

NSF's draft strategic plan acknowledges the crosscutting nature of
its work.  However, the draft plan does not show evidence of
interagency coordination.  While the strategic plan emphasizes the
importance of NSF's many partners in the research and education
enterprise, it does not identify who these partners are or provide
sufficient information to determine the extent to which NSF's and its
partners' functions are duplicative or overlapping. 

While we have not analyzed NSF's data and information systems,
inadequacies in both financial information and information technology
at NSF have been identified by NSF's Office of Inspector General. 
NSF's OIG and an independent public accounting firm completed the
first audit of NSF's consolidated, agencywide fiscal year 1996
Statement of Financial Position (balance sheet).  Except for
inadequate documentation to support the reported amounts for
property, plant, and equipment, the auditors concluded that NSF's
assets, liabilities, and net position are reliable.  (No audit was
performed on the 1996 Statement of Operations and Changes in Net
Position.  Therefore, the reliability of NSF's revenue and expense
information is uncertain.  It is our understanding that this
information will be audited for fiscal year 1997.) In addition, the
auditors found that NSF had not yet met the requirements in the Chief
Financial Officers (CFO) Act to develop an integrated agency
accounting and financial management system that provides for
reporting cost information and the systematic measurement of
performance.  Furthermore, linkages between NSF's technology and its
programs' missions and goals were not included in its strategic plan. 


   NSF'S STRATEGIC PLAN PARTLY
   FULFILLS THE REQUIREMENTS OF
   THE RESULTS ACT
------------------------------------------------------------ Letter :4

While five of the six required elements are addressed, at least four
of them need further development, and the sixth element--key external
factors--is not included in the current draft.  NSF's draft plan is
not specific enough to allow the Congress to evaluate whether NSF's
goals and objectives are achievable.  In its plan, the Foundation
describes itself as an investment agent, setting its mission, goals,
and supporting strategies toward the performance of its investment
portfolios through grants, contracts, and cooperative agreements. 
The draft plan indicates that NSF's annual performance goals for
results will appear as descriptive standards, as allowed by the
Results Act's option to set performance goals in an alternative
format.  The draft plan also indicates that the performance goals for
process will be largely quantitative.  A brief section on the use of
program evaluation in establishing strategic goals is included.  NSF
also included a section in its plan that discusses the processes and
essential coordination functions managed by NSF staff that must
operate effectively if NSF's outcome goals are to be met.  In this
section, the Foundation addresses other factors critical to the
successful management of the agency. 


      NSF ADDRESSES REQUIREMENTS
      FOR ITS MISSION STATEMENT
---------------------------------------------------------- Letter :4.1

NSF's mission statement focuses on strengthening the nation's
potential for research and education in science and engineering. 
Included with its mission statement are those areas initiated and
supported by NSF pursuant to its core statutory responsibilities. 
These are:  basic scientific and engineering research; programs
strengthening scientific and engineering research potential; science
and engineering education; and an information and policy base for
science and engineering.  The section on NSF's mission statement and
the supporting text is comprehensive and results-oriented and
fulfills public needs and statutory responsibilities.  Although
brief, it defines the basic purpose of the agency, focusing
particularly on its core programs and activities. 


      GOALS AND OBJECTIVES ARE
      DEFINED IN THE DRAFT PLAN
---------------------------------------------------------- Letter :4.2

The draft plan sets out three overarching goals intended to guide
NSF's strategic direction, as well as four broadly worded outcome
goals.  The overarching goals do not represent the unique functions
and operations of NSF.  The draft plan states that these goals cannot
be achieved unilaterally.  According to NSF's Deputy Director, Office
of Legislative and Public Affairs, the overarching goals were
developed by the National Science Board 3 years ago in the context of
a broader effort to establish goals for all science agencies. 

The outcome goals set out the long-term programmatic, policy, and
management goals to be accomplished through NSF's program office
investments.  Because many of these goals are not expressed in a
measurable form, it is unclear whether the Foundation and the
Congress will be able to assess whether the goals are achieved. 
While in some cases these goals provide the immediate context for
NSF's investment decisions, in other cases they do not.  For example,
one goal is to encourage "improved achievement in the essential
mathematics and science skills needed by all Americans." The
achievement of this goal is targeted at results over which NSF has a
reasonable degree of influence--instructional materials developed
through NSF awards, for example.  However, another stated goal seeks
"discoveries at and across the frontier of science and engineering."
It is unclear how NSF will be able to assess whether this goal is
achieved.  When goals are defined in a way that precludes a direct,
future determination of achievement, the performance goals and
indicators in the annual performance plan should be used to provide
the basis for the assessment, according to OMB Circular A-11. 
Therefore, NSF may want to elaborate on how it is achieving certain
goals in its performance plan. 


      STRATEGIES TO ACHIEVE NSF'S
      GOALS LACK PRECISION
---------------------------------------------------------- Letter :4.3

NSF's plan provides some general dates for achieving its goals but
does not provide the underlying assumptions, projections, or a
schedule for initiating or completing significant actions.  Also
lacking is the process for communicating goals and objectives
throughout the agency and for assigning accountability to managers
and staff for the achievement of goals. 

Each of NSF's four outcome goals is supported by three to four
investment strategies.  For example, the first goal is to encourage
"discoveries at and across the frontier of science and engineering."
The draft plan identifies four key investment strategies to meet this
goal.  The first strategy is to "seek out the most innovative ideas,
actively shaping the portfolio in ways that influence capabilities
for the future." NSF officials agree that it would be helpful if the
language were more specific.  One way to achieve specificity may be
to link the stated outcome goals to the relevant statutory objective. 
Although the Results Act does not require such a linkage, including
such information may help NSF management to better formulate its
goals.  It could also facilitate constructive consultation between
the agency and the congressional oversight committees about the
agency's goals and its priorities for achieving them. 

Although not discussed in detail in NSF's plan, a single investment
can work toward many of the outcome goals articulated.  For example,
goal three is to achieve "a diverse, productive globally-oriented
workforce of scientists and engineers." NSF's Faculty Early Career
Development (Career) program for junior-level faculty members at
colleges and universities encourages them to contribute to research
and education early in their careers.  The Career program supports
research that leads to linking discovery and learning, one of the
supporting strategies for NSF's first goal.  Moreover, according to
NSF, frequently the most original and innovative ideas come from
young scientists whose ideas are often of interest to industry and to
other agencies.  Thus, investments made through this program may also
support the outcome goals for discoveries and connections. 


      THE PLAN ADDRESSES THE
      RELATIONSHIP BETWEEN NSF'S
      LONG-TERM GOALS AND ANNUAL
      PERFORMANCE GOALS
---------------------------------------------------------- Letter :4.4

As required by OMB's guidance, NSF's draft plan outlines the type of
performance goals to be included in its performance plan. 
Performance goals for NSF will include (1) quantitative goals
measuring the process of investing in or facilitating research and
education projects, as well as performance goals for facilities
operations, and (2) descriptive standards for assessing the results
of NSF's investments at an aggregate level.  The relationship between
the performance goals and NSF's outcome goals is described briefly as
is the relevance and use of performance goals in helping to determine
the achievement of general goals and objectives.  In addition, the
plan addresses the linkage between NSF's budget and its annual
performance plan.  Performance plans for an upcoming year are to be
developed in the light of the analysis of past performance, an
assessment of how recent or projected changes in the investment
portfolio will influence future performance, and how the portfolio
fits with the outcome goals identified in the strategic plan. 

As we have reported in the past, the very nature of the innovative
process makes measuring the performance of science-related activities
difficult, since outcomes may not be seen for many years.\1 An NSF
official told us that the alternative form of performance assessment
for the agency's annual performance plan should provide some basis
for assessing whether NSF's goals have been met.  According to
information on a 1995 NSF discussion paper,\2 the ultimate outcomes
of NSF's programs, such as new technologies or improved quality of
life, are too far from NSF's sphere of influence to be used for
setting performance goals.  As a result, the discussion paper argues
that NSF would need to manage toward intermediate outcomes, such as
major new conceptual frameworks, enduring partnerships, and cadres of
trained technical talent, all of which involve qualitative elements
not lending themselves to quantitative indicators.  NSF has focused
its efforts on developing outcome indicators, as encouraged by the
Results Act. 


--------------------
\1 Managing for Results:  Key Steps and Challenges in Implementing
GPRA in Science Agencies (GAO/T-GGD/RCED-96-214, July 10, 1996). 

\2 National Science Foundation Case Study:  Development and Use of
Outcome Information by the National Science Foundation (Oct.  25,
1996). 


      NSF'S PLAN DOES NOT DISCUSS
      EXTERNAL FACTORS THAT COULD
      AFFECT GOAL ACHIEVEMENT
---------------------------------------------------------- Letter :4.5

NSF's draft plan neither addresses key external factors nor describes
how achieving particular goals could be affected by external factors. 
Such factors as the extent to which schools and universities
emphasize mathematics and science or subsidize faculty research are
influences outside of NSF's control.  These influences could affect
NSF's realization of its goal aimed at improving the achievement of
mathematics and science skills needed by all Americans, or its goal
of making discoveries at and across the frontier of science and
engineering.  Therefore, the identification of these factors and
related actions that could reduce or ameliorate their potential
impact could be useful in the Congress's review of and NSF's
implementation of the strategic plan. 


      NSF'S PLAN LACKS A COMPLETE
      DESCRIPTION OF HOW PROGRAM
      EVALUATIONS WILL BE USED AND
      A SCHEDULE FOR FUTURE
      PROGRAM EVALUATIONS
---------------------------------------------------------- Letter :4.6

NSF's draft plan does not discuss how the agency used specific
program evaluations to develop its strategic goals or the other
components of the plan.  However, beginning in fiscal year 1998, the
plan indicates that the Foundation will structure its internal and
external assessment processes using this strategic plan and the
performance plan and that performance reports will be requested
annually from all NSF units.  Further details are needed on a
schedule for future evaluations, the scope of and methodology for
future evaluations, and how the findings could be useful in assessing
NSF's goals and performance plans. 


   STATUTORY RESPONSIBILITIES ARE
   GENERALLY REFLECTED IN NSF'S
   STRATEGIC PLAN
------------------------------------------------------------ Letter :5

NSF's draft plan highlights the agency's core responsibilities under
the National Science Foundation Act of 1950 (Public Law 81-507), as
amended.  However, the draft plan does not mention additional
authorities given to NSF under the Science and Engineering Equal
Opportunities Act, Title I of the Education for Economic Security
Act, or other related legislation, such as that concerning polar
research and conservation.  Although these mandates are not
specifically mentioned, the core responsibilities appear general
enough to encompass the additional responsibilities.  Nonetheless, it
might be helpful if NSF provided a list, as a supplement to its plan,
of its statutory authorities and the major responsibilities that flow
from such legislation. 


   CROSSCUTTING FUNCTIONS ARE NOT
   ADEQUATELY RECOGNIZED IN THE
   STRATEGIC PLAN
------------------------------------------------------------ Letter :6

NSF's draft plan does not identify specific programs and activities
that are crosscutting or similar to those of other federal agencies. 
The draft plan does briefly describe the role NSF plays in the
federal science and engineering enterprise.  Although NSF captures
investment opportunities across the spectrum of science, mathematics,
and engineering to influence the nation's capabilities in all aspects
of these endeavors, the plan provides little evidence to suggest that
interagency coordination occurred to address the potential issues of
duplication and overlap.  Because overlapping and fragmented programs
can waste scarce funds, confuse and frustrate program customers, and
limit the overall effectiveness of the federal effort, it is
important for NSF to address crosscutting programs in its plan. 

In the science and technology area, where the federal government
spent $60 billion in fiscal year 1996 and the potential for
unnecessary overlap is particularly high, close coordination is
essential.  The Foundation's mission includes promoting the progress
of science, and one of its overarching goals is to enable the United
States to uphold a position of world leadership in all aspects of
science, mathematics, and engineering.  However, our review of other
agencies' draft strategic plans identified the following examples of
agencies with missions that could potentially overlap NSF's mission: 

  -- The Department of Energy's (DOE) science mission is to maintain
     leadership in basic research and to advance scientific
     knowledge. 

  -- The Department of Commerce's mission includes keeping America
     competitive with cutting-edge science and technology. 

There are additional examples of potential overlap among federal
agencies.  NSF's authorizing legislation directs it to initiate and
support science and engineering education programs at all levels and
in all fields of science and engineering.  Similarly, DOE's draft
strategic plan states that it will use its laboratories and the
nation's universities to contribute to the nation's science and
mathematics education. 

According to NSF's Assistant to the Director for Science Policy and
Planning, the Foundation participates in a number of groups such as
the National Science and Technology Council, the Committee on
Fundamental Science, and the Research Roundtable.  While one purpose
of the groups is to coordinate, NSF's draft plan neither mentions
these groups nor the frequency of their discussions.  However,
relating NSF's strategic plan to other agencies' crosscutting
programs would enhance the Congress's ability to assess any concerns
about potential overlap and duplication. 


   NSF'S PLAN DOES NOT FULLY
   ADDRESS ITS CAPACITY TO PROVIDE
   RELIABLE INFORMATION ON THE
   ACHIEVEMENT OF STRATEGIC GOALS
------------------------------------------------------------ Letter :7

While NSF's draft plan recognizes the importance of information
technology, the draft plan could benefit from a clearer and more
detailed discussion of how it specifically intends to use information
technology to improve performance, to reduce costs, and ultimately to
achieve its missions, goals, and objectives.  Furthermore, NSF's
draft plan does not address the "year 2000 problem" (which requires
that computer systems be changed to accommodate dates beyond the year
1999) or significant weaknesses in information security--two issues
that we have identified as high risk across government. 

Recent information technology reform legislation, including the
Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996,
set forth requirements that promote more efficient and effective use
of information technology in support of agencies' missions and of
improved program performance.  Under these acts, agencies are to
better link their technology plans and information technology use to
their programs' missions and goals. 

According to officials in NSF's OIG, one of NSF's primary financial
management challenges is to prepare and audit consolidated,
agencywide financial statements, as required by the Chief Financial
Officers (CFO) Act and the Government Management Reform Act of 1994. 
Financial statements are required to be prepared and audited to
instill greater accountability and to provide reliable financial
information for formulating budgets, managing government and program
operations, and making difficult policy decisions.  NSF received a
qualified opinion on its fiscal year 1996 statement of financial
position because of inadequate documentation for property, plant, and
equipment in the possession of its contractors and grantees.  (No
audit was performed on the 1996 Statement of Operations and Changes
in Net Position.  Therefore, the reliability of NSF's revenue and
expense information is uncertain.  It is our understanding that this
information will be audited for fiscal year 1997.) Reliable
information on investments in the property assets used to carry out
agency's mission is an essential part of performance measurement. 

In addition, to support the Results Act's implementation and to
ensure that NSF is complying with federal cost-accounting system
standards, the Foundation will need to relate costs to financial and
program performance data.  Key requirements of the CFO Act are the
development of cost information to enable the systematic measurement
of performance and the integrations of systems (i.e., program,
accounting, and budget systems).  NSF plans to modify its cost-
accounting system to support reporting on the Results Act once its
strategic plan is approved. 


   AGENCY COMMENTS AND OUR
   EVALUATION
------------------------------------------------------------ Letter :8

We provided a draft of this report to the National Science Foundation
for its review and comment.  (NSF's comments appear in enc.  I.) In
general, NSF agreed that additional information on two of the
required elements would be useful and provided additional context to
explain NSF's approach to meeting the remaining requirements of the
Results Act.  NSF stated its intention to update its draft plan by
providing additional detail on the external factors that could
influence its ability to meet its goals; current and future program
evaluation efforts and their contributions to the formulation of
investment strategies; and how NSF addresses issues of potential
duplication and overlap and its appropriate role in interagency
activities.  In regard to information technology, we believe that the
implementation of the tenets of the Clinger-Cohen and Paperwork
Reduction Acts is essential to effectively using information
technology to improve performance and carry out an agency's mission,
goals, and objectives.  Successfully addressing "year 2000" problems
is also critical.  It is therefore important for NSF, in its
strategic plan, to articulate how it plans to use information
technology in meeting these key challenges. 

While we questioned the measurability of NSF's outcome goals, NSF
believes that by using expert judgment (peer review) to assess a
range of factors about a project's results, it can appraise whether
the observed outcomes meet the stated goals.  We agree that
quantitative and qualitative indicators are widely used as proxies to
assess research and development (R&D) results because of the
difficulties in identifying the impacts of research.  Yet, while
implying a degree of precision, these indicators were not originally
intended to measure long-term R&D results.  Furthermore, while peer
review provides detailed information, it relies on the subjective
decisions of individuals and can be expensive.\3


--------------------
\3 Measuring Performance:  Challenges in Evaluating Research and
Development (GAO/T-RCED-97-130, Apr.  10, 1997). 


---------------------------------------------------------- Letter :8.1

As arranged with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 30 days after the date of this letter.  At that time, we will
send copies of this report to the Minority Leader of the House of
Representatives; the Ranking Minority Members of your Committees and
the Chairmen and Ranking Minority Members of the other Committees
that have jurisdiction over NSF; the Director of NSF; and the
Director, Office of Management and Budget.  We will send copies to
others on request. 

Please call me at (202) 512-3841 if you or your staff have any
questions concerning this letter. 

Victor S.  Rezendes
Director, Energy, Resources,
 and Science Issues

Enclosure




(See figure in printed edition.)Enclosure I
COMMENTS FROM THE NATIONAL SCIENCE
FOUNDATION
============================================================== Letter 



(See figure in printed edition.)

pp.  5 & 6. 



(See figure in printed edition.)


*** End of document. ***