Federal Management: Observations on the National Science Foundation
(Testimony, 07/24/97, GAO/T-RCED/AIMD-97-168).
GAO discussed a number of management and program issues at the National
Science Foundation (NSF) as recently reported by the NSF Office of
Inspector General (OIG) and GAO, focusing on: (1) duplicate funding in
the Small Business Innovation Research (SBIR) program at NSF and
elsewhere; (2) challenges for NSF in implementing the Government
Performance and Results Act (GPRA); (3) the status of NSF's efforts to
prepare and have audited agencywide financial statements as required by
the Chief Financial Officers Act (CFO); and (4) a framework for
assessing NSF's readiness to meet computer challenges that will arise in
the year 2000.
GAO noted that: (1) among the management issues identified by NSF's OIG
most in need of reform at NSF is the SBIR program; (2) GAO has reported
that duplicate funding of similar or even identical SBIR proposals
submitted to more than one agency has occurred at NSF, the National
Aeronautics and Space Administration, and Department of Defense ; (3)
NSF has revised its certification form to require applicants to certify,
under criminal penalties for perjury, exactly what, if any, applications
for similar research were pending in other agencies; (4) in implementing
the results Act, NSF, like most agencies, is still struggling to develop
the mission based goals and the performance measurement requirements of
the Act; (5) measuring the performance of science-related projects can
be extremely difficult because a wide range of factors determine if and
how a particular research and development (R&D) project will result in a
commercial application or have other benefits, and it can take years for
a research project to realize a successful outcome; (6) GAO recently
reported that there is no single indicator or evaluation method that
captures the results of R&D; (7) as a result, determining the specific
outcomes resulting from federal R&D is a challenge that will not easily
be resolved; (8) as required by the CFO Act, one of NSF's primary
financial management challenges is to prepare and audit consolidated
agencywide financial statements; (9) NSF received its first-time audit
of its agencywide financial statements for fiscal year 1996; (10) NSF's
auditor concluded that the reported property, plant, and equipment
account balance is unreliable; (11) in response to its auditor's
concerns, NSF is in the process of developing a framework for
identifying and developing performance measures in response to the CFO
Act's and GPRA's requirements; (12) NSF like other federal agencies that
have system or application programs that use dates to perform
calculations, comparisons, or sorting may generate incorrect results
when working with years after 1999; (13) converting systems to a
four-digit year can be a massive undertaking for agencies such as NSF
because it involves identifying computer systems, developing conversion
strategies and plans, and dedicating sufficient resources to converting
and adequately testing their computer systems and programs before
January 1, 2000; (14) GAO has developed an assessment guide that
provides a structured approach for assessing the readiness of agencies'
year 2000 programs.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: T-RCED/AIMD-97-168
TITLE: Federal Management: Observations on the National Science
Foundation
DATE: 07/24/97
SUBJECT: Small business assistance
Research grants
Grant award procedures
Strategic planning
Research and development
Financial statement audits
Accounting procedures
Systems conversions
Research program management
Interagency relations
IDENTIFIER: NSF Small Business Innovation Research Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Before the Senate Committee on Commerce, Science,
and Transportation
U.S. Senate
For Release
on Delivery
Expected at
9:30 a.m., EDT
Thursday,
July 24, 1997
FEDERAL MANAGEMENT - OBSERVATIONS
ON THE NATIONAL SCIENCE FOUNDATION
Statement of Susan D. Kladiva,
Acting Associate Director,
Energy, Resources, and Science Issues,
Resources, Community and Economic
Development Division
GAO/T-RCED/AIMD-97-168
GAO/RCED/AIMD-97-168T
(141063)
Abbreviations
=============================================================== ABBREV
NSF -
OIG -
CFO -
DOD -
GPRA -
OMB -
FASAB -
SBIR -
NASA -
R&D -
SBA -
GMRA -
PP&E -
============================================================ Chapter 0
Mr. Chairman and Members of the Committee:
We are pleased to be here today to discuss a number of management and
program issues at the National Science Foundation (NSF) as recently
reported by the NSF Office of Inspector General (OIG) and GAO. Many
of the management issues have been acknowledged by the agency's
senior management and are receiving varying levels of attention. We
have not specifically reviewed NSF's management and program areas,
other than its implementation of the Government Performance and
Results Act (Results Act). Therefore, as agreed with the Committee,
my statement will focus on steps to address some of the systemic
problems that are based on similar concerns across the federal
government. Specifically, I will address (1) duplicate funding in
the Small Business Innovation Research (SBIR) program at NSF and
elsewhere; (2) challenges for NSF in implementing the Results Act;
(3) the status of NSF's efforts to prepare and have audited
agencywide financial statements as required by the Chief Financial
Officers (CFO) Act as amended; and (4) a framework for assessing
NSF's readiness to meet the computer challenges that will arise in
the year 2000.
In summary, Mr. Chairman:
-- Among the management issues identified by NSF's OIG most in need
of reform at NSF is the SBIR program. Regarding the SBIR
Program, we have reported that duplicate funding of similar or
even identical SBIR proposals submitted to more than one agency
has occurred at NSF, NASA, and DOD.\1 According to agency
officials, a few companies received funding for the same
proposals twice, three times, and even five times before
agencies became aware of the duplication. Several factors
contribute to this problem, including (1) the evasion of
certification procedures, so that companies fail to identify
similar proposals submitted to other agencies, (2) the lack of a
consensus on what constitutes a duplicate proposal, and (3) the
general lack of interagency access to and exchange of current
information about recent awards by other agencies. NSF has
revised its certification form to require applicants to certify,
under criminal penalties for perjury, exactly what, if any,
applications for similar research were pending in other
agencies. In response to our recommendations, the Administrator
of the Small Business Administration (SBA) has taken steps to
address all three issues.
-- In implementing the Results Act, NSF, like most agencies, is
still struggling to develop the mission based goals and the
performance measurement requirements of the Act. We recently
found that NSF's draft strategic plan is incomplete and not
specific enough to allow the Congress to evaluate whether the
agency's goals are achievable.\2 This is to be expected during
the initial efforts of such a challenging management reform
effort. Measuring the performance of science-related projects
can be extremely difficult because a wide range of factors
determine if and how a particular research and development (R&D)
project will result in a commercial application or have other
benefits, and it can take years for a research project to
realize a successful outcome. Moreover, we recently reported
that there is no single indicator or evaluation method that
adequately captures the results of R&D.\3
As a result, determining the specific outcomes resulting from federal
R&D is a challenge that will not easily be resolved. The Army
Research Laboratory, which was designated as a pilot project for
performance measurement under the Results Act, has developed a
multifaceted approach using quantitative indicators, peer review, and
customer feedback to evaluate the results of R&D. This response to
the challenges in measuring the impacts of research shows that some
progress is being made in response to the Results Act and may provide
useful guidance to other science agencies.
-- As required by the CFO Act as amended by the Government
Management Reform Act (GMRA), one of NSF's primary financial
management challenges is to prepare and audit consolidated
agencywide financial statements. Financial statements are
required to be prepared and audited to instill greater
accountability; to provide reliable financial information for
managing government and program operations and for making
difficult policy decisions. NSF received its first-time audit
of its agencywide financial statements for fiscal year (FY)
1996. NSF's auditor concluded that the reported property,
plant, and equipment account balance is unreliable. Because
about 99 percent of NSF's property is in the custody of R&D
contractors and grantees, NSF is considering corrective actions
that will include a change in accounting treatment as well as
adequate inventory controls. Also, in response to its auditor's
concerns, NSF is in the process of developing a framework for
identifying and developing performance measures in response to
the CFO Act's and GPRA's requirements.
-- The year 2000 problem is caused by systems that typically use
two digits to represent years thereby making the year 2000
indistinguishable from 1900, 2001 indistinguishable from 1901,
and so on. NSF like other federal agencies that have system or
application programs that use dates to perform calculations,
comparisons, or sorting may generate incorrect results when
working with years after 1999. Converting systems to a
four-digit year can be a massive undertaking for agencies such
as NSF because it involves identifying computer systems,
developing conversion strategies and plans, and dedicating
sufficient resources to converting and adequately testing their
computer systems and programs before January 1, 2000. GAO has
developed an assessment guide that provides a structured
approach for assessing the readiness of agency's year 2000
programs.\4
--------------------
\1 Federal Research: Interim Report on the Small Business Innovation
Research Program (GAO/RCED-95-59, Mar. 8, 1995).
\2 Results Act: Observations on the National Science Foundation's
Draft Strategic Plan (GAO/RCED-97-203R, July 11, 1997).
\3 Measuring Performance: Strengths and Limitations of Research
Indicators (GAO/RCED-97-91, Mar. 21, 1997).
\4 Year 2000 Computing Crisis: An Assessment Guide
(GAO/AIMD-10.1.14, Feb. 1997).
BACKGROUND
---------------------------------------------------------- Chapter 0:1
In recent years, the Congress has put in place a statutory framework
for improving management in the federal government, and for helping
the Congress and the executive branch make the difficult trade-offs
that the current budget environment demands. This framework includes
as its essential elements the Chief Financial Officers Act;
information technology reform legislation; and the Government
Performance and Results Act. We have begun a body of work on each of
these management areas which provides some broad guidance of use to
this Committee in its current review of NSF.
DUPLICATE FUNDING OF SBIR
PROPOSALS
---------------------------------------------------------- Chapter 0:2
Because of the mismanagement of funds, the OIG identified SBIR as one
program most in need of reform. Duplicate funding of similar or even
identical proposals submitted to more than one agency has occurred in
NSF, NASA, and DOD; however, steps have been taken to address the
issue. Several factors have contributed to the problem of duplicate
funding. First, the companies proposing projects have failed to
identify identical proposals they have made to other agencies,
thereby evading the certification procedure that requires them to
provide such information. In response to this problem, NSF's OIG
officials were concerned about the need for more complete
certification requirements and recommended that the NSF certification
form be revised and strengthened. NSF has implemented this
recommendation.
Second, the lack of definitions and guidelines for the key terms,
such as "similar" or "overlapping" research, has resulted in
disagreement about what constitutes duplicate research. SBA's prior
policy directives and the individual agencies' solicitations did not
define key terms and thus provided little or no guidance in avoiding
the risk of duplicate funding. In general, the absence of
substantive definitions for key terms placed the burden of judgment
on the company. In some cases, as we stated in our March 1995
report, the appropriate certification may be difficult to determine
and can lead to conflicts of opinion that may harm the SBIR program
as well as the individual company. In response to this problem,
officials in NSF's OIG defined "overlapping" research and noted that
the scope and funding of some SBIR awards in 1994 were reduced to
eliminate overlapping work.
Third, the agencies lacked interagency access to and exchange of
current information about recent awards that might help to prevent or
detect duplicate awards. At the time of our report, SBA maintained a
database for the SBIR program that it used primarily to produce its
annual report to the Congress regarding the program. The information
had a "time lag" of about 9 months because it was first processed by
each agency and then forwarded to SBA. Some SBIR officials, however,
believed that the existing methods might not be adequate for
detecting duplication when the agencies were receiving 20,000
proposals annually.
In response to these three issues, SBA generally agreed with our
concerns and modified its SBIR policy directive in July 1995 to
address these problems. SBA refined the certification statement on
possible duplication, clarified the definition of similar awards, and
began work on developing a computer system that would enable all
agencies to access timely information on awards being made by other
agencies. The system is now operational, but, according to an SBIR
program official at SBA, further work is under way to strengthen it.
IMPLEMENTATION OF THE RESULTS
ACT IN SCIENCE AGENCIES
---------------------------------------------------------- Chapter 0:3
Under the Results Act, federal agencies are to set strategic and
annual goals, measure performance, and report on the degree to which
goals are met. The Results Act requires that an agency's strategic
plan contain the following six critical elements: a mission
statement; agencywide goals and objectives; strategies and resources
needed to achieve the goals and objectives; the relationship between
the long-term goals and objectives and the annual performance goals;
key external factors that could affect the achievement of goals; and
a description of how program evaluations were and will be used to
establish or revise strategic goals.
While NSF addressed five of the six required elements in its draft
strategic plan, at least four of them need further development, and
one element--key external factors--is not included in the current
draft. NSF's outcome goals set out the long-term programmatic,
policy, and management goals to be accomplished through its program
office investments; but many of these goals are not expressed in a
measurable form. NSF's plan provides some general dates for
achieving its goals but does not provide the underlying assumptions,
projections, or a schedule for initiating or completing significant
actions. The strategies for achieving NSF's goals lack precision,
making it unclear whether the Foundation and the Congress will be
able to assess whether the goals are achieved. Also unclear is the
process for communicating goals and objectives throughout the agency
and for assigning accountability to managers and staff for achieving
the goals. Finally, NSF's draft strategic plan does not discuss how
the agency used specific program evaluations to develop its strategic
goals or the other components of the plan.
We recognize that NSF is currently revising its draft strategic and
performance plans based on comments received by various parties.
Nonetheless, as expected during the initial efforts of such a
challenging management reform effort, most science agencies like NSF,
are still struggling to develop mission based goals and performance
measures. Determining the specific outcomes resulting from federal
R&D is a challenge that will not easily be resolved. The experts in
research measurement have tried for years to develop indicators that
would provide a measure of the results of R&D. However, the very
nature of the innovative process makes measuring the performance of
science-related projects difficult. For example, a wide range of
factors determines if and how a particular R&D project will result in
a commercial application or have other benefits. It can also take
many years for a research project to achieve results.
Because of these difficulties, there is no single indicator or
evaluation method that adequately captures the results of R&D.
Decisionmakers have developed quantitative and qualitative indicators
as proxies to assess the results of R&D activity. Our March 1997
report discusses the strengths and limitations in both types of
indicators. The amount of money spent on R&D, the primary indicator
of research investment, is useful as a measure of how much research
is being performed. Having been refined over many years, these data
are generally available for both the public and private sectors.
However, the level of spending is not a reliable indicator of the
level of research results.
Quantitative output indicators focus mainly on return on investment,
patenting rates, and bibliometrics--the study of publication-based
data. While implying a degree of precision, these indicators were
not originally intended to measure long-term R&D results.
Qualitative assessment, such as peer review, provides detailed
information, but depends on criteria that are inherently difficult to
measure and on subjective judgment that is vulnerable to bias.
Science agencies, like other agencies, must guard against the
understandable tendency to overly rely on goals and measures that are
easily quantifiable, such as the numbers of research grants provided
and completed, at the expense of what is truly important but more
difficult to measure, such as the difference the program makes in
people's lives. However, the legislative history that accompanied
the Results Act states that agencies should not trivialize
measurement by seeking to measure performance in a forced or
artificial way simply to present quantifiable measures.
The Army Research Laboratory, which was designated as a pilot project
for performance measurement under the Act, developed a multifaceted
approach using quantitative indicators, peer review, and customer
feedback to evaluate the results of R&D. Although this is not the
only approach that can be taken, this response to the challenges in
measuring the impacts of research shows that some progress is being
made in response to the Results Act and may provide useful guidance
to other science agencies.
IMPLEMENTATION OF THE CFO ACT
---------------------------------------------------------- Chapter 0:4
One of NSF's primary financial management challenges is to prepare
and have its consolidated agencywide financial statements audited, as
required by the Chief Financial Officers Act of 1990, as amended by
the Government Management Reform Act (GMRA) of 1994. Financial
statements are required to be prepared and audited to instill greater
accountability and provide reliable financial information for
managing government and program operations and making difficult
policy decisions. For FY 1991 through FY 1995, NSF had only prepared
and had audited the financial statements for its Donations Account
(Trust Fund). To meet the challenge of preparing and auditing
consolidated agencywide financial statements, NSF's CFO contracted
with an independent public accounting firm to prepare proforma
consolidated financial statements for FY 1994 and FY 1995. The CFO
also contracted for assistance in preparing agencywide FY 1996
financial statements.
In preparation for the FY 1996 agencywide financial statement audits,
the OIG assessed NSF's financial and administrative controls,
including property controls, and its general ledger accounting
system, the effectiveness of electronic data processing controls, and
adherence to core financial system requirements. Because NSF's OIG
lacked adequate audit resources, the IG contracted with a separate
independent public accounting firm for the audit of NSF's FY 1996
Statement of Financial Position (balance sheet).
The IG and NSF's contract auditor issued a qualified opinion on NSF's
FY 1996 balance sheet due to inadequate supporting documentation and
lack of a system to confirm account balances for approximately 99
percent of NSF's property, plant and equipment (PP&E), which is held
by NSF's contractors and grantees. In its efforts to resolve these
accounting problems, NSF management is considering the appropriate
accounting treatment under the new federal accounting standards. NSF
management believes that NSF has only a reversionary interest\5 in
PP&E held by its contractors and grantees. Under the new federal
accounting standards, where this is the case, the assets are not to
be reported as PP&E in federal agency accounts unless the assets
actually revert to federal ownership. The new accounting standards
provide for such assets to be accounted for and reported as R&D
investment. Under the standard, NSF would account for the
acquisition cost of PP&E for its contractors and grantees as an
annual investment (expense). As a result, NSF would not be required
to report related PP&E asset values in its financial statements.
While PP&E held by contractors and grantees would still be subjected
to inventory controls and audit, NSF's OIG is concerned about the
adequacy of its contractors and grantees auditors' scope for federal
financial statement audit purposes.
NSF has requested formal approval of the accounting changes from OMB
and FASAB. Upon approval by OMB and FASAB, NSF plans to implement
the R&D investment standard for its FY 1997 financial statements, one
year ahead of the implementation date in the standard.
In addition, the IG and NSF's contract auditor reported that NSF has
not yet met the CFO Act requirement for systematic measurement of
performance. NSF is in the process of developing a framework for
identifying and developing performance measures in response to CFO
Act and GPRA requirements.
Further, to support GPRA implementation and to ensure that NSF is
complying with federal cost accounting standards, the CFO is
considering hiring an independent public accounting firm to evaluate
its current cost accounting system capability against the
requirements in the new federal cost accounting standards. NSF's
current financial system is capable of capturing direct costs by
organization and program activities and allocating indirect costs.
However, when NSF determines what performance measures will be used
to capture program and financial outputs and outcomes, it will need
to modify the system to relate costs to financial and program
performance data. Along these lines, NSF plans to have an
activity-based accounting system module operational by the end of FY
1998. Also, NSF plans to report performance measures related to R&D
investment, including program outputs and outcomes.
--------------------
\5 A reversionary interest arises when NSF reserves the right to take
back at the end of the grant or contract term, assets that it funded
for use by a contractor or grantee.
YEAR 2000 COMPUTER CHALLENGES
---------------------------------------------------------- Chapter 0:5
The year 2000 computing problem is rooted in the way dates are
recorded and computed in many computer systems. For the past several
decades, systems have typically used two digits to represent the year
such as "97" representing 1997, in order to conserve on electronic
data storage and reduce operating costs. With this two-digit format,
however, the year 2000 is indistinguishable from 1900, 2001 from 1901
and so on. As a result of this ambiguity, system or application
programs that use dates to perform calculations, comparisons, or
sorting may generate incorrect results when working with years after
1999.
Because converting systems to a four-digit year can be a massive
undertaking, agencies need to identify their inventories of
mission-critical computer systems, develop conversion strategies and
plans, and dedicate sufficient resources to converting and adequately
testing their computer systems and programs before January 1, 2000.
Although we have not evaluated NSF's efforts, the agency reported in
June that it had assessed which systems to change, replace or
discard.
To assist agencies in achieving year 2000 compliance, we developed an
assessment guide which provides a framework and a checklist for
assessing the readiness of federal agencies. The guide addresses
issues that will be common to most year 2000 needs. It provides
information on the scope of the challenge, and offers a structured
approach for reviewing the adequacy and agency planning and
management of the year 2000 program. This approach includes five
phases addressing awareness, assessment, renovation, validation and
implementation and are supported by program and project management
activities. However, because each agency is different, there is no
single, cookie cutter approach. Each agency must tailor its year
2000 program in response to its unique needs.
-------------------------------------------------------- Chapter 0:5.1
In summary, a statutory framework exists for improving management in
the federal government. Fully resolving the problems, however, will
require sustained management attention. Congressional oversight,
such as is being given in this instance by the Chairman and the
Committee, is also key.
This concludes my statement. I would be happy to respond to any
questions you or the Members of the Committee may have.
*** End of document. ***