Managing for Results: Observations on NASA's Fiscal Year 1999 Performance
Plan (Letter Report, 06/05/98, GAO/NSIAD-98-181).

Pursuant to a congressional request, GAO reviewed the National
Aeronautics and Space Administration's (NASA) performance plan for
fiscal year (FY) 1999, focusing on: (1) NASA's goals and objectives,
including how the agency plans to measure its progress toward achieving
these goals and objectives; (2) the agency's strategies and resources
needed to achieve its goals; and (3) the availability and reliability of
data necessary to achieve progress.

GAO noted that: (1) NASA's FY 1999 performance plan could provide a
clearer picture of intended performance across the agency, does not
fully portray how NASA's strategies and resources will help it achieve
the plan's performance goals, and partially provides confidence that the
information NASA will use to assess performance will be accurate,
complete, and credible; (2) among its strengths, NASA's performance plan
reflects the mission statement and goals in its strategic plan and
provides good linkage between these strategic goals and the plan's
performance goals and targets; (3) it incorporates performance measures
that are generally objective, quantifiable, and useful for assessing
progress toward the plan's performance objectives; and provides for
annual external assessments by its Advisory Council and semi-annual
internal assessments by its Senior Management Council to validate
progress toward meeting the agency's goals and objectives; (4) to make
the plan more useful for purposes of the Government Performance and
Results Act, NASA's performance plan should: (a) better link performance
goals and measures to the program activities in NASA's budget; (b) more
fully explain NASA's procedures for verifying and validating performance
data by recognizing the limitations that affect the credibility of data
that will be used to measure performance; and (c) acknowledge NASA's
major management challenges and associated corrective actions in order
to provide a more comprehensive understanding of the importance of the
goals and performance measures chosen for its internal crosscutting
processes; (5) some of the concerns GAO has regarding NASA's performance
plan are similar to its observations on NASA's strategic plan issued on
September 30, 1997; and (6) for example, NASA's strategic plan did not
contain evidence that NASA had coordinated the plan with those agencies
whose programs and activities complement NASA's and did not discuss the
human, capital, and information resources needed to achieve the goals
and objectives in the plan.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-98-181
     TITLE:  Managing for Results: Observations on NASA's Fiscal Year 
             1999 Performance Plan
      DATE:  06/05/98
   SUBJECT:  Strategic planning
             Information resources management
             Program evaluation
             Space exploration
             Congressional/executive relations
             Agency missions
             Data integrity
             Interagency relations
IDENTIFIER:  NASA Earth Observing System Data Information System
             GPRA
             Government Performance and Results Act
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Chairman and Ranking Minority Member, Committee on
Science, House of Representatives

June 1998

MANAGING FOR RESULTS -
OBSERVATIONS ON NASA'S FISCAL YEAR
1999 PERFORMANCE PLAN

GAO/NSIAD-98-181

NASA's Fiscal Year 1999 Performance Plan

(707313)


Abbreviations
=============================================================== ABBREV

  EOSDIS - Earth Observing System Data and Information System
  FAA - Federal Aviation Administration
  NASA - National Aeronautics and Space Administration
  OMB - Office of Management and Budget

Letter
=============================================================== LETTER


B-279979

June 5, 1998

The Honorable F.  James Sensenbrenner, Jr.
Chairman
The Honorable George E.  Brown
Ranking Minority Member
Committee on Science
House of Representatives

As requested, this report conveys the results of our review of the
National Aeronautics and Space Administration's (NASA) performance
plan for fiscal year 1999.  Under the Government Performance and
Results Act of 1993, also known as "the Results Act," NASA was
required to submit to Congress its first performance plan, for fiscal
year 1999, along with the President's fiscal year 1999 budget
request.  According to NASA officials, the plan was provided to
Congress on March 27, 1998.  NASA views its performance plan as the
third component of its "trilogy of Results Act-supporting" documents,
the other two being the annual budget justification and the strategic
plan. 

Our report discusses (1) NASA's goals and objectives, including how
the agency plans to measure its progress toward achieving these goals
and objectives; (2) the agency's strategies and resources needed to
achieve its goals; and (3) the availability and reliability of data
necessary to achieve progress. 


   BACKGROUND
------------------------------------------------------------ Letter :1

The Results Act requires that the Office of Management and Budget
(OMB) direct each executive agency to prepare an annual performance
plan, beginning with fiscal year 1999.  The performance plan is one
of three components of the Results Act, the others being the
strategic plan, submitted by agencies in September 1997, and the
annual report due March 31, 2000.  The performance plan should
describe (1) annual performance goals and measures, (2) the
strategies and resources to achieve these goals, and (3) procedures
to verify and validate reported performance.  The act requires that
performance goals be linked to the program activities in agencies'
budgets and be expressed in an objective, quantifiable, and
measurable form.  OMB is to use agency performance plans to develop
the overall federal government performance plan submitted annually to
Congress, beginning with the President's fiscal year 1999 budget. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :2

We found that NASA's fiscal year 1999 performance plan could provide
a clearer picture of intended performance across the agency, does not
fully portray how NASA's strategies and resources will help it
achieve the plan's performance goals, and partially provides
confidence that the information NASA will use to assess performance
will be accurate, complete, and credible. 

Among its strengths, NASA's performance plan reflects the mission
statement and goals in its strategic plan and provides good linkage
between these strategic goals and the plan's performance goals and
targets.  It incorporates performance measures that are generally
objective, quantifiable, and useful for assessing progress toward the
plan's performance objectives; and provides for annual external
assessments by its Advisory Council and semi-annual internal
assessments by its Senior Management Council to validate progress
toward meeting the agency's goals and objectives. 

To make the plan more useful for purposes of the Results Act, (1)
NASA's performance plan should better link performance goals and
measures to the program activities in NASA's budget, (2) more fully
explain NASA's procedures for verifying and validating performance
data by recognizing the limitations that affect the credibility of
data that will be used to measure performance, and (3) acknowledge
NASA's major management challenges and associated corrective actions
in order to provide a more comprehensive understanding of the
importance of the goals and performance measures chosen for its
internal crosscutting processes. 

Some of the concerns we have regarding NASA's performance plan are
similar to our observations on NASA's strategic plan issued on
September 30, 1997.  For example, NASA's strategic plan did not
contain evidence that NASA had coordinated the plan with those
agencies whose programs and activities complement NASA's and did not
discuss the human, capital, and information resources needed to
achieve the goals and objectives in the plan. 


   NASA'S PERFORMANCE PLAN COULD
   PROVIDE A CLEARER PICTURE OF
   INTENDED PERFORMANCE ACROSS THE
   AGENCY
------------------------------------------------------------ Letter :3

Generally, the performance goals in NASA's plan are objective and the
performance measures are quantifiable and useful for assessing
progress.  While the plan reflects the mission statement and
strategic goals in NASA's strategic plan and provides clear links
between strategic goals and the annual performance goals and
measures, it does not clearly associate the plan's performance goals
and measures with program activities in the budget.  Although NASA's
plan recognizes that other agencies and partners have complementary
missions and activities and acknowledges that NASA must work closely
with them to achieve its goals, it does not discuss the extent to
which it has coordinated either its strategies for achieving goals or
its development of performance measures. 


      DEFINING EXPECTED
      PERFORMANCE
---------------------------------------------------------- Letter :3.1

The performance plan provides near-term goals for each of NASA's four
business enterprises and at least one performance target to address
each of the goals for fiscal year 1999.  The four enterprises, which
were characterized by NASA in its fiscal year 1999 strategic plan as
analogous to business units in the private sector, are Space Science,
Earth Science, Human Exploration and Development of Space, and
Aeronautics and Space Transportation Technology.  In addition, the
performance plan includes goals and measures for the internal
processes it characterizes as crosscutting. 

NASA officials told us that the enterprise performance targets are
indicators of progress toward the agency's near-term goals. 
Generally, the enterprise near-term goals are objective and free from
bias.  The plan provides both quantitative and qualitative measures
for aspects of some of the goals.  For example, in addition to
quantitative measures for exploring the role of gravity in physical,
chemical, and biological processes, the plan provides for using "the
data obtained by fluid physics experiments on suspensions of
colloidal particles on MSL-1 [Microgravity Science Laboratory] to
answer fundamental questions in condensed matter physics.  .  .  ."
NASA officials acknowledged that this would require the use of
qualitative measures to determine whether the fundamental questions
have been answered. 

OMB recommends that outcome goals be included in a performance plan
whenever possible, but recognizes that agencies will supplement
outcome goals with output goals.  Some of NASA's performance measures
are outcome oriented; others are output oriented.  NASA officials
told us that outputs are used when demonstrating intermediate
progress in support of future outcomes and often reflect program
commitments made in conjunction with science users.  For instance,
they said that the performance target of orbiting the asteroid Eros
closer than
50 kilometers--a performance measure associated with NASA's objective
of exploring the solar system--matches Near-Earth Rendezvous program
expectations and specifications.  A few performance goals are not
specific to fiscal year 1999; rather, they span multiple years.  For
example, the plan provides for reducing "average spacecraft cost for
Space Science and Earth Science missions to $190 million from $590
million." However, the performance target is based on comparing
spacecraft programs completed in the time period of fiscal years 1990
through 1994 with those completed between fiscal year 1995 and 1999. 
NASA officials told us that expressing multiple year programs in a
single-year context required by the Results Act was difficult and a
source of frustration for the agency. 

In order to provide a more comprehensive understanding of the
importance of the goals and performance measures chosen for the
agency's crosscutting processes, it would have been helpful for the
plan to acknowledge the major management challenges and associated
corrective actions.  For example, NASA officials told us that the
plan's target of achieving 70 percent or greater of resource
authority obligated relates to funding carryovers--an issue we have
identified as a major challenge\1 and which the NASA Administrator
has recognized in congressional testimony as a top priority.  We have
also identified other management challenges.  These include

  -- improving systems and information for monitoring contractor
     activities and complying with contract management requirements,

  -- implementing a new accounting system and the importance of this
     effort to providing Congress and agency management with better
     financial information and allowing the agency to move to full
     cost accounting, and

  -- resolving uneven progress toward better NASA and Department of
     Defense cooperation and developing a national perspective on
     aerospace test facilities. 

OMB guidance states that performance goals for corrective steps for
major management problems should be included for problems whose
resolution is mission-critical, such as problems that could
materially impede the achievement of program goals.  Recognizing
those management challenges appears to meet the intent of OMB's
guidance. 

Furthermore, although the performance plan includes performance
targets related to contracting, such as increasing the obligated
funds for Performance-Based Contracts to 80 percent, the performance
measure can be better understood by referencing the agency's efforts
to ensure that it has relevant and reliable methods for timely and
accurate monitoring of its contract management activities.  For
example, the performance plan is silent on NASA's planned corrective
efforts in fiscal year 1999 to evaluate the effectiveness of its
contract management activities at its field centers.\2


--------------------
\1 NASA:  Major Management Problems (GAO/T-NSIAD-97-178, July 24,
1997). 

\2 High-Risk Program:  Information on Selected High-Risk Areas
(GAO/HR-97-30, May 16, 1997). 


      CONNECTING GOALS, MISSIONS,
      AND ACTIVITIES
---------------------------------------------------------- Letter :3.2

NASA's performance goals are closely aligned with the agencywide and
enterprise strategic mission, goals, and objectives, and provide good
linkages between the agency's near-term strategic goals and the
enterprise near-term goals, objectives, and performance targets. 
This is accomplished through well-laid out charts summarizing these
relationships. 

OMB Circular A-11 provides that the annual performance plan should
also show how specific performance goals are related to the specific
program activities contained in the agency's program and financing
schedules in the President's budget.  NASA's performance plan
contains a table that allocates NASA's $13.5-billion fiscal year 1999
budget request among its four enterprises and several other
consolidated budget accounts.  Each of the enterprises is related to
a set of performance goals.  However, the plan does not clearly
explain how NASA's program activities relate to the budget totals
presented for each enterprise.  There is no crosswalk to facilitate
an understanding of the relationship between the specific program
activities in the budget and the goals and measures in the
performance plan.  Therefore, we believe NASA's performance plan
cannot help Congress understand which performance goals and measures
in the plan cover which program activities or whether all program
activities in NASA's budget are covered by its performance goals. 

NASA officials with whom we spoke during our review indicated that
these relationships can be determined by simultaneously analyzing
information in its "trilogy" of documents, namely the annual budget
justification, strategic plan, and performance plan.  However, they
recognize that a clearer connection between performance goals and the
program activities in the budget needs to be provided in future
performance plans to facilitate analysis.  According to those
officials, an initiative is underway to align these documents. 


      RECOGNIZING CROSSCUTTING
      EFFORTS
---------------------------------------------------------- Letter :3.3

The performance plan points out that collaborative efforts with other
agencies and international partners will continue to be key to the
successful implementation of enterprise strategies.  Although the
plan identifies the specific agency or international partner involved
in carrying out specific efforts, it does not discuss the extent to
which other agencies have been coordinated with in establishing the
goals, objectives, and associated performance targets.  For example,
in describing the objective of developing next-generation
computational design tools, the plan indicates that NASA's efforts
are part of the Federal High Performance Computing and Communications
initiative.  However, there is no discussion as to whether NASA
coordinated its performance target of a 200-fold improvement with
other federal partners, nor is there an explanation of how NASA's
effort will contribute to the overall federal initiative. 

NASA's performance plan would be strengthened if it identified
activities being undertaken to address crosscutting issues of broad
national concern and discussed how achieving the agency's goals will
contribute to addressing the crosscutting issues.  To indicate how
NASA's programs contribute to addressing a crosscutting issue, the
plan could present output goals, output measures, intermediate
outcome goals, and intermediate outcome measures that reflect the
agency's contributions to addressing the crosscutting issues. 

The performance plan could be improved if it discussed efforts to
coordinate NASA's programs with other federal programs performing
related activities and provided evidence of coordination such as
joint planning and coordination or development of partnerships aimed
at improving program efficiency and effectiveness. 


   NASA'S PERFORMANCE PLAN DOES
   NOT FULLY DISCUSS HOW THE
   AGENCY'S STRATEGIES AND
   RESOURCES WILL HELP ACHIEVE ITS
   GOALS
------------------------------------------------------------ Letter :4

The performance plan does not fully discuss how the agency's
strategies and resources will help achieve its goals.  The
implementation strategy descriptions in each enterprise section are
unclear and do not provide sufficient information on how NASA will go
about accomplishing its near-term enterprise goals.  The plan only
partially discusses resources; it does not describe the operational
processes, skills and technology, human, capital, and information
resources required to achieve the performance goals. 


      CONNECTING STRATEGIES TO
      RESULTS
---------------------------------------------------------- Letter :4.1

The NASA plan does not fully describe how the agency will achieve its
performance goals, nor does it reference other documents that show
how the strategies will contribute to achieving the agency's
performance goals.  Enterprise implementation strategy descriptions
lack detailed information on how NASA will accomplish its enterprise
goals.  In several instances, these descriptions appear to outline
philosophies rather than describe enabling actions.  For instance,
application of the "faster, better, cheaper" approach to spacecraft
development identified in both the Space Science and Earth Science
implementation strategy sections is not clearly explained.  At a
minimum, it would have been helpful if the plan had included an
explicit discussion on what is involved in applying that approach and
how the establishment of "prudent risks" is part of it. 

While not required by the Results Act, we believe that a similar
discussion of external factors and discussion of how NASA will
mitigate or use the identified factors to achieve performance goals
could provide additional context regarding anticipated performance. 
With the exception of stating that NASA assumes that International
Space Station partners will meet their own schedules, the plan does
not explain how NASA will address external factors that could affect
the agency's performance during the coming fiscal year.  In its
strategic plan, NASA had identified several key external factors,
such as the stability of future budgets.  We believe that a similar
discussion in the performance plan would be helpful.  This is
particularly pertinent for NASA's budgetary programmatic priorities,
including the International Space Station, which could potentially
consume a large portion of NASA's future resources and affect the
timely implementation of other programs. 


      CONNECTING RESOURCES TO
      STRATEGIES
---------------------------------------------------------- Letter :4.2

NASA's performance plan partially discusses the resources it will use
to achieve the performance goals.  Characterization of resources
needed by NASA to meet its performance goals is limited to
identifying fiscal year 1999 funding requirements by enterprise.  In
addition, the plan does not discuss operational processes, skills and
technology, human and capital resources required to meet the
performance goals. 

Information technology issues are discussed in the context of the
internal crosscutting objective entitled "improving technology
capability and services." The performance target is identified as
improving information technology return on investment and customer
satisfaction by maintaining a positive return on investment and a
"satisfactory" rating from information technology customers.  The
performance measure of "maintaining a positive return on investment"
is vague and does not indicate how the target of improving
information technology return on investment will be achieved. 
Likewise, although maintaining a satisfactory rating from customers
is a positive measure, it does not indicate whether any improvement
is being achieved. 

We observe that the information technology goal and measure
identified in the performance plan do not specifically address any of
the important information technology management issues that NASA
currently faces.  Specifically, the plan does not discuss how

  -- NASA plans to measure compliance with the Clinger-Cohen Act of
     1996, which calls for agencies to implement a framework of
     modern technology management based on practices followed by
     leading private-sector and public-sector organizations that have
     successfully used technology to dramatically improve performance
     and meet strategic goals;

  -- the agency's new Chief Information Officer management structure,
     which depends on the cooperation of NASA's diverse enterprises
     and field centers, will measure its success in providing
     strategic direction to information technology investments; and

  -- NASA plans to address the "year 2000 problem" (which requires
     that computer systems be changed to accommodate dates beyond the
     year 1999) as well as any significant information security
     weaknesses--two issues that we have identified as high risk
     across the federal government.\3 Both the "year 2000 problem"
     and information security weaknesses could seriously undermine
     the performance and integrity of NASA's information systems if
     not adequately addressed. 

In addition, NASA's performance plan contains no discussion of its
strategy for achieving improvements in information technology
capability and services, nor does it refer to a separate information
technology annual plan that might contain such information.  As a
result, there is no way to assess whether NASA is reasonably prepared
to undertake any significant improvement in information technology
capability and services or whether such a goal is attainable. 
Furthermore, the performance plan gives no indication of the
resources associated with the information technology improvement
goal. 

Furthermore, the performance of NASA's major capital investments in
information technology are not specifically addressed in its
performance plan.  For example, the Earth Science Enterprise section
discusses an objective of improving dissemination of Earth Science
research results but does not mention the Earth Observing System Data
and Information System (EOSDIS).  NASA has requested $257 million for
EOSDIS in fiscal
year 1999.  No planned measures of the performance of EOSDIS are
included. 


--------------------
\3 The year 2000 problem refers to the potential for computer
programs to generate incorrect results when using dates from the year
2000 and beyond. 


   NASA'S PERFORMANCE PLAN
   PARTIALLY PROVIDES CONFIDENCE
   THAT ITS PERFORMANCE
   INFORMATION WILL BE CREDIBLE
------------------------------------------------------------ Letter :5

While NASA's performance plan identifies internal and external
organizations that will evaluate performance, it does not include a
discussion of the procedures that will be used to verify and validate
performance data and the limitations that could exist with internal
sources of data that NASA plans to use to assess performance. 
Although the Results Act does not require that limitations of
external sources of data be discussed in performance plans, we
believe NASA's performance plan would be improved with the inclusion
of such a discussion. 


      VERIFYING AND VALIDATING
      PERFORMANCE
---------------------------------------------------------- Letter :5.1

NASA's performance plan states that performance will be evaluated by
internal and external processes.  Internal assessments will be
conducted by agency management councils throughout the year and
semi-annually by the Senior Management Council.  External assessments
will be conducted annually by the NASA Advisory Council and
organizations such as the National Academy of Sciences.  For example,
the plan indicates that the Aeronautics and Space Transportation
Technology Committee of the NASA Advisory Council will conduct annual
assessments of the progress made by the Aeronautics and Space
Transportation Technology Enterprise.  This committee will provide a
qualitative progress measurement, including commentary to clarify and
supplement the qualitative measures.  Such commentary would
constitute a means to verify and validate measured values. 


      RECOGNIZING DATA LIMITATIONS
---------------------------------------------------------- Letter :5.2

The plan falls short in this area.  NASA's plan does not address
limitations to the data from internal or external sources.  To gauge
its progress, NASA enterprises will likely rely extensively on
internal sources of data, such as program management reports. 
However, the plan does not discuss any limitations that these data
sources have.  For example: 

  -- As previously discussed, the performance target of "improving
     information technology" will require assessments of the trend in
     the ratio of benefits achieved to information technology costs. 
     The plan would be more complete if it identified the challenges
     associated with determining benefits attributable to information
     technology investments. 

  -- In describing its internal crosscutting process characterized as
     "Communicate Knowledge," one of the objectives described is the
     development of educational outreach programs.  One of the
     associated performance targets in fiscal year 1999 is to
     increase the number of students reached through its
     NEWEST/NEWMAST programs to 42,000 students from 37,000 students. 
     It is not clear whether NASA will do its own survey or need to
     rely on other external information sources. 

NASA's performance plan could also benefit from a discussion of the
accuracy and reliability of data from external sources, even though
such a discussion is not required by the Results Act.  As previously
discussed, the plan recognizes that securing collaboration and
cooperation with other agencies and research partners is key to the
successful accomplishment of its goals.  In its description of the
objective of making major scientific contributions to national and
international environmental assessments, the plan can be improved by
indicating whether NASA will rely on information provided by its
partner, the Federal Aviation Administration (FAA), to assess
atmospheric effects of aviation, and if so, how the accuracy and
reliability of FAA's data will be assessed.  Since the performance
target associated with this objective is to make significant
contributions--further defined by NASA officials as being referenced
in technical literature--such added information could influence
greater use of the information generated. 

As previously noted, a critical component to NASA's evaluation of its
performance is the establishment of a financial management system and
its integration with full cost accounting.  Full cost accounting
facilitates decision-making and increases accountability by
automating the agency's operations and linking data results and costs
to major agency activities, budgets, accounts, and reports. 
Unfortunately, all the information NASA needs to measure the costs
associated with its performance is currently not available.  NASA's
plans to implement full cost accounting are contingent upon the
timely completion of its new financial management system, which NASA
plans to implement in 1999.  Until the new integrated financial
management system is operational, performance assessments relying on
cost data may be incomplete. 


   AGENCY COMMENTS AND OUR
   EVALUATION
------------------------------------------------------------ Letter :6

We provided a draft of this report to NASA for review and comment. 
In its comments, NASA stated that it will continually improve the
content of its annual plan and recognizes the usefulness of our
comments and recommendations.  NASA intends to use its strategic
plan, budget justification, and performance plan as a "trilogy of
documents" to be reviewed as a package.  Even though the areas we
identified in our report as lacking specificity may be addressed in
NASA's strategic plan, budget justification, and other documents, we
maintain that the performance plan could be more useful by better
linking performance goals and measures to program activities in the
budget; more fully explaining procedures for verifying and validating
performance data and recognizing the limitations that affect the
credibility of data that will be used to measure performance; and by
acknowledging major management challenges and associated corrective
actions.  Including these areas in the plan would provide a more
comprehensive understanding of the importance of NASA's goals and
performance measures, making it more useful for purposes of the
Results Act. 

NASA's comments and our evaluation of them are presented in appendix
I. 


   SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :7

The scope of our work was limited to the information contained in
NASA's fiscal year 1999 performance plan and clarifying discussions
we had with NASA officials.  The criteria we used to conduct this
review were the Results Act; OMB's guidance on developing the plans
(circular A-11, part 2); our February 1998 guidance for congressional
review of the plans
(GGD/AIMD 10.1.18); and the December 17, 1997, letter to OMB Director
Raines from several congressional leaders.  For purposes of our
analysis, we collapsed the six requirements for annual performance
plans in the Results Act and the related guidance into three core
questions: 

(1) To what extent does the agency's performance plan provide a clear
picture of intended performance across the agency? 

(2) How well does the performance plan discuss the strategies and
resources the agency will use to achieve its performance plan? 

(3) To what extent does the agency's performance plan provide
confidence that its performance information will be credible? 

We conducted our work between December 1997 and April 1998 in
accordance with generally accepted government auditing standards. 


---------------------------------------------------------- Letter :7.1

As requested, we plan no further distribution of this report until 7
days after its issue date.  At that time, we will send copies to the
Director of the Office of Management and Budget, the Administrator of
NASA, and appropriate congressional committees.  We will also make
copies available to other interested parties on request. 


If you or your staff have any questions concerning this report,
please contact me on (202) 512-4841.  The major contributors to this
report are listed in appendix II. 

Allen Li
Associate Director
Defense Acquisitions Issues




(See figure in printed edition.)Appendix I
COMMENTS FROM THE NATIONAL
AERONAUTICS AND SPACE
ADMINISTRATION
============================================================== Letter 



(See figure in printed edition.)



(See figure in printed edition.)


The following are our comments on the National Aeronautics and Space
Administration's (NASA) letter, dated April 24, 1998. 

GAO COMMENTS

1.  Responding to the issues raised in this report, NASA concludes
that it has complied with the requirements of the Government
Performance and Results Act.  We agree.  Our point in the report was
that the plan contains areas that could be strengthened, making it
more useful to Congress.  In discussing the strengths of NASA's
performance plan, our report recognizes that the plan provided good
linkage between strategic goals and the goals in the performance plan
and that performance measures were generally objective, quantifiable,
and useful for assessing progress.  However, our report also clearly
states that NASA's plan could provide a clearer picture of intended
performance across the agency and does not fully portray how NASA's
strategies and resources will help it achieve the plan's performance
goals.  The report also states that NASA's plan does not clearly
explain how program activities relate to the budget and that there is
no crosswalk to facilitate an understanding of the relationship
between the specific program activities in the budget and the goals
and measures in the performance plan.  As stated in our report, NASA
is in agreement to merge the budget justification and performance
plan process in an effort to provide a crosswalk showing the
relationship of performance targets to program activities and
resources. 

2.  In commenting on measures for major management challenges, NASA
recognizes that the plan does not include measures to improve the
management challenges we address.  While these management issues may
be addressed in other management documents, such as the Chief
Financial Officer's Five-Year Plan, Chief Information Officer's
Information Technology Implementation Plan, and implementation
planning documents being developed by the Office of Procurement, to
which NASA alludes, we did not review these documents, nor does the
performance plan include reference to them.  Referencing these
documents in the plan would have been useful in providing specific
sources for obtaining a more comprehensive understanding of the
importance of these issues.  We continue to believe that an
acknowledgement of major management challenges and associated
corrective actions would make the plan more useful for purposes of
the Results Act. 

3.  In addressing the issue of resource requirements to achieve
performance, NASA correctly restates our concern that the plan only
addresses funding requirements for fiscal year 1999 at the enterprise
level.  As in their comment to our point on not identifying
management challenges, NASA makes the assumption that, since a
discussion of operational processes, skills and technology, human and
capital resources required to meet the performance goals is provided
in the budget justification, it need not be included in the
performance plan.  Referencing these documents in the plan would have
been useful in providing specific sources for obtaining a more
comprehensive understanding of the importance of these issues. 

4.  Responding to our concern that the plan's implementation strategy
of applying the "faster, better, cheaper" approach to spacecraft
development is not clearly explained, NASA contends that the
statement that follows in the plan is a clear enough description of
its approach.  We do not agree.  The statement NASA alludes to says
that

     "Program managers are encouraged to accept prudent risk, shorten
     development time of technologies and missions, explore new
     conceptual approaches, streamline management, and incorporate
     innovative methods and technologies to enhance efficiency and
     effectiveness."

In our opinion, this statement does not provide a clear explanation
of NASA's implementation strategy.  We continue to believe that a
more explicit discussion of this approach is needed. 

5.  Our issue regarding Earth Observing System Data and Information
System (EOSDIS) is not in its lack of identification in the section
discussing crosscutting information technology.  Rather, we take
issue with EOSDIS not being specifically mentioned either in the
references cited in NASA's comments or anywhere else in the
performance plan, despite the fact that NASA plans to spend well over
$1 billion on the system.  NASA's comment assumes that the reader of
the plan would recognize EOSDIS as the vehicle for improving
dissemination of Earth Science research results.  We believe that a
large investment such as EOSDIS should be explicitly addressed in the
performance plan in order to ensure that its actual performance is
understood.  For example, a test of one part of EOSDIS in March 1998
revealed software problems that will force the launch of the EOS AM-1
satellite to be delayed by at least 6 months.  We believe that the
cost and schedule impact of this and other EOSDIS development
problems needs to be explicitly and objectively assessed. 

6.  Responding to our concern about the lack of any discussion in the
plan about the credibility of performance information, NASA officials
expressed the belief that the data it collects will be credible and
that we may be premature in passing judgment.  As support, NASA
references the audit of its fiscal year 1996 and fiscal year 1997
Accountability Reports by Arthur Andersen, LLP, which the agency says
found no significant issues with the credibility of NASA's
performance information plans.  We have three points regarding NASA's
comment.  First, the plan would have been strengthened by referencing
the results of the Arthur Andersen, LLP, audit in the performance
plan.  Second, while a review of the accountability reports was not
in the scope of our review, we believe that the objectives and
performance measures in the performance plan are likely to be more
results oriented than those included in the accountability reports
NASA references.  Third, while the plan does identify internal and
external organizations that will evaluate its performance, we believe
that it is not premature for NASA's performance plan to include a
discussion of procedures that will be used to verify and validate
performance data and the limitations that could exist with internal
sources of data that NASA plans to use to assess performance. 


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II

NATIONAL SECURITY AND
INTERNATIONAL AFFAIRS DIVISION,
WASHINGTON, D.C. 

Richard J.  Herley
Samuel N.  Cox
Shirley B.  Johnson

ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION

Joseph A.  deFerrari

ATLANTA FIELD OFFICE

Elizabeth M.  Mixon

OFFICE OF THE GENERAL COUNSEL

William T.  Woods

RELATED GAO PRODUCTS

NASA Budget:  Carryover Balances in Selected Programs
(GAO/NSIAD-96-206, July 16, 1996). 

High-Risk Program:  Information on Selected High-Risk Areas
(GAO/HR-97-30, May 16, 1997). 

Results Act:  Observations on NASA's May 1997 Draft Strategic Plan
(GAO/NSIAD-97-205R, July 22, 1997). 

NASA:  Major Management Problems (GAO/T-NSIAD-97-178, July 24, 1997). 

Managing for Results:  Agencies' Annual Performance Plans Can Help
Address Strategic Planning Challenges (GAO/GGD-98-44, Jan.  30,
1998). 

Aerospace Testing:  Promise of Closer NASA/DOD Cooperation Remains
Largely Unfulfilled (GAO/NSIAD-98-52, Mar.  11, 1998). 

*** End of document. ***