NASA: Status of Plans for Achieving Key Outcomes and Addressing
Major Management Challenges (27-NOV-01, GAO-02-184).
GAO reviewed the National Aeronautics and Space Administration's
(NASA) fiscal year 2002 performance plan for the selected key
outcomes: to expand scientific knowledge of the Earth's system,
to expand the commercial development of space, and to deploy and
operate the International Space Station. NASA has improved its
fiscal year 2002 performance plan and responded to
recommendations by GAO and others to make its plan more
useful--particularly by providing more comprehensive explanations
of how it plans to verify and validate performance data and by
better portraying how its performance goals will benefit the
public. NASA's annual performance goals appear to be objective
and should help to measure progress toward the outcomes. However,
the plan still does not explain the reasons for changes in
performance goals. Not having these explanations could hinder the
ability to assess NASA's performance.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-184
ACCNO: A02484
TITLE: NASA: Status of Plans for Achieving Key Outcomes and
Addressing Major Management Challenges
DATE: 11/27/2001
SUBJECT: Agency missions
Cost effectiveness analysis
Performance measures
Safety
Space exploration
Strategic planning
Government Performance and Results Act
GPRA
NASA Integrated Financial Management
System
NASA International Space Station Alpha
Progam
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-184
Report to the Ranking Minority Member, Committee on Governmental Affairs, U.
S. Senate
United States General Accounting Office
GAO
November 2001 NASA Status of Plans for Achieving Key Outcomes and Addressing
Major Management Challenges
GAO- 02- 184
Page i GAO- 02- 184 Status of Plans for Achieving Key Outcomes Letter 1
Results in Brief 2 Background 5 Assessment of NASA?s Performance Goals and
Strategies for
Accomplishing Selected Key Outcomes 6 Comparison of NASA?s Fiscal Year 2002
Performance Plan With the
Prior Year Plan 12 NASA?s Efforts to Address its Major Management Challenges
Identified by GAO 15 Scope and Methodology 16 Agency Comments 16
Appendix I Observations on NASA?s Efforts to Address Its Major Management
Challenges 19
Appendix II Comments From the National Aeronautics and Space Administration
33
Tables
Table 1: Major Management Challenges 20 Contents
Page 1 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
November 27, 2001 The Honorable Fred Thompson Ranking Minority Member
Committee on Governmental Affairs United States Senate
Dear Senator Thompson: As you requested, we reviewed the National
Aeronautics and Space Administration?s (NASA) fiscal year 2002 performance
plan required by the Government Performance and Results Act of 1993 (GPRA) 1
to assess the agency?s planned performance for the selected key outcomes
that you identified as important mission areas for the agency. 2 These are
the same outcomes we addressed in our June 2000 review of NASA?s fiscal year
1999 performance report and fiscal year 2001 performance plan to provide a
baseline by which to measure the agency?s performance from year to year. 3
We also addressed these outcomes in our July 2001 review of NASA?s fiscal
year 2000 performance report. 4 As agreed with your office, we are issuing
this separate report on our assessment of the selected outcomes in NASA?s
fiscal year 2002 performance plan. The selected key outcomes are to
expand scientific knowledge of the Earth system, expand the commercial
development of space, and deploy and operate the International Space
Station (ISS) safely and cost
effectively. 5 1 P. L. 103- 62, 107 Stat. 285. 2 This report is one of a
series of reports on the 24 Chief Financial Officers (CFO) Act of 1990 (P.
L. 101- 576, 104 Stat. 2838) agencies? fiscal year 2000 performance reports
and fiscal year 2002 performance plans. 3 Observations on the National
Aeronautics and Space Administration?s Fiscal Year 1999
Performance Report and Fiscal Year 2001 Performance Plan (GAO/ NSIAD- 00-
192R, June 30, 2000). 4 NASA: Status of Achieving Key Outcomes and
Addressing Major Management
Challenges (GAO- 01- 868, July 31, 2001). 5 NASA?s fiscal year 2002
performance plan identifies an objective closely related to this key
outcome. That objective is to operate the space station to advance science,
exploration, engineering, and commerce. We based our assessment on that
objective.
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
As agreed, using the selected key outcomes for NASA as a framework, we (1)
assessed NASA?s planned progress toward achieving these outcomes, and the
strategies the agency has in place to achieve them; and (2) compared NASA?s
fiscal year 2002 performance plan with the agency?s prior year performance
plan for these outcomes. Additionally, we agreed to analyze how NASA
addressed its major management challenges, including the governmentwide
high- risk areas of strategic human capital management and information
security, that we and NASA?s Office of Inspector General (OIG) identified.
Appendix I provides detailed information on how NASA addressed these
challenges. Appendix II contains NASA?s comments on a draft of our report.
NASA has improved its fiscal year 2002 performance plan and responded to
recommendations or suggestions by us and others to make its plan more
useful- particularly by providing more comprehensive explanations on how it
plans to verify and validate performance data and by better portraying how
its performance goals will benefit the public. Generally, NASA?s annual
performance goals for its outcomes appear to be objective and help to
measure progress toward the outcomes. However, the plan still does not
explain the reasons for changes in performance goals. Not having these
explanations could hinder the ability to assess NASA?s performance over
time.
Planned outcome: Expanding scientific knowledge of the Earth system.
NASA?s annual performance goals for this outcome appear objective; and many
of the supporting performance indicators increase the measurability of the
performance goals. Also, in response to the NASA Advisory Council?s specific
recommendation, NASA explains how each performance goal will benefit the
public. However, these performance goals and indicators do not reflect
programs and activities being undertaken with other agencies for the
strategic goal, ?observe, understand, and model the Earth system to learn
how it is changing and the consequences for life on Earth,? even though
NASA?s recently updated Strategic Plan identifies 16 federal partners
contributing to this goal. 6
6 We based our assessment of the Earth Science outcome on this strategic
goal. The performance plan does reflect collaboration with other partners
for one strategic goal within this outcome not covered by our review.
Results in Brief
Page 3 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
NASA?s strategies for achieving this outcome appear to be clear and
reasonable.
Planned outcome: Expanding the commercial development of space. NASA
generally presents performance goals that appear objective and measurable
for this outcome and explains how each will benefit the public. Although the
agency?s strategies for meeting these performance goals are generally clear
and reasonable, there are some exceptions. For example, the goal to develop
and test competing technologies for human missions beyond low earth orbit in
cooperation with international partners does not address how NASA will
actually test those technologies.
Planned outcome: Deploying and operating the International Space Station
safely and cost- effectively.
NASA?s performance goals for this outcome appear objective and measurable,
and NASA explains how they will benefit the public. However, one of the
indicators for the safety performance goal is not articulated
understandably, which makes it difficult to ascertain its relationship to
the performance goal or assess its measurability. Also, the plan does not
clearly indicate how NASA will ensure that the safety goal is achieved, nor
does it show how the agency will achieve the remaining three performance
goals of (1) demonstrating space station progress and readiness at a level
sufficient to show adequate readiness in the assembly schedule, (2)
successfully completing 90% of the space station?s planned mission
objectives, and (3) demonstrating progress toward developing space station
research hardware. NASA does not sufficiently elaborate on the nature of its
space station budget accountability reforms mentioned in the plan, nor how
the reforms will address space station cost growth, a longstanding and
ongoing management problem. Furthermore, it does not acknowledge anywhere in
the plan that the International Space Station cost control issue is a
management challenge- although it does so for some of the other challenges.
NASA has made improvements in its fiscal year 2002 performance plan when
compared to its 2001 plan. In addition to its verification and validation
efforts, NASA adds several features that enhance the format and/ or content
of its performance plan, including a better characterization of its annual
performance measures as ?goals? instead of ?targets.? It also adds
discussions on how its annual performance goals will benefit the public and
displays annual performance goals and associated performance assessments for
fiscal years 1999 to 2002, to help demonstrate cumulative
Page 4 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
progress towards achievement of strategic goals and objectives. Moreover,
NASA adds an agencywide human capital objective and associated annual
performance goals and indicators to the plan. However, NASA can further
enhance the credibility of its verification and validation efforts by being
more forthcoming about possible limitations in its performance data.
Furthermore, it can relate its human capital performance goals and
indicators to specific programs where issues of critical staffing shortages
have been identified. It can also enhance understanding of its rationale for
characterizing successful achievement despite not meeting all supporting
performance indicators for its goals by including in the plan an explanation
for its decision to use this approach. And it can provide a clear rationale
for how information technology- related strategies and programs will
contribute specifically to the achievement of its goals and show the
allocation of information technology- related dollars and personnel to
performance goals.
In assessing how NASA addressed major management challenges, we found the
agency?s performance plan has strategic objectives and performance goals and
measures related to two of our governmentwide high- risk areas- strategic
human capital management and information security. While NASA?s inclusion of
human capital as a strategic objective is an improvement, it did not tie its
performance goals and indicators to specific programs facing human capital
shortages, such as the space shuttle program. Regarding information
security, the plan?s performance goals are not fully responsive to
recommendations we made in our 1999 report.
In addition, we identified three other challenges facing NASA: (1)
correcting contract management weaknesses, (2) controlling International
Space Station costs, and (3) effectively implementing the faster-
bettercheaper approach to space exploration projects. We found that NASA?s
performance plan has an annual performance goal and measures directly
related to the challenge of correcting contract management weaknesses.
However, until NASA?s Integrated Financial Management System- which is
central to providing effective management and oversight over its procurement
dollars- is operational, performance and cost assessments may be incomplete.
The plan contains an annual performance goal and indicator that are
indirectly related to the challenge of controlling International Space
Station costs. But the plan does not indicate the extent NASA will address
space station cost growth. Furthermore, the plan does not include goals and
measures directly related to effective implementation of the faster- better-
cheaper approach to space exploration projects.
Page 5 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
We provided copies of a draft of this report to NASA for its review and
comment. In written comments on the report, NASA generally agreed with the
information presented in the report and noted several improvements it would
make to its fiscal year 2001 performance report and/ or fiscal year 2003
performance plan. NASA?s written comments are included as appendix II.
GPRA is intended to shift the focus of government decisionmaking,
management, and accountability from activities and processes to the results
and outcomes achieved by federal programs. New and valuable information on
the plans, goals, and strategies of federal agencies has been provided since
federal agencies began implementing GPRA. Under GPRA, annual performance
plans are to clearly inform the Congress and the public of (1) the annual
performance goals for agencies? major programs and activities, (2) the
measures that will be used to gauge performance, (3) the strategies and
resources required to achieve the performance goals, and (4) the procedures
that will be used to verify and validate performance information. These
annual plans, issued soon after transmittal of the President?s budget,
provide a direct linkage between an agency?s longer- term goals and mission
and day- to- day activities. 7 Annual performance reports are to report
subsequently on the degree to which performance goals were met. The issuance
of the agencies? performance reports, due by March 31 of each year,
represents a new and potentially more substantive phase in the
implementation of GPRA- the opportunity to assess federal agencies? actual
performance for the prior fiscal year and to consider what steps are needed
to improve performance, and reduce costs in the future. 8 NASA?s final
performance plan was provided to the Congress on July 17, 2001.
NASA?s mission encompasses human exploration and development of space, the
advancement and communication of scientific knowledge, and research and
development of aeronautical and space technologies. Its activities span a
broad range of complex and technical endeavors- from investigating and
evaluating the composition and resources of Mars; to working with
international partners to complete and operate the International Space
Station; to providing satellite and aircraft observations
7 The fiscal year 2002 performance plan is the fourth of these annual plans
under GPRA. 8 The fiscal year 2000 performance report is the second of these
annual reports under GPRA. Background
Page 6 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
of earth for scientific and weather forecasting purposes; to developing new
technologies designed to improve air flight safety.
This section discusses our analysis of NASA?s performance goals and measures
and strategies the agency has in place, particularly strategic human capital
management 9 and information technology, for accomplishing the outcomes. In
discussing these outcomes, we have also provided information drawn from our
prior work on the extent to which the agency provides assurance that its
reported performance information will be credible.
NASA revised its strategic goal and most of its objectives for this key
outcome in its fiscal year 2002 performance plan. We based our assessment on
the strategic goal to observe, understand, and model the Earth system to
learn how it is changing and the consequences of change for life on this
planet. The previous goal was to expand scientific knowledge by
characterizing the Earth system.
NASA?s performance plan does not explain why it adjusted its previous
strategic goal and objectives, nor does it explain why it developed newly
formulated annual performance goals and supporting indicators for fiscal
year 2002. In discussing the reasons for these changes, NASA officials told
us that changes in goals and measures were necessitated by the formulation
of new strategic science questions for the Earth Science Enterprise and a
refocused strategic plan. In our view, providing this explanation in the
plan would have been useful. The performance plan includes a chart that
displays annual performance goals and associated performance assessments for
fiscal years 1999 to 2002, to help demonstrate cumulative progress towards
achievement of strategic goals and objectives and to facilitate performance
trend analysis. However, explaining changes in goals and measures over time
would improve the performance trend analyses, and clarify the reasons for
the new measures. On the other hand, by changing its performance goals
annually, NASA
9 Key elements of modern strategic human capital management include
strategic human capital planning and organizational alignment; leadership
continuity and succession planning; acquiring and developing staffs whose
size, skills, and deployment meet agency needs; and creating results-
oriented organizational cultures. Assessment of NASA?s
Performance Goals and Strategies for Accomplishing Selected Key Outcomes
Scientific Knowledge of the Earth System
Page 7 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
could hinder its ability to make comparisons between fiscal years and
effectively analyze trends in performance.
Generally, NASA?s fiscal year 2002 annual performance goals for this outcome
appear to be objective and help to measure progress toward achieving it.
Specific ways to measure the performance goals are established through two
or more indicators that in many cases provide specific, quantifiable values
that increase the measurability of the performance goal. An example of such
an indicator is: ?Increase the coverage of space- based maps of coral reef
distribution by 25 percent beyond current estimates using remotely sensed
imagery.? NASA added discussions on how each performance goal would benefit
the public, as recommended by the NASA Advisory Council in its evaluation of
NASA?s fiscal year 2000 performance report. In some cases these discussions
clearly articulate the benefit to the public; in other cases they only
provide descriptive or background information. For example, one performance
goal calls for increasing the understanding of stratospheric ozone changes,
as the abundance of ozone- destroying chemicals decreases and new
substitutes increases. The public benefit statement, ?Reduction in
atmospheric ozone amounts leads to an increased flux of ultraviolet
radiation at the Earth?s surface, with harmful effects on plant and animal
life including human health,? explains the effect of reduced ozone amounts
rather than how the goal will benefit the public.
NASA indicates that it will consider many of the Earth Science goals as
fully met if a specified number of the supporting indicators (such as 3 out
of 4) are achieved in fiscal year 2002. NASA officials told us that this
approach allows for some flexibility in rating success. Specifically, since
research and development by its very nature is unpredictable, these
officials believe that, for example, not meeting all indicators still
implies significant progress in achieving scientific goals. NASA could fully
explain in the plan why it does not believe it has to meet all supporting
indicators. This would put its actual performance in the proper perspective.
The following is one example of such a goal:
?Annual performance goal - Increase understanding of global precipitation,
evaporation and how the cycling of water through the Earth system is
changing by meeting at least 3 of 4 performance indicators.
Page 8 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Indicators Combine analysis of global water vapor, precipitation and wind
data
sets to decipher variations (and possible trends) in the cycling of water
through the atmosphere and their relation to sea surface temperature
changes. Analyze data from polar and geostationary satellites in a
consistent
fashion over at least two decades to evaluate whether the detectable
moisture fluxes are increasing beyond the expected ranges of natural
variability. Determine the time and spatial variability of the occurrence
of
strong convection regions, precipitation events, and areas of drought to
assess whether or not there are discernable global changes in the
distribution of moisture availability useful to food and fiber production
and management of fresh water resources. Establish passive and active
rainfall retrievals of zonal means to
establish a calibration point for long- term data records of the World
Climate Research Program, Global Precipitation Climatology Project (GPCP).?
Concerning interagency and crosscutting activities, we note that within this
outcome, NASA does not include annual performance goals and indicators that
reflect programs or activities being undertaken with other agencies in
fiscal year 2002 for the strategic goal covered by our review, even though
NASA?s latest Strategic Plan identifies 16 federal agencies that contribute
to this goal. 10 These agencies include the departments of Defense and
Commerce, the National Oceanic and Atmospheric Administration, and the
Federal Emergency Management Agency. If NASA has planned collaborative
efforts related to the performance goals and indicators for this outcome,
these are not identified in the performance plan. The performance plan
provides an opportunity to evidence coordination among crosscutting programs
and reflect the expected contribution of other agencies toward related
goals.
NASA states that its implementation strategy for Earth Science research
programs is focused on a set of strategic science questions directed at
understanding how the Earth system is changing and the consequences for
10 The performance plan does include an objective and annual performance
goals that reflect collaboration with other agencies for one strategic goal
within this outcome not covered by our review. That strategic goal is to
develop and adopt advanced technologies to enable mission success and serve
national priorities.
Page 9 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
life on Earth. The plan indicates that these questions can be addressed
effectively with NASA?s capabilities, which include observational programs,
research and analysis, modeling, and advanced technology. In general, the
plan provides clear and reasonable information technology. One annual
performance goal within this outcome is to successfully disseminate Earth
Science data to enable NASA?s Earth Science research and applications goals
and objectives. To achieve this goal, NASA set several specific performance
indicators, such as increasing the number of distinct NASA Earth Observing
System Data and Information System (EOSDIS) customers by 20 percent compared
to fiscal year 2001; increasing scientific and applications- data products
delivered from the Earth Observing System (EOS) Distributed Active Archive
Centers (DAAC) by 10 percent compared to fiscal year 2001; and increasing
the number of favorable comments from DAAC and other users over fiscal year
2001 and decreasing the total percentage of order errors by 5 percent over
fiscal year 2001. These indicators provide specific, quantifiable ways to
measure increases in output from NASA?s EOSDIS and DAACs. Based on NASA?s
reported success in meeting similar indicators for fiscal year 2000, these
indicators appear to be reasonable for fiscal year 2002.
In some instances, NASA revised or added strategic goals and objectives, and
annual performance goals within this outcome for fiscal year 2002. For
example, the agency added (1) a new strategic goal to provide commercial
industry with the opportunity to meet NASA?s future launch needs, including
human access to space, with new launch vehicles that promise to dramatically
reduce cost and improve safety and reliability and (2) a new strategic
objective to develop new capabilities for human space flight and commercial
applications through partnerships with the private sector. Furthermore, most
of the annual performance goals are either new or revised from targets in
NASA?s prior year performance plan. Further, NASA does not provide any
rationale or reasons for the changes in the plan. NASA officials told us
that, generally, the changes were made to (1) improve NASA?s ability to
assess progress toward achieving goals and objectives; (2) reflect
commitment to safety and privatization efforts; or (3) reflect the broader
scope of programs and activities and shifts in Enterprise responsibilities.
Again, providing such explanations in the performance plan, in our view,
would have been useful.
NASA displays its annual performance goals and associated performance
assessments for fiscal years 1999 to 2002 to help demonstrate cumulative
progress towards achievement of strategic goals and objectives and to
facilitate performance trend analysis. As emphasized earlier, changing
Commercial Development
of Space
Page 10 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
performance goals annually could hinder NASA?s ability to make comparisons
between years and effectively analyze trends in performance. NASA generally
presents performance goals that appear to be objective and help to measure
progress toward this outcome. The agency also explains how each performance
goal will benefit the public.
The strategies for achieving the performance goals are generally clear and
reasonable. However, there are some exceptions. For example, one performance
goal involves developing and testing - on the ground and in space- competing
technologies for human missions beyond low earth orbit in cooperation with
international partners. One indicator related to this performance goal
involves organizing and conducting an ?international forum? at which
preliminary concepts, plans, and technology options for future human/
robotic exploration and development of space would be reviewed. However, the
indicator does not address the testing of competing technologies. Also, NASA
has a performance goal to engage the commercial community and encourage non-
NASA investment in commercial space research by meeting at least three of
four performance indicators, but the plan does not state why all of the
supporting indicators will not be achieved.
Since the selected key outcome of deploying and operating the space station
safely and cost effectively is not included in NASA?s fiscal year 2002
performance plan as a specific strategic goal or objective, we based our
assessment of it on a related strategic objective in the plan- to operate
the space station to advance science, exploration, engineering, and
commerce.
NASA set four annual performance goals for this outcome for fiscal year
2002. The performance goals are new, but the plan does not provide any
rationale for the changes. This is an important omission, because as pointed
out earlier, explaining changes in goals and measures over several years
would improve performance trend analyses and clarify why such changes were
made. In discussing the reasons for these changes, NASA officials told us
that for the International Space Station, the fiscal year 2001 goals and
objectives relied on milestones that were reported as either complete or
incomplete, with no provision for reporting progress toward completion. The
improved goals and objectives for fiscal year 2002 are tied to milestones
that allow reporting of progress in terms of the percent of the milestones
completed. Thus, the new measures will provide greater visibility and
improve NASA?s ability to assess progress toward achieving goals and
objectives. In our view, providing this explanation in the plan
International Space
Station?s Operation, Cost, and Safety
Page 11 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
would have enhanced NASA?s discussion on this outcome. Also, similar to the
previous outcomes, NASA displays performance goals and associated
assessments for fiscal years 1999 to 2002 to help demonstrate cumulative
progress towards achievement of strategic goals and objectives and to
facilitate performance trend analysis. However, changing performance goals
annually could hinder NASA?s ability to make comparisons between fiscal
years and effectively analyze trends in performance.
Generally, NASA?s four annual performance goals and supporting indicators
for the space station outcome appear to be objective and help to measure
progress toward this outcome. In addition, the agency explains how the goals
will benefit the public, stating how completing them successfully will
provide many benefits of space research through new discoveries and improved
technological applications in areas such as medicine, industrial processes,
and fundamental knowledge. One performance goal addresses space station
safety. Specifically, the goal is to demonstrate space station on- orbit
vehicle?s operational safety, reliability, and performance. The goal has an
indicator that provides for zero safety incidents (such as no on- orbit
injuries), which appears reasonable. The other indicator is not articulated
understandably, making it difficult to ascertain its relationship to the
performance goal or to assess its measurability. (The language is phrased
as: ?Actual resources available to the payloads measured against the planned
payload allocation for power, crew time, and telemetry.?) Also, the plan
does not clearly indicate the means or strategies NASA will use to ensure
that the safety performance goal is achieved in fiscal year 2002. Similarly,
the plan does not provide clear strategies for achieving the remaining three
performance goals of (1) demonstrating space station progress and readiness
at a level sufficient to show adequate readiness in the assembly schedule,
(2) successfully completing 90% of the space station?s planned mission
objectives, and (3) demonstrating progress toward space station research
hardware development.
NASA does not address space station cost control as part of this outcome.
However, within its commercialization of space outcome, NASA set a
performance goal in fiscal year 2002 to develop and execute a management
plan and open future ISS hardware and service procurements to cost-
effective innovation through competition, including launch services and a
non- governmental organization for space station research. NASA?s indicator
for the management plan includes reforms that (1) strengthen its
headquarters involvement, (2) increase communications, (3) provide more
accurate assessment, and (4) maintain budget accountability. NASA reports
that the benefit to the public of the
Page 12 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
management plan and reforms is to ensure that future space station costs
will remain within the President?s fiscal year 2002 budget plan. In our
view, NASA?s discussion of the proposed management plan is minimal and lacks
specificity. While the management plan will reportedly include ISS budget
accountability reforms, NASA does not elaborate on the nature of such
reforms or indicate to what extent this plan will address space station cost
growth, a long- standing management problem. Furthermore, NASA does not
acknowledge anywhere in the performance plan that space station cost control
is a major management challenge, although it has done so for some of the
other challenges. In past years and as recently as January 2001, we have
identified the need to control space station costs as a major management
challenge for NASA. 11 We believe that the agency has the opportunity to use
the completed management plan to facilitate the development of space station
cost control measures in future annual performance plans. The lack of
performance measures that address space station cost control is a
shortcoming that we have identified in our previous reviews of the agency?s
annual performance plans and reports.
For the selected key outcomes, this section describes major improvements or
remaining weaknesses in NASA?s fiscal year 2002 performance plan in
comparison with its fiscal year 2001 performance plan. It also discusses the
degree to which the agency?s fiscal year 2002 plan addresses concerns and
recommendations by the Congress, GAO, NASA?s OIG, and others.
NASA?s fiscal year 2002 performance plan differs in several significant ways
from the prior plan. First, NASA portrays its planned efforts to verify and
validate performance information more comprehensively than in 2001,
providing greater confidence that the performance results will be credible.
In our review of NASA?s 2001 plan, we criticized the agency for not
explicitly describing those efforts and for not addressing data limitation
issues and problems. The 2002 plan includes specific agency data bases and
describes methods NASA will rely on to support the credibility of reported
performance information. For example, the plan references the NASA Personnel
Payroll System, Incident Reporting System, Financial and
11 Major Management Challenges and Program Risks: National Aeronautics and
Space Administration (GAO- 01- 258, Jan. 2001). Comparison of
NASA?s Fiscal Year 2002 Performance Plan With the Prior Year Pl an
Comparison of Performance Plans for Fiscal Years 2001 and 2002
Page 13 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Contractual Status of Programs System, and NASA Environmental Tracking
System as specific data bases that will be used to verify and validate
performance data. The plan describes specific processes in place to support
performance claims associated with NASA?s Integrated Financial Management
System, performance- based contracts, contracts awarded to small and small
disadvantaged businesses, and information technology. And it describes a
broad array of methods to verify and validate reported performance data such
as monthly reports from NASA field centers, Web statistics, count of
publications, and NASA?s Education Computer Aided Tracking System. Despite
improvements in addressing data verification and validation methods, NASA
still does not acknowledge data limitations that could hinder performance
measurement. We continue to believe that NASA can further enhance the
credibility of its verification and validation procedures and the usefulness
of its performance data by disclosing the expected limitations of its
performance data in its annual performance plans. A March 2001 NASA Office
of Inspector General report identified limitations in NASA?s fiscal year
2000 performance data and indicated that NASA would discuss anticipated data
limitations in its performance planning beginning with its fiscal year 2002
final performance plan. 12 However, we reviewed the final version of the
plan, and such a discussion is not included.
Second, several added features help to enhance the format and/ or content of
the fiscal year 2002 plan. NASA?s use of ?annual performance goals? in the
plan characterizes its annual performance measures more clearly than the
?annual performance targets,? used in previous plans. The addition of
discussions on how annual performance goals benefit the public helps to
better understand the linkage between the goal and the expected results,
although in some cases additional clarification could even better convey the
actual benefit to the public. Value is added to the plan by NASA?s display
of annual performance goals and associated performance assessments for
fiscal years 1999 to 2002, to help demonstrate cumulative progress towards
achievement of strategic goals and objectives and facilitate performance
trend analysis. However, changes in performance goals over many years could
hinder NASA?s ability to make comparisons between years and effectively
analyze trends in performance.
12 Validation And Verification Of Selected NASA FY 2000 Performance Data
Related To The Government Performance And Results Act (GPRA), (IG- 01- 020,
Mar. 30, 2001).
Page 14 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Also, this year?s plan includes an agencywide strategic objective to invest
in the use of human capital. NASA set two annual performance goals for
fiscal year 2002 as progress towards this objective: (1) align management of
human resources to best achieve agency strategic goals and (2) attract and
retain a workforce that is representative at all levels of America?s
diversity. However, there are no human capital initiatives specifically
linked to the outcomes or annual performance goals and indicators that link
to specific programs, such as the space shuttle program. (See details under
management challenges.)
Third, NASA could explain in the plan what procedures it has used to
characterize its performance goals as fully achieved when it has not met all
of the supporting indicators for those goals. This is particularly true for
the Earth Science outcome. Providing such an explanation would put the
actual performance in the proper perspective.
Fourth, similar to the prior plan, the fiscal year 2002 plan still does not
provide a clear rationale for how information technology- related strategies
and programs will contribute specifically to achievement of NASA?s goals or
show any allocation of information technology- related dollars and personnel
to performance goals. Goals for managing information technology are
generally stated in terms of broad categories for improvement, such as
increased capability and efficiency and enhanced security, and include few
quantitative measures. One exception is the goal of increasing dissemination
of Earth Science data, which is accomplished through EOSDIS. The plan sets
several specific goals for increasing the volume and distribution of Earth
Science data and products.
Lastly, in our review of NASA?s fiscal year 2001 plan, we suggested that
NASA document in its annual performance plans and reports, the rationale for
establishing new performance targets to clarify the reasons for adding such
targets. We had noted that while many of NASA?s annual performance targets
were new each year, there was no stated basis for the changes. In its fiscal
year 2002 performance plan, NASA has formulated new annual performance goals
and has changed many of its strategic goals and objectives without including
the reasons for doing so. We continue to believe that providing the
rationale for these changes will clarify the reasons for the new goals and
measures and augment the value of performance trend analyses. Also, the plan
does not indicate whether or not achieving any specific goals would be
negatively affected by external factors. However, like the prior plan, the
fiscal year 2002 plan states that successful execution of NASA?s strategic
goals and objectives depends on receipt of its requested appropriations, as
well as provision of funds,
Page 15 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
materials, or services, that have been committed to the cooperative
agreements or partnerships which are referenced in the performance plan.
We have identified two governmentwide high- risk areas: strategic human
capital management and information security. Regarding strategic human
capital management, NASA?s fiscal year 2002 performance plan contains a
strategic objective and annual performance goals and indicators directly
related to human capital. Concerning information security, NASA?s
performance plan contains a strategic objective, an annual performance goal,
and indicators directly related to this management challenge. The plan
states that safety and security is one of four areas on which NASA?s
information technology planning is focused. The fiscal year 2002 plan is an
improvement over the 2001 plan, which did not include quantifiable measures
for improving information security. However, the plan?s performance goals do
not fully respond to the recommendations we made in 1999 when we reported
that the agency lacked an effective agencywide security program. 13 For
example, the plan sets a performance indicator of completing 90 percent of
information technology security plans for critical systems. However, we
recommended that all systems be formally authorized before they became
operational and at least every 3 years thereafter.
In addition, we have identified three major management challenges facing
NASA: (1) correcting contract management weaknesses, (2) controlling
International Space Station costs, and (3) effectively implementing the
faster- better- cheaper approach to space exploration projects. We found
that NASA?s performance plan contains an annual performance goal and
indicators directly related to the problem of contract management. It is
important to note that until NASA?s Integrated Financial Management System--
which is central to providing effective management and oversight over its
procurement dollars- is operational, performance assessments relying on cost
data may be incomplete and full costing will be only partially implemented.
While NASA?s performance plan contains an annual performance goal and an
indicator that indirectly addresses the challenge of controlling space
station costs, it does not indicate the extent that NASA will address space
13 Information Security: Many NASA Mission- Critical Systems Face Serious
Risks
(GAO/ AIMD- 99- 47, May 20, 1999). NASA?s Efforts to
Address its Major Management Challenges Identified by GAO
Page 16 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
station cost growth. As we discussed in our January 2001 report, the
International Space Station Program continues to face cost control
challenges. As with contract management, until the Integrated Financial
Management System is operational, NASA may lack the cost information needed
to control space station costs.
Further, NASA?s performance plan did not directly address the challenge of
effectively implementing the faster- better- cheaper approach to space
exploration projects. In January 2001, we also reported that NASA faces
significant challenges as it attempts to create highly reliable missions and
foster open communications under the budget constraints of the agency?s
faster- better- cheaper space exploration strategy. In addition, the real
success of this strategy will require a comprehensive integration of lessons
learned from failures on an agencywide basis. Until NASA resolves these
problems, its financial resources are vulnerable to inefficient use.
Appendix I provides detailed information on how NASA addressed these
challenges and high- risk areas as identified by GAO and NASA?s Office of
Inspector General (OIG).
As agreed, our evaluation was generally based on the requirements of GPRA,
guidance to agencies from the Office of Management and Budget (OMB) for
developing performance plans (OMB Circular A- 11, Part 2), previous reports
and evaluations by us and others, our knowledge of NASA?s operations and
programs, our identification of best practices concerning performance
planning, and our observations on NASA?s other GPRA- related efforts. We
also discussed our review with NASA officials and with officials of NASA?s
OIG. The agency outcomes that were used as the basis for our review were
identified by the Ranking Minority Member of the Senate Committee on
Governmental Affairs as important mission areas for NASA and do not reflect
the outcomes for all of NASA?s programs or activities. The major management
challenges confronting NASA, including the governmentwide high- risk areas
of strategic human capital management and information security, were
identified in our January 2001 performance and accountability series and
high risk update, and by NASA?s OIG in December 2000. We conducted our
review from August 2001 through October 2001 in accordance with generally
accepted government auditing standards.
We provided copies of a draft of this report to NASA for its review and
comment. In written comments on the report, NASA generally agreed with the
information presented in the report and noted several improvements it Scope
and
Methodology Agency Comments
Page 17 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
would make. Concerning our suggestion that NASA could fully explain in its
performance plan why it believes it is not necessary to achieve all
performance indicators to demonstrate annual performance goal achievement,
NASA stated that it would provide a statement containing the supporting
rationale for this approach in its fiscal year 2003 performance plan.
In responding to our observation that the fiscal year 2002 plan lacked
sufficient detail on the nature of the ISS?s budget accountability reforms
or how the reforms will address longstanding and ongoing management
problems, including cost growth, NASA commented that the reforms are
contained in its Program Management Action Plan that will be referred to in
the fiscal year 2003 performance plan. We note that the ISS Program is being
restructured in response to a potential cost growth of $4. 8 billion. The
restructuring has raised widespread concerns about the potential science
benefits to be realized by the United States and international partners. For
this reason, we believe it is increasingly important for NASA?s performance
plan to provide a clear path showing how NASA intends to implement the
needed reforms and how the reforms will add credibility to future ISS
budgets and resolve the uncertainties concerning the utility of the ISS.
NASA also commented on a statement in our draft report that the agency does
not provide a clear rationale for how IT- related strategies and programs
will contribute specifically to achievement of its goals or show the
allocation of IT- related dollars and personnel to performance goals.
According to NASA, the IT service delivery metric in the plan aggregates
each major IT service, such as NASA?s Integrated Services Network. Remaining
IT investments are embedded in each NASA project and managed as part of the
project. While this statement may be true on the individual program level,
it does not address GPRA objectives to demonstrate how IT- related
strategies and programs contribute specifically to the achievement of agency
goals or show the allocation of related resources.
Finally, in response to our observation regarding NASA?s lack of
explanations in the 2002 plan for annual performance changes, NASA agreed
that including such explanations in the plan would be useful and that it
would characterize reasons for annual performance changes in its 2003
performance plan.
As arranged with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days after
the
Page 18 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
date of this letter. At that time, we will send copies to appropriate
congressional committees; the NASA Administrator; and the Director, Office
of Management and Budget. Copies will also be made available to others on
request.
If you or your staff have any questions, please call me at (202) 512- 4841.
Key contributors to this report were Richard J. Herley, Shirley B. Johnson,
Charles W. Malphurs, Christina Chaplain, John de Ferrari, Diane G. Handley,
and Fannie M. Bivins.
Sincerely yours, Allen Li Director, Acquisition and
Sourcing Management
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 19 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
The following table identifies the major management challenges confronting
NASA, including the governmentwide high- risk areas of strategic human
capital management and information security. The first column of the table
lists the management challenges that we and/ or NASA?s Office of Inspector
General (OIG) have identified. The second column discusses the extent to
which NASA?s fiscal year 2002 performance plan includes performance goals
and measures to address the challenges that we and the OIG identified. Of
the agency?s fifteen major management challenges, its performance plan has
(1) goals and measures that are directly related to thirteen of the
challenges; (2) a goal and measure indirectly applicable to one challenge;
and (2) no goals and measures directly related to one of the challenges.
Some of the NASA performance plan?s goals and measures we discuss may not
track specifically with the key considerations of NASA OIG?s management
challenges since the challenges themselves were presented in a broad
context. GAO has performed reviews affecting a number of the areas
mentioned. This appendix highlights the results of our assessments, where
applicable. Appendix I: Observations on NASA?s Efforts
to Address Its Major Management Challenges
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 20 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Table 1: Major Management Challenges Major management challenge Applicable
goals and measures in the fiscal year 2002
performance plan GAO- designated governmentwide high risk
Strategic Human Capital Management: GAO has identified shortcomings at
multiple agencies involving key elements of modern human capital management,
including strategic human capital planning and organizational alignment;
leadership continuity and succession planning; acquiring and developing
staffs whose size, skills, and deployment meet agency needs; and creating
results- oriented organizational cultures.
In August 2000, we reported that several internal NASA studies had shown
that the agency?s space shuttle program?s workforce had been affected
negatively by NASA?s downsizing, much of which occurred after 1995. We also
reported that NASA had begun taking actions to address its shuttle workforce
problems. a In September 2001, we reported in testimony that while NASA
continues to make progress in revitalizing the shuttle program?s workforce,
considerable challenges remain. b
In January 2001, we also reported the need to implement a human capital
approach in NASA?s workforce management strategies as a major management
challenge. c
The plan contains a strategic objective, annual performance goals and
indicators directly related to this management challenge. The plan?s
strategic objective is to invest wisely in NASA?s use of human capital,
developing and drawing upon the talents of all NASA?s people. This objective
applies to all of NASA?s workforce and has two related performance goals:
(1) align management of human resources to best achieve agency strategic
goals and objectives and (2) attract and retain a workforce that is
representative at all levels of America?s diversity. NASA further indicated
in discussion under the second performance goal: (1) in fiscal year 2002,
the agency will develop a process by which the centers will implement
consistent workforce planning resulting in a plan for each center that links
staffing, funding resources, mission and activities, and core competencies;
(2) NASA intends in fiscal year 2002 to develop an initiative to enhance the
centers? recruitment capabilities, focusing on hiring new college graduates
to counterbalance the aging of the workforce due to the halt in the influx
of new college graduates during the years of downsizing; and (3) NASA
requested additional fiscal year 2002 resources to expand training delivery
methods and emphasize the development of computer- based training
alternatives that can be accessed at all locations and levels. While NASA?s
inclusion of human capital in its plan as a strategic objective is an
improvement, it needs to relate its human capital annual performance goals
and indicators to specific programs where issues of critical staffing needs
have been identified, such as the space shuttle program.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 21 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Information Security: Our January 2001 high- risk update noted that the
agencies? and governmentwide efforts to strengthen information security have
gained momentum and expanded. Nevertheless, recent audits continue to show
federal computer systems are riddled with weaknesses that make them highly
vulnerable to computer- based attacks and place a broad range of critical
operations and assets at risk of fraud, misuse, and disruption.
In 1999, we reported that NASA lacked an effective agencywide security
program that includes improvements in five areas: assessing risks and
evaluating needs, implementing policies and controls, monitoring compliance
with policy and effectiveness of controls, providing computer security
training, and coordinating responses to security incidents. d The need for
such a framework was serious; tests we conducted at 1 of NASA?s 10 field
centers found that mission- critical information systems were vulnerable to
unauthorized access. We successfully penetrated several of these systems,
including one responsible for calculating detailed positioning data for
earth- orbiting spacecraft and another that processes and distributes
scientific data received from these spacecraft.
The plan contains a strategic objective and an annual performance goal and
indicators directly related to this management challenge. The plan states
that safety and security is one of four areas on which NASA?s information
technology planning is focused. A performance goal is established to enhance
information technology security by meeting established performance
indicators in three critical areas: (1) vulnerabilities detected, (2)
training, and (3) information technology security plans. The plan includes
specific target percentages for reducing known system vulnerabilities,
training NASA personnel, and preparing information technology security
plans. These are all areas where problems have been documented at NASA in
the past, including our review of information security for NASA?s mission
critical systems, completed in 1999. The fiscal year 2002 plan is an
improvement over the 2001 plan, which did not include quantifiable measures
for improving information technology security. However, the plan?s
performance goals are not fully responsive to the recommendations we made in
1999. For example, the plan sets a performance indicator of completing 90
percent of information technology security plans for critical systems. We,
on the other hand, recommended that all systems be formally authorized
before they became operational and at least every 3 years thereafter.
Likewise, NASA?s performance indicator for information technology security
training sets employee training targets between 80 and 95 percent for
providing awareness training to different types of NASA employees. We
recommended that NASA establish a more comprehensive program that would
include certifying that NASA civil servants and contract employees are
competent to discharge their information technology security- related
responsibilities.
GAO- designated major management challenges
The Need to Correct Weaknesses in NASA?s Contract Management: We have
reported that NASA?s contract management is a continuing area of high risk.
Implementation of the financial management system and its integration with
full cost accounting have been delayed. Until the Integrated Financial
Management System is operational, performance assessments relying on cost
data may be incomplete. We have also reported that NASA is continuing to
rely on undefinitized change orders- that is, contract changes initiating
new work before NASA and the contractor agree on a final estimated cost and
fee- to complete work on its largest space station contract. We stated that
this is a risky way of doing business because it increases the potential for
unforeseen cost increases and scheduling delays.
The plan contains an annual performance goal and indicators directly related
to this management challenge. The performance goal is to improve the
agency?s financial management and accountability. This goal has two
indicators (1) cost at least 75 percent of the budget available to cost
during fiscal year 2002; and (2) initiate the pilot phase of the core
financial project and at least one other project module of NASA?s third
attempt at implementing an Integrated Financial Management System (IFMS).
The plan also states (1) an agency- level project team is in place at the
Marshall Space Flight Center, the lead center for the core financial
project, and the design phase will begin in February 2002 and (2) three
?pathfinder? projects have begun to test out the processes and technical
requirements for agency- wide implementation of new administrative systems.
The IFMS is key to producing accurate and reliable information for full-
cost accounting. Furthermore, the plan does not address the issue of
undefinitized change orders.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 22 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
The Need to Control International Space Station (ISS) Development and
Support Costs: We have reported that the ISS program continues to face cost
control challenges. e NASA?s OIG also reported that the ISS program
continued to experience cost overruns and scheduling delays.
The plan contains an annual performance goal and indicator that indirectly
addresses ISS cost control issues. The performance goal is to develop a
management plan and initiate ISS reforms, but the plan does not indicate to
what extent NASA will address space station cost growth. This challenge is
discussed in detail in the outcomes section of this report.
The Need to Effectively Implement the Faster- Better- Cheaper Approach to
Space Exploration Projects: We have reported that NASA faces significant
challenges as it attempts to create highly reliable missions and foster open
communications under the budget constraints of the agency?s faster- better-
cheaper space exploration strategy. In addition, real success will require a
comprehensive integration of lessons learned from failures on an agencywide
basis. Until NASA resolves these problems, its financial resources are
vulnerable to inefficient use.
This was designated as a new major management challenge in January 2001.
The plan does not have goals and measures directly related to this
management challenge.
OIG- designated major management challenges
Safety and Mission Assurance: NASA?s OIG has reported that safety and
mission assurance has become a serious challenge for NASA. Key
considerations to ensure safety in future NASA operations include (1)
ensuring an appropriate level of training for staff who conduct safety
reviews and evaluations; (2) maintaining adequate safety reporting systems;
(3) ensuring variances to standard safety procedures are appropriately
justified, reviewed, and approved; (4) maintaining an effective emergency
preparedness program; (5) ensuring NASA and contractor compliance with
safety standards and regulations; (6) ensuring product safety and
reliability; and (7) ensuring the space shuttle and the ISS maintain crew
safety.
In September 2001, we reported while NASA is making strides in revitalizing
its workforce, its ability to implement safety upgrades in a timely manner
is uncertain. f
The plan contains strategic objectives, annual performance goals and
indicators directly related to this management challenge. In fact, NASA?s
plan contains a high emphasis on safety. The plan has a strategic objective
to protect the safety of people and facilities and the health of the
workforce. This objective?s performance goal directs NASA to increase the
safety of its infrastructure and the health of its workforce through
facilities? safety improvements, reduced environmental hazards, increased
physical security, enhanced safety and health awareness, and appropriate
tools and procedures for health enhancement. There are eight indicators for
this performance goal, which include: (1) no fatalities will result from
NASA mishaps and (2) per the Federal Worker 2000 Initiative, reduce the
overall occurrence of injuries (due to occupational injury or illness) by 3%
per year from the fiscal year 1997 baseline to 1.15 occurrences per 100
workers. The plan also has strategic objectives to (1) provide and make use
of safe, affordable, and improved access to space and (2) ensure the health,
safety, and performance of humans living and working in space. Furthermore,
the Aerospace Technology Enterprise has two related strategic goals (1) to
revolutionize aviation - enable the safe, environmentally- friendly
expansion of aviation and (2) to advance space transportation - create a
safe, affordable highway through the air and into space. The plan also
states that NASA management, in the interest of safety, encourages space
shuttle program managers to set aside metrics when dealing with launches
planned versus launches achieved during a given fiscal year. Our September
2001 testimony noted that NASA is still assessing the full package of its
planned space shuttle program workforce and safety improvements, and some
projects have already encountered funding and scheduling problems.
Overcoming challenges related to the safety upgrades is critical since NASA
will be relying on the space shuttle longer than originally anticipated.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 23 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
International Space Station: NASA?s OIG has reported that the ISS is a
significant management challenge due to significant problems related to ISS
cost, contingency planning, and the X- 38/ Crew Return Vehicle. Key
considerations for continued ISS assembly and operation are (1) managing the
political, financial, technical, and safety challenges presented by an
international partnership; (2) overcoming technical challenges inherent in
manufacturing, assembling, and testing complex hardware and software
components provided by different nations and integrated in space; (3) safely
maintaining, upgrading, and operating a structure as complicated as the
space station; and (4) maximizing the beneficial use of the space station
for scientific research and technology development.
In June 2001, we reported that inadequate planning and design led to ISS
propulsion module failure. g
NASA OIG?s description of the ISS major management challenge is more broadly
focused than our related management challenge. (See discussion under GAO-
designated major management challenge, the need to control ISS development
and support costs.) The plan contains strategic objectives and annual
performance goals directly related to this management challenge. The plan
contains a strategic objective to conduct engineering research on the
International Space Station to enable exploration beyond Earth orbit. This
objective has a performance goal to test, at the International Space
Station, competing technologies for human missions beyond low earth orbit,
in cooperation with other agencies and international partners and with U. S.
industry. The plan also has a strategic objective to operate the
International Space Station to advance science, exploration, engineering,
and commerce. This objective has four performance goals: (1) demonstrate ISS
on- orbit vehicle operational safety, reliability, and performance; (2)
demonstrate ISS program progress and readiness at a level sufficient to show
adequate readiness in the assembly schedule; (3) successfully complete 90
percent of the ISS planned mission objectives; and (4) demonstrate progress
toward ISS research hardware development. The Biological and Physical
Research Enterprise has (1) strategic objectives to develop strategies to
maximize scientific research output on the International Space Station and
other space research platforms; (2) a strategic objective to foster
commercial research endeavors with the International Space Station and other
assets; and (3) a performance goal to highlight ISS- based commercial space
research at business meetings and conferences. The plan also has a strategic
objective to foster commercial endeavors with the International Space
Station and other assets. This objective has a performance goal to develop
and execute a management plan and open future station hardware and service
procurements to innovation and cost- saving ideas through competition,
including launch services and a non- government organization for space
station research. However, the plan does not indicate the extent that NASA
will address space station cost growth. This challenge is discussed in
detail in the outcomes section of this report.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 24 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Information Technology: NASA?s OIG has reported that information technology
has become a serious challenge for NASA. Key considerations for an effective
information technology program include (1) ensuring data security,
integrity, and application controls; (2) protecting operations and
communications with spacecraft; and (3) monitoring and evaluating the
streamlining of operations through outsourcing information technology
operations for cost efficiencies, dependency on the vendor for technological
direction, vulnerability of strategic information to outsiders, and
dependency on the viability of the vendor.
NASA?s OIG reported that during fiscal year 2000 NASA continued to have a
fragmented information technology (IT) security program without clear lines
of authority, policies, guidelines, or enforcement. The OIG reported that
audits of several mission critical information systems disclosed that NASA
had not implemented adequate basic controls in areas such as system access,
protection of critical files, system backup and restore procedures,
privileged operations controls, and system audit and monitoring
capabilities.
(See discussion under governmentwide high- risk challenge: information
security for additional details.)
The plan contains a strategic objective, annual performance goals, and
indicators directly related to this management challenge. The plan states
that safety and security is one of four areas on which NASA?s information
technology planning is focused. The plan?s strategic objective is to enhance
the security, efficiency, and support provided by NASA?s information
technologies resources. The objective has three performance goals to (1)
improve information technology infrastructure service delivery by providing
increased capability and efficiency while maintaining a customer rating of
satisfactory; (2) enhance mission success through seamless, community-
focused electronic service delivery; and (3) enhance information technology
security by meeting established performance indicators in three critical
areas: (a) vulnerabilities detected, (b) training, and (c) information
technology security plans. The plan includes specific target percentages for
reducing known system vulnerabilities, training NASA personnel, and
preparing information technology security plans. These are all areas where
problems have been documented at NASA in the past, including our review of
information security for NASA?s mission critical systems, completed in 1999.
The fiscal year 2002 plan is an improvement over the 2001 plan, which did
not include quantifiable measures for improving information technology
security. However, the plan?s goals are not as comprehensive as the
recommendations we made in 1999. For example, the plan sets a target of
completing 90 percent of information technology security plans, including
authorizations to process, for its critical systems. We, on the other hand,
recommended that all systems be formally authorized before they become
operational and at least every 3 years thereafter. Likewise, NASA?s
performance goal indicator for information technology security training sets
targets between 80 and 95 percent for providing awareness training to
different types of NASA employees. We recommended that NASA establish a more
comprehensive program that would include certifying that NASA civil servants
and contract employees are competent to discharge their information
technology securityrelated responsibilities.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 25 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Procurement: NASA?s OIG has reported that procurement is an ongoing
challenge for NASA. Key considerations for effective procurement at NASA
include (1) ensuring proper levels of staffing to perform contracting
requirements; (2) providing sufficient controls over and monitoring both
prime and subcontractors; (3) implementing or increasing the use of
innovative procurement procedures such as earned value management and
performance incentive fees; and (4) ensuring costs billed to NASA cost- type
contracts, due to the changing industry environment, are reasonable and
allowable.
In August 2001, we reported that NASA?s inability to provide timely data on
obligations or support for actual costs related to the space station and
shuttle raises concerns about NASA being able to achieve the discipline and
accountability called for by the National Aeronautics and Space
Administration Authorization Act for Fiscal Year 2000 (P. L. 106- 391). h
The plan contains a strategic objective, annual performance goals, and
indicators directly related to this management challenge. The plan?s
strategic objective is to achieve the most productive application of federal
acquisition policies. This objective?s first performance goal is to continue
to take advantage of opportunities for improved contract management by
maintaining a high proportion of performance based contracts (PBCs). This
performance goal?s indicator is to maintain PBC obligations at greater than
80% of funds available for PBCs. The objective?s second performance goal is
to continue integrating small, small disadvantaged, and women- owned
businesses together with minority universities into the competitive base
from which NASA can purchase goods and services. NASA further indicated in
discussion under the strategic objective that (1) NASA?s Office of
Procurement has undertaken proactive management approaches in three key
areas: human capital, outsourcing and oversight, and electronic commerce;
and (2) on November 2000, the Associate Administrator for Procurement and
the Associate Administrator for Safety and Mission Assurance jointly
announced the establishment of the Surveillance Planning Team to provide
policy direction and procedural guidance on appropriate surveillance
planning for NASA- contracted work based on the risk associated with the
work and contractor involvement. The plan also contains a performance goal
to continue implementation of planned and new privatization efforts through
the space shuttle prime contract and further efforts to safely and
effectively transfer civil service positions and responsibilities to the
space shuttle contractor with associated performance goal indicators. The
plan also contains a performance indicator to initiate the pilot phase of
the core financial module and at least one other module of the integrated
financial management system. If effectively implemented, this system should
provide systems and processes to oversee procurement activities. However,
this effort will require continued management attention to correct problems
and keep projects on schedule. Furthermore, the plan does not address the
issue of limiting NASA?s frequent use of undefinitized contract change
orders - that is, unnegotiated contract changes. Relying on unnegotiated
changes as a way of doing business is risky because it increases the
potential for additional unanticipated cost growth.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 26 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Fiscal Management: NASA?s OIG has reported that fiscal management continues
to be a significant challenge for NASA. Key considerations to improved
fiscal management include: (1) monitoring contractor performance of
financial statement audits to ensure that the statements are properly
prepared and thoroughly reviewed; (2) ensuring adequate integration and
testing of newly developed automated accounting modules or capability; and
(3) ensuring that NASA continues to properly account for and record
financial transactions as new capability is implemented.
In March 2001, we reported that NASA?s fiscal year 1999 Statement of
Budgetary Resources was misstated by a reported $644 million due, in part,
to a misinterpretation of guidance and errors in NASA?s ad hoc process for
generating budgetary information. i In August 2001, we reported on NASA?s
inability to provide timely data on obligations or support for actual costs
related to the space station and shuttle. j
The plan contains a strategic objective, annual performance goals, and
indicators directly related to this management challenge. The plan?s
strategic objective is to manage NASA?s fiscal and physical resources
optimally. This objective has an annual performance goal to improve NASA?s
financial management and accountability. The plan states two target
indicators will measure the agency?s progress in meeting this performance
goal: (1) cost at least 75 percent of the budget available to cost during
the fiscal year and (2) initiate the pilot phase of the core financial
module and at least one other module of the integrated financial management
system (IFMS). While the goal of improving financial management and
accountability is commendable, additional target indicators should be
incorporated to measure NASA?s progress in achieving its goal. Additional
performance indicators are needed to ensure that NASA?s systems and
processes provide a direct linkage between financial and program operations.
For example, both the efficiency and effectiveness measures should be
designed to assess the finance organization?s ability to support NASA?s
mission. Without performance indicators that provide a clear linkage between
financial management improvement and improved program results, NASA may not
be able to accurately track its progress in achieving its performance goals.
Furthermore, the implementation of the IFMS is key to producing accurate and
reliable information for full- cost accounting. Until the system is
operational, performance assessments relying on cost data may be incomplete
and full costing will be only partially implemented. Measuring costs is key
to measuring performance in terms of efficiency and cost- effectiveness.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 27 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Program and Project Management: NASA?s OIG has reported that NASA faces
significant challenges in program and project management. Key considerations
to effectively managing NASA programs include (1) improving planning to
enable NASA to accomplish its missions in the face of budget and human
capital issues; (2) eliminating duplication in programs and improving
coordination with other research and development organizations; (3) ensuring
that programs and projects accurately assess their progress and successfully
achieve their goals; and (4) effectively using technology developments to
increase NASA productivity.
Although we did not identify this issue as a management challenge, we have
reported in September 2001 that there are fundamental weaknesses in the
collection and sharing of lessons learned in NASA by program and project
managers. k In June 2001, we also reported that the initial ISS propulsion
module project did not meet performance, cost, and schedule goals largely
because NASA proceeded with the contractor?s proposal without following
fundamental processes involving project planning and execution. l
The plan includes a strategic goal and objectives, annual performance goals,
and indicators directly related to this management challenge. The plan has a
strategic goal to enable NASA?s strategic enterprises and their centers to
deliver products and services to customers more effectively and efficiently.
This goal has four related objectives: (1) enhance program safety and
mission success in the delivery of products and operational services; (2)
improve NASA?s engineering capability to remain as a premier engineering
research and development organization; (3) capture engineering and
technological best practices and process knowledge to continuously improve
NASA?s program/ project management; and (4) facilitate technology insertion
and transfer and utilize commercial partnerships in research and development
to the maximum extent practicable. The plan contains a performance goal to
capture a set of best practices/ lessons learned from each program, to
include at least one from each of the four ?provide aerospace products and
capabilities crosscutting process? subprocesses, commensurate with current
program status. The plan contains a performance goal to earn external review
rating of ?achieved performance target? on making progress in the following
area: design, develop, and launch projects to support future research in
pursuit of Strategic Plan science objectives. This goal has an indicator to
meet no fewer than 75 percent of the development performance objectives for
?major programs/ projects,? supported by completion of performance
objectives in majority of ?other projects.? The plan has a performance goal
to identify and evaluate candidate approaches for 100- to 1000- day human
missions capable of a 5- to 10- fold cost reduction- while increasing safety
and effectiveness (compared to 1990s projections). The plan contains a
strategic objective to meet sustained space operations needs while reducing
costs. This objective has two performance goals: (1) the Space
Communications program will conduct tasks that enable commercialization and
will minimize investment in government infrastructure for which commercial
alternatives are being developed; and (2) performance metrics for each
mission will be consistent with detailed program and project operations
requirements in project service level agreements.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 28 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Launch Vehicles: NASA?s OIG reported on challenges in (1) ensuring the
availability of small expendable launch vehicles to ensure schedule
milestones and cost- effectiveness of NASA missions; (2) evaluating whether
NASA is providing the majority of developmental funds and assigning
technology rights to its industry partners in the development of the new
reusable launch vehicles in the best interest of the government; and (3)
ensuring that plans are in place and are effectively implemented to address
shuttle systems obsolescence, logistics support, technical/ safety upgrades,
and funding.
Although we did not identify this issue as a management challenge, we have
reported and testified on the factors that contributed to the difficulties
experienced by the X- 33 and X- 34 programs and the steps needed to avoid
repeating those problems within the Second Generation Reusable Launch
Vehicle Program. m
The plan includes a strategic goal and objective, annual performance goal
and indicators directly related to this management challenge. The Aerospace
Technology Enterprise has a strategic goal to provide commercial industry
with the opportunity to meet NASA?s future launch needs, including human
access to space, with new launch vehicles that promise to dramatically
reduce cost and improve safety and reliability. The strategic objective is
to utilize NASA?s Space Transportation Council in combination with an
External Independent Review Team (EIRT) to assure agency- level integration
of near- and farterm space transportation investments. The performance goal
is to review results of NASA and commercial- sector performed launch system
architecture studies, related requirements, and refinements in planned risk-
reduction investments. There are two performance goal indicators (1)
complete an assessment of the space launch initiative architectures and
requirements by EIRT; the EIRT will submit a written report on their
evaluation within 45 days following completion of the review; and (2) the
Space Transportation Council will review progress and planning of the space
launch initiative at least twice during the fiscal year, including the
report filed by the EIRT. However, NASA?s plan does not contain enough
specific information about how NASA will ensure that the government?s best
interests will be served in these joint government- and industry- funded
programs.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 29 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Technology Development: NASA?s OIG has reported that technology development
has become a serious challenge for NASA. Key considerations to effective
technology development include (1) achieving a balance between scientific
research and technology development and demonstration projects; (2)
continuing to refine the technology transfer process to ensure that U. S.
industry achieves the maximum benefit from the new technologies identified;
(3) determining if NASA?s organizational structure effectively supports
technology development and transfer; (4) forming innovative partnership
arrangements with U. S. industry to share both the risk and costs of
technology demonstration and commercialization; (5) ensuring that NASA
technology demonstrations do not unfairly distort the marketplace; (6)
ensuring that adequate controls exist on cooperative technology development
programs; and (7) ensuring adequate protection of NASA- developed
technology.
The plan has strategic goals, objectives and annual performance goals
directly related to this management challenge. The space science enterprise
contains a strategic goal: ?Technology/ LongTerm Future Investments: develop
new technologies to enable innovative and less expensive research and flight
missions.? The strategic objectives for this goal are to: (1) acquire new
technical approaches and capabilities; (2) validate new technologies in
space; and (3) apply and transfer technology. The Earth Science Enterprise
has a strategic goal to develop and adopt advanced technologies to enable
mission success and serve national priorities. This goal has related
objectives, performance goals, and indicators. The plan has a strategic
objective to invest in the development of high- leverage technologies to
enable safe, effective and affordable human/ robotic exploration. This
objective has a performance goal to begin development of high- leverage
technologies to enable safe, effective and affordable human/ robotic
exploration missions beyond low earth orbit (LEO). The plan also includes
performance goals to: (1) test at the International Space Station competing
technologies for human missions beyond LEO, in cooperation with other
agencies and international partners, and with US industry; (2) select and
fund at least 3- 5 proposals through the Human Exploration and Development
of Space Enterprise?s (HEDS) Technology and Commercialization Initiative-
focused research and technology program that feature: highly innovative new
technology development efforts in selected areas associated with human
safety and performance in space; and (3) conduct a competitive solicitation
and selection process that will fund through the HEDS research and
technology program a HEDS Technology and Commercialization Initiative: (a)
systems studies assessing the commercial potential associated with various
prospective HEDS infrastructures/ capabilities; and (b) new technology
development and demonstration efforts with potential longer- term commercial
space value. The Biological and Physical Research Enterprise has a
performance goal to engage the commercial community and encourage non- NASA
investment in commercial space research by meeting at least three of four
performance indicators. The plan also has a strategic objective to transfer
NASA technologies and innovations to private industry and the public sector
so that the public can benefit economically as well as intellectually
through clear, effective communications concerning NASA?s activities.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 30 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
International Agreements: NASA?s OIG reported that international agreements
are needed to ensure effective and efficient programs. Key considerations
include (1) program and project vulnerability to schedule delays and cost
overruns that require diplomatic rather than contractual solutions; (2)
security controls on technology that impacts national security; (3) controls
to assure the quality and timeliness of the goods and services provided; (4)
mechanisms to assure a balance between program needs and national
considerations; (5) plans with specific critical paths and planned
alternative courses of action to maintain program/ project continuity; and
(6) proper controls over access to NASA facilities by foreign national
visitors.
Although we did not identify this issue as a major management challenge, in
November 1999, we recommended measures to enhance NASA?s ability to oversee
and implement its export controls of ISS- related technologies. n
The plan has a strategic objective and annual performance goals directly
related to this management challenge. The plan has performance goals to (1)
collaborate with other federal and international agencies in developing and
implementing better methods for using remotely sensed observations; (2) test
at the International Space Station competing technologies for human missions
beyond LEO in cooperation with other agencies and international partners and
with US industry; (3) develop and test -
on the ground and in space- competing technologies for human missions beyond
LEO in cooperation with international partners; (4) conduct a competitive
solicitation and selection process that will fund through the HEDS research
and technology program the HEDS Technology and Commercialization Initiative.
The fourth performance goal includes a performance indicator to develop, in
conjunction with discussions with key international space organizations, and
seeks management approval for an approach for undertaking the formulation of
international partnerships for the development and/ or demonstration of HEDS
capabilities. The plan also states that in response to recommendations from
the NASA OIG, NASA is clarifying the definition of ?foreign national? in its
foreign visitors policy to ensure appropriate and consistent use of the term
in the agency?s foreign visitors review program. However, NASA?s indicators
for this challenge lack sufficient specificity. For example, the plan does
not specify actions taken to ensure security controls on technology that
impacts national security.
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 31 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Major management challenge Applicable goals and measures in the fiscal year
2002 performance plan
Environmental Management: NASA?s OIG reported that environmental management
is a significant management challenge. Key considerations include: (1)
prioritizing and addressing environmental obligations; (2) developing
consistent procedures under an agencywide policy; and (3) negotiating
costsharing agreements for environmental cleanup with previous government
and private sector tenants that are also responsible parties.
The plan includes annual performance goals directly related to this
management challenge. The plan also states that NASA?s strategy for
achieving its environmental vision includes four focus areas: prevention,
compliance, restoration, and conservation. However, NASA?s plan only has
performance metrics for the areas of compliance and restoration. Additional
performance indicators for the environmental focus areas of prevention and
conservation are needed so that NASA can accurately and fully track its
progress toward achieving its environmental vision. The compliance related
performance goal is that NASA will increase the safety of its infrastructure
and the health of its workforce through facilities safety improvements,
reduced environmental hazards, increased physical security, enhanced safety
and health awareness, and appropriate tools and procedures for health
enhancement. This performance goal includes an indicator to reduce the level
of agency environmental noncompliance incidents and releases in order to
achieve a 5 percent reduction from the fiscal year 2000 level by 2005. The
environmental restoration- related performance goal is to revitalize agency
facilities and reduce environmental liability. This performance goal
includes an indicator to reduce NASA?s unfunded environmental liability
through a long- term strategy, annually investing an amount of not less than
3- 5% of the agency?s environmental liability in environmental compliance
and restoration funding. The plan also states management controls need to be
strengthened to ensure greater visibility of and more consistent
implementation of the National Environmental Policy Act (NEPA) process. The
plan states review of existing management controls, development and advocacy
of improvements, and training activities have been planned and are being
initiated. Furthermore, the Aerospace Technology Enterprise also has a
strategic objective to reduce emissions - protect local air quality and our
global climate with a related performance goal and indicators.
a Space Shuttle: Human Capital and Safety Upgrade Challenges Require
Continued Attention (GAO/ NSIAD/ GGD- 00- 186, Aug. 15, 2000). b Space
Shuttle Safety: Update on NASA?s Progress in Revitalizing the Shuttle
Workforce and Making Safety Upgrades (GAO- 01- 1122T, Sept. 6, 2001). c
Major Management Challenges and Program Risks: National Aeronautics and
Space Administration (GAO- 01- 258, Jan. 2001). d Information Security: Many
NASA Mission- Critical Systems Face Serious
Risks (GAO/ AIMD- 99- 47, May 20, 1999). e (GAO- 01- 258, Jan. 2001).
f (GAO- 01- 1122T, Sept. 2001). g Space Station: Inadequate Planning and
Design Led to Propulsion Module
Project Failure (GAO- 01- 633, Jun. 20, 2001). h NASA: International Space
Station and Shuttle Support Cost Limits
(GAO- 01- 1000R, Aug. 31, 2001). i Financial Management: Misstatement of
NASA?s Statement of Budgetary
Resources (GAO- 01- 438, Mar. 30, 2001) j (GAO- 01- 1000R, Aug. 31, 2001).
k Survey of NASA?s Lessons Learned Process (GAO- 01- 1015R, Sept. 5, 2001)
Appendix I: Observations on NASA?s Efforts to Address Its Major Management
Challenges
Page 32 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
l (GAO- 01- 633, June 2001). m Space Transportation: Status of the X- 33
Reusable Launch Vehicle Program
(GAO/ NSIAD- 99- 176, Aug. 11, 1999); Space Transportation: Progress of the
X- 33 Reusable Launch Vehicle Program (GAO/ T- NSIAD- 99- 243, Sept. 29,
1999); and
Space Transportation: Critical Areas NASA Needs to Address in Managing Its
Reusable Launch Vehicle Program (GAO- 01- 826T, June 20, 2001). n Export
Controls: International Space Station Technology Transfers
(GAO/ NSIAD- 00- 14, Nov. 3, 1999).
Appendix II: Comments From the National Aeronautics and Space Administration
Page 33 GAO- 02- 184 Status of Plans for Achieving Key Outcomes
Appendix II: Comments From the National Aeronautics and Space Administration
Appendix II: Comments From the National Aeronautics and Space Administration
Page 34 GAO- 02- 184 Status of Plans for Achieving Key Outcomes (120091)
The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO?s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO?s Web site (www. gao. gov) contains abstracts and full- text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
site daily. The list contains links to the full- text document files. To
have GAO E- mail this list to you every afternoon, go to our home page and
complete the easy- to- use electronic order form found under ?To Order GAO
Products.?
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U. S. General Accounting Office P. O. Box 37050 Washington, D. C. 20013
To order by phone: Voice: (202) 512- 6000 TDD: (301) 413- 0006 Fax: (202)
258- 4066
GAO Building Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)
Washington, D. C. 20013
Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail:
fraudnet@ gao. gov, or 1- 800- 424- 5454 (automated answering system).
Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S.
General Accounting Office, 441 G. Street NW, Room 7149, Washington, D. C.
20548 GAO?s Mission
Obtaining Copies of GAO Reports and Testimony
Order by Mail or Phone Visit GAO?s Document Distribution Center
To Report Fraud, Waste, and Abuse in Federal Programs
Public Affairs
*** End of document. ***