NASA: Better Mechanisms Needed for Sharing Lessons Learned
(30-JAN-02, GAO-02-195).
The National Aeronautics and Space Administration (NASA) uses
several mechanisms to capture and disseminate lessons learned.
NASA has established an agency-wide Lessons Learned Information
System that managers are required to review on an ongoing basis.
NASA uses training, program reviews, and periodic revisions to
agency policies and guidelines to communicate lessons learned.
Several NASA centers and key programs also maintain lessons
learned systems that are geared toward their own staff. To
improve the way it captures and shares information, NASA has
developed a strategic plan, assembled a management team to
coordinate knowledge management and activities at NASA's centers,
and begun several information technology pilot projects. Despite
the processes and procedures in place to capture and share
lessons learned, there is no assurance that lessons are being
applied toward future missions success. Although some lessons
learning does take place, lessons are not routinely identified,
collected, or shared by programs and project managers.
Furthermore, there are challenges or cultural barriers to the
sharing of lessons learned, such as the lack of time to capture
or submit lessons and a perception of intolerance for mistakes.
Although NASA's current efforts should lead to improved sharing
of agency lessons and knowledge, they lack components critical to
the success of knowledge management at leading organizations.
Successful industry and government organizations overcome
barriers by making a strong management commitment to knowledge
sharing, developing a well-defined business plan for implementing
knowledge management, providing incentives to encourage knowledge
sharing, and building technology systems to facilitate
information access. The application of these principles could
help NASA to better carry out its mission.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-195
ACCNO: A02703
TITLE: NASA: Better Mechanisms Needed for Sharing Lessons
Learned
DATE: 01/30/2002
SUBJECT: Agency missions
Best practices
Cost control
Data collection
Information resources management
Internal controls
Program evaluation
Space exploration
Strategic planning
Climate Orbiter Spacecraft
Mars Polar Lander Spacecraft
Mars Surveyor Program
NASA International Space Station Program
NASA Lessons Learned Information System
NASA Skylab Program
Mars Global Surveyor Spacecraft
Mars Pathfinder Spacecraft
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-195
United States General Accounting Office
GAO Report to the Subcommittee on Space and Aeronautics, Committee on
Science, House of Representatives
January 2002
NASA
Better Mechanisms Needed for Sharing Lessons Learned
GAO-02-195
Contents
Letter 1
Executive Summary 2
Purpose 2
Background 2
Results in Brief 2
Principal Findings 4
Recommendations 6
Agency Comments and Our Evaluation 7
Chapter 1
Introduction
Faster, Better, Cheaper: A Management Philosophy
Mars Surveyor Program
Mishap Investigations Raised Concern That NASA Did Not Heed
Lessons Learned
NASA Efforts to Apply Lessons Learned from Mars Mishaps
Lessons Learning: A Mechanism to Learn from Successes As Well
As Mistakes
Objectives, Scope, and Methodology
8
8 9
10 13
13 15
Chapter 2 NASA Policies, Processes, and Systems for Lessons
Learning and Knowledge Sharing 19
NASA's Policies and Procedures Require Lessons Learning 19
NASA Uses Numerous Methods for Lessons Learning 20
Additional Steps Taken by NASA to Facilitate Lessons Learning 23
Chapter 3 Lessons Learned Are Not Being Consistently Shared
or Applied Toward Future Program Success 27
Fundamental Weaknesses Exist in NASA's Processes, Procedures,
and Systems for Lesson Learning 27
Chapter 4 Creating an Environment for Lessons Learning
through Knowledge Management 36
Principles of Knowledge Management Could Help Mitigate
Fundamental Weaknesses in Lessons Learning 36
Chapter 5 Conclusions and Recommendations 43
Recommendations for Executive Action 44
Agency Comments and Our Evaluation 45
Appendix I Comments from the National Aeronautics and
Space Administration
Appendix II GAO Contacts and Staff Acknowledgments 50
GAO Contacts 50 Acknowledgments 50
Tables
Table 1: Reasons for Spacecraft Failures 12
Table 2: NASA's Framework for Knowledge Management 26
Table 3: Program and Project Manager Observations and
Suggestions to Improve LLIS 34
Figures
Figure 1: Generic Lessons Learned Process 14 Figure 2: Program and Project
Managers Knowledge of Lessons Learned 28 Figure 3: Many Program and Project
Managers Have Not Submitted Lessons to the Lessons Learned Information
System 30 Figure 4: Most Program and Project Managers Could Not Identify
Helpful Lessons for Their Program or Project 31
United States General Accounting Office Washington, DC 20548
January 30, 2002
The Honorable Dana Rohrabacher
Chairman, Subcommittee on Space and Aeronautics Committee on Science House
of Representatives
The Honorable Bart Gordon
Ranking Minority Member, Subcommittee on Space and Aeronautics Committee on
Science House of Representatives
In response to your request, this report discusses the mechanisms that the
National Aeronautics and Space Administration (NASA) has in place to
capture, disseminate, and apply past lessons learned towards future
mission success. We found that NASA's processes, procedures, and
systems do not effectively capture and share lessons learned and
therefore, NASA has no assurance that lessons are being applied towards
future missions. We include recommendations to the NASA administrator
on ways to strengthen the agency's lessons learning processes and
systems.
As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 3 days from
its
issue date. At that time we will send copies of this report to the NASA
administrator and interested congressional committees. We will also
provide copies to others on request. If you or your staffs have any
questions regarding this report, please contact me at (202) 512-4841, or
John Oppenheim, assistant director, at (202) 512-3111. Key contributors
are listed in appendix II.
Allen Li
Director
Acquisition and Sourcing Management
Executive Summary
Purpose
Background
Results in Brief
In the early 1990s, the National Aeronautics and Space Administration (NASA)
administrator challenged the agency to complete projects faster, better, and
cheaper. The intent was to reduce costs, become more efficient, and increase
scientific results by conducting more and smaller missions in less time.
Although NASA maintained a high success rate under the faster, better, and
cheaper strategy, a few significant mission failures also
occurred-particularly the loss of the Mars Polar Lander and Climate Orbiter
spacecraft. NASA investigations of these failures, as well as its review of
other programs, raised concern that lessons from past experiences were not
being applied to current programs and projects.
At the request of the Chairman and Ranking Minority Member, Subcommittee on
Space and Aeronautics, House Committee on Science, GAO assessed whether NASA
has adequate mechanisms in place to ensure that past lessons learned from
mission failures are being applied. Specifically, GAO (1) identified the
policies, procedures, and systems NASA has in place for lessons learning,
(2) assessed how effectively these policies, procedures, and systems
facilitate lessons learning, and (3) determined whether further efforts are
needed to improve lessons learning.
NASA's procedures and guidelines require that program and project managers
review and apply lessons learned from the past throughout a program's or
project's life cycle and to document and submit any significant lessons to
the agency's Lessons Learned Information System (LLIS) in a timely manner.
NASA defines a lesson learned as knowledge or understanding gained by
experience. The experience may be positive, such as a successful test or
mission, or negative, such as a mishap or failure. A lesson must be
significant in that it has a real or assumed impact on operations; valid in
that it is factually correct; and applicable in that it identifies a
specific design, process, or decision that reduces or eliminates the
potential for failures and mishaps, or reinforces a positive result.
NASA recognizes the importance of learning from the past to ensure future
mission success and uses several mechanisms to capture and disseminate
lessons learned. The principal source NASA has established for the
agency-wide collection and sharing of lessons is the LLIS, a Web-based
lessons database that managers are required to review on an ongoing basis.
In addition, NASA uses training, program reviews, and periodic revisions to
agency policies and guidelines to communicate lessons. Several NASA centers
and key programs also maintain lessons learned
Executive Summary
systems that are geared toward their own staff. Recently, NASA has taken
steps to improve the way it captures and shares information by developing a
business strategy called knowledge management. Knowledge management can be
defined as the way that organizations create, capture, and reuse knowledge
to achieve their objectives. According to NASA officials, knowledge
management has the potential to link agency staff with the knowledge and
resources they need to complete tasks faster, better, and cheaper. In
pursuit of knowledge management, NASA has developed a strategic plan,
established a management team to coordinate knowledge management activities
at NASA's centers, and initiated several information technology pilot
projects.
Despite the processes and procedures in place to capture and share lessons
learned, there is no assurance that lessons are being applied toward future
missions success. A GAO survey of NASA program and project managers revealed
weaknesses in the collection and sharing of lessons learned agency-wide.
While some lessons learning does take place, our survey found that lessons
are not routinely identified, collected, or shared by programs and project
managers. Respondents reported that they are unfamiliar with lessons
generated by other centers and programs. In addition, many respondents
indicated that they are dissatisfied with NASA's lessons learned processes
and systems. Managers also identified challenges or cultural barriers to the
sharing of lessons learned, such as the lack of time to capture or submit
lessons and a perception of intolerance for mistakes. They further offered
suggestions for areas of improvement, including enhancements to LLIS and
implementing mentoring and "storytelling," or after-action reviews, as
additional mechanisms for lessons learning.
While NASA's current knowledge management efforts should lead to some
improvement in the sharing of agency lessons and knowledge, they lack
ingredients that have been shown to be critical to the success of knowledge
management at leading organizations. Cultural resistance to sharing
knowledge and the lack of strong support from agency leaders often make it
difficult to implement an effective lessons learning and knowledge sharing
environment. We found that successful industry and government organizations
have overcome barriers by making a strong management commitment to knowledge
sharing, developing a well-defined business plan for implementing knowledge
management, providing incentives to encourage knowledge sharing, and
building technology systems to facilitate easier access to information. The
application of these principles could increase opportunities for NASA to
Executive Summary
perform its basic mission of exploring space faster, better, and cheaper
more successfully.
Principal Findings
NASA's Policies and Procedures for Lessons Learning
NASA uses various mechanisms to communicate lessons garnered from past
programs and projects. Policies and guidelines, programmatic and technical
reviews, mentoring and training programs, the Academy of Program and Project
Leadership, and LLIS are the mechanisms employed by NASA for capturing and
sharing lessons learned. LLIS is the "official" agency-wide repository for
such lessons. Lessons entered in the LLIS database are screened for
relevance and to ensure that they do not contain sensitive or proprietary
information. Initial reviews of lessons are usually conducted by the
centers,1 with a final review by the Office of Safety and Mission Assurance.
After a lesson is entered into the system, it remains in the database
indefinitely and is not reviewed for currency or relevance. Currently, the
system contains over 900 lessons on topics ranging from program management
to technical cause of failure.
In response to the Mars Program failures and the recommendations of agency
reviews of program and project execution, NASA has recently taken action to
improve its policies and practices for capturing and sharing knowledge by
developing a business strategy referred to as knowledge management.
Implementation of knowledge management can lead to increased productivity,
collaboration, and innovation in the workplace. To coordinate and guide its
efforts, NASA recently formed a knowledge management team, which developed a
strategic plan that laid out broad goals and objectives for knowledge
management. In addition, several pilot projects are underway at various NASA
centers to enhance knowledge sharing.
1 NASA consists of NASA headquarters, nine centers, the Jet Propulsion
Laboratory (operated under contract to NASA by the California Institute of
Technology), and several ancillary installations and offices in the United
States and abroad. The implementation of NASA programs and aeronautical and
space/earth science research occurs primarily at the centers.
Executive Summary
Fundamental Weaknesses Exist in the Collection and Sharing of Lessons
Learned
A survey we conducted of all NASA program and project managers revealed
fundamental weaknesses in the collection and sharing of lessons learned
agency-wide. Although NASA's processes and procedures require that program
and project managers review and apply lessons learned throughout a program's
or project's life cycle, our survey found that managers do not routinely
identify, collect, or share lessons. Respondents indicated that LLIS, NASA's
primary method for disseminating lessons learned agency-wide, is not the
primary source for lessons learning. Instead, managers identified program
reviews and informal discussions with colleagues as their principal sources
for lessons learned. One reason LLIS is not widely used, according to one
center official, is because its lessons cover so many topics that it is
difficult to search for an applicable lesson. Another respondent indicated
that it is difficult to weed through all the irrelevant lessons to get to
the few "jewels" that you need to find.
Respondents also identified challenges or cultural barriers to the sharing
of lessons learned as well as areas of improvement. Managers noted that
there is a reluctance to share negative lessons for fear that they might not
be viewed as good project managers, and there is a lack of time for lessons
learning to take place. One manager stated, "Until we can adopt a culture
that admits frankly to what really worked and didn't work, I find many of
these tools to be suspect." Managers suggested that NASA could improve
lessons learning by implementing mentoring and "storytelling" activities,
and it could enhance LLIS by increasing its search functions, including more
positive lessons, and developing a mechanism to disseminate key lessons to
users.
In discussions with NASA officials, we found there was general agreement
with the results of our survey as well as suggested improvements for lessons
learning. Officials indicated that lessons learning has taken on greater
importance in recent years due to the implementation of more programs and
projects under the faster, better, cheaper strategy and the continuing loss
of agency expertise due to attrition. They acknowledge that LLIS has not
been an effective mechanism for agency-wide sharing of lessons. Although the
system is viewed as providing a useful repository for storing lessons,
officials agreed with managers' concerns about the difficulties involved in
searching the system and finding relevant lessons, the inconsistent quality
of information contained in the system, and the lack of lessons about
positive project experiences. However, while program and project managers'
suggested improvements would help increase the usability of LLIS, they have
not targeted some of the more fundamental problems hampering NASA's ability
to share lessons, such as persistent cultural barriers.
Executive Summary
Creating an Environment for Lessons Learning through Knowledge Management
Recommendations
Leading organizations are discovering that actively managing knowledge
creates value by increasing productivity and fostering innovation. Likewise,
NASA's paramount concern should be about capturing and sharing
organizational knowledge and using it to perform its basic mission of
exploring space faster, better, and cheaper. Although NASA has recently
taken action to improve the way in which the agency captures, organizes, and
shares knowledge, these efforts do not fully address the fundamental
weaknesses in lessons learning identified by our survey: namely, cultural
resistance to sharing knowledge and the lack of an effective strategic
framework and management attention for overcoming such resistance.
NASA has made a reasonable start by developing a strategic plan for
knowledge management, but the agency has not made a good business case for
how it will implement and use knowledge management within the organization.
In addition, while successful industry and government organizations have
made a firm commitment to making knowledge management practices work, NASA
has not provided the leadership, support, and resources needed for effective
knowledge management to take place. Furthermore, knowledge management
organizations have employed incentives, processes, and systems designed to
address cultural barriers to continuous lessons learning and knowledge
sharing. For example, organizations that value knowledge sharing have
encouraged employees to spend time sharing knowledge, helped facilitate
communities of practice based around common interests, and provided rewards
when knowledge has been shared and applied. NASA has not done so on an
agency-wide basis.
NASA needs to strengthen its lessons learning in the context of its overall
efforts to develop and implement an effective knowledge management program.
Improvement of NASA's lessons learning processes and systems can help to
ensure that knowledge is gained from past experiences and applied to future
missions.
We recommend that the NASA administrator strengthen the agency's lessons
learning processes and systems by
* articulating the relationship between lessons learning and knowledge
management through an implementation plan for knowledge management;
* designating a lessons learned manager to lead and coordinate all agency
lessons learning efforts;
Executive Summary
* establishing functional and technical linkages among the various
center-level and program-level lessons learning systems;
* developing ways to broaden and implement mentoring and "storytelling" as
additional mechanisms for lessons learning;
* identifying incentives to encourage more collection and sharing of lessons
among employees and teams, such as links to performance evaluations and
awards;
* enhancing LLIS by coding information and developing an easier search
capability to allow users to identify relevant lessons, including more
positive lessons, providing a means to disseminate key lessons to users; and
soliciting user input on an ongoing basis; and
* tracking and reporting on the effectiveness of the agency's lessons
learning efforts using objective performance metrics.
In written comments on a draft of this report, NASA generally concurred with
our recommendations for improving the agency's lessons learned processes and
systems. NASA stated that it must do a better job of communicating the
various lessons learned sources to employees, improving mechanisms to link
these sources, and ensuring appropriate training for employees in order to
maximize lessons learning.
Agency Comments
and Our Evaluation
Chapter 1: Introduction
Faster, Better, Cheaper: A Management Philosophy
The 1990's was a decade of significant challenge for the National
Aeronautics and Space Administration (NASA) brought about, in part, by
decreased budgets, a reduced work force, and technological innovation. To
adapt to this changing environment, the NASA administrator challenged agency
personnel to do projects faster, better, and cheaper (FBC) by streamlining
practices and becoming more efficient. The goal was to shorten program
development times, reduce cost, and increase scientific return by flying
more and smaller missions in less time. However, the failure of the Mars
Polar Lander and Climate Orbiter spacecraft raised concerns that lessons
learned from past experiences were not being passed along and applied toward
future mission success. To ensure that individuals learn from past
experiences, effective processes and systems must be in place to collect,
store, and disseminate lessons learned.
In 1992 NASA adopted the FBC philosophy as a way of managing programs and
projects. An important element of this approach was a reduction in NASA
headquarters management and moving more program responsibility to NASA's
centers. This philosophy also increased the demand for program and project
managers at a time when NASA was experiencing a significant reduction in
staff due to retirements, downsizing, and departures to industry. Prior to
FBC, there were fewer missions, and program and project managers accumulated
significant first-hand experience before managing a program. Under FBC, with
a threefold increase in projects and fewer staff, this was not always the
case. Relatively unseasoned managers who were challenged to be more
efficient and innovative and to take greater risks in designing and
implementing missions led many projects.
NASA's record in designing, developing, and operating smaller spacecraft
under the FBC concept, while mostly successful, has experienced a few
notable failures. A number of projects including Clementine, Near Asteroid
Rendezvous, Mars Pathfinders, and Mars Global Surveyor proved successful
within the schedule and fiscal constraints imposed under the FBC concept.
However, for other programs, including the Mars Polar Lander and Climate
Orbiter, the challenge was too great, resulting in undue risk-taking and
failure. After-action reports commissioned by NASA found that the Mars
Program failures resulted from cost and schedule constraints and a lack of
rigorous attention to sound process and practices.
Chapter 1: Introduction
Mars Surveyor Program
In 1993 NASA initiated the Mars Surveyor Program with the objective of
conducting an on-going series of missions to explore Mars. NASA's Jet
Propulsion Laboratory (JPL) was selected as the lead center for this program
and was responsible for mission design, spacecraft and payload development,
as well as integration, testing, and launch of the proposed spacecraft.
The first Mars Surveyor Program spacecraft, the Mars Global Surveyor and the
Mars Pathfinder, were launched in 1996 and both were highly successful.
After completing its primary mission of acquiring knowledge of the climate,
subsurface resources, and topography of Mars, the Mars Global Surveyor is
conducting reconnaissance of future Mars landing sites. The Mars Pathfinder
and Sojourner Rover were a demonstration of a way to use technology to
deliver an instrumented lander and a free-ranging robotic rover to the
surface of Mars. Two additional Mars Surveyor Program spacecraft were
launched in late 1998 and early 1999, respectively, to explore and collect
additional scientific data on Mars.
The Mars Climate Orbiter, which cost $75 million to develop, was launched
December 11, 1998, and was intended to observe Mars' seasonal climate and
daily weather from a low orbit around the planet. Nine and a half months
after launch, in September 1999, the spacecraft was supposed to fire its
main engine to achieve an elliptical orbit around Mars and then pass through
its upper atmosphere for several weeks before moving into a low circular
orbit around the planet. On September 23, 1999, the Orbiter spacecraft was
destroyed when it entered the Martian atmosphere on a lower than expected
trajectory, causing the spacecraft to burn up. A NASA mishap investigation
team determined that the loss was due to the mistaken use of English rather
than metric units in the navigation software. As a result, an incorrect
trajectory was computed and the spacecraft was lost.
The Mars Polar Lander was launched January 3, 1999, and cost $113 million to
develop. The Lander was a robotic spacecraft designed to land softly near
the South Pole of Mars to study the planet's layered polar terrain. Attached
to the Lander was the Deep Space 2 mission-two small probes designed to
separate from the Lander about 5 minutes prior to touchdown and penetrate at
high speed into the Martian soil to search for traces of vaporized water
ice. The Deep Space 2 probes cost $29.6 million to develop. Unfortunately,
the Lander was lost December 3, 1999, while attempting to land. The two
small probes also were lost. NASA's mishap investigation team concluded that
the premature shutdown of the Lander's
Chapter 1: Introduction
Mishap Investigations Raised Concern That NASA Did Not Heed Lessons Learned
descent engines was the most probable cause for the loss of the spacecraft.
The failure of the Mars Polar Lander and Climate Orbiter spacecraft raised
concern that lessons learned from past mishaps and programs were not being
applied effectively toward future mission success. Support for this comes
from reports commissioned by NASA that reviewed spacecraft failures in the
Mars Program, Shuttle wiring problems, and an assessment of the agency's
approach in executing "Faster, Better, Cheaper" projects.1 These reports
identified the root and contributing causes for failure as well as making
broader recommendations on ways NASA might improve its general approach to
executing programs and projects.
NASA's decision to leave critical information-gathering mechanisms off the
Mars Polar Lander is a case in point. The importance of downlink telemetry2
during critical mission events was a lesson learned from the failure of
NASA's 1992 Mars Observer mission that was not heeded during design of the
Mars Polar Lander seven years later. The design of Lander precluded
transmission of critical communications or telemetry data during entry,
descent, and landing on Mars. A Special Review Board determined that the
probable cause of the loss of the spacecraft was due to a premature shutdown
of the descent engines due to spurious signals generated when the Lander's
legs deployed. The spurious signals gave a false indication that the
spacecraft had landed. The board also determined that the spurious signals
resulted from inadequate software design and systems testing.
Concern that NASA had not heeded past lessons learned was also confirmed in
a recent study3 conducted by RAND's National Security
1 Mars Program Independent Assessment, chaired by Mr. A. Thomas Young
(retired), Lockheed Martin; Mars Climate Orbiter (MCO) Mishap Investigation,
chaired by Mr. Arthur Stephenson, director, Marshall Space Flight Center;
NASA Faster, Better, Cheaper Task Force, chaired by Mr. Anthony Spear
(retired), Jet Propulsion Laboratory; and Shuttle Independent Assessment,
chaired by Dr. Henry McDonald, director, Ames Research Center.
2 Telemetry is information on the condition of the spacecraft and its
subsystems, such as the temperature and voltage of spacecraft batteries,
that is transmitted to spacecraft operators on the ground.
3 Sarsfield, Liam. The Application of Best Practices to Unmanned Spacecraft
Development: An Exploration of Success and Failure in Recent Missions.
(Santa Monica, CA: RAND, 2000).
Chapter 1: Introduction
Research Division. The purpose of the study was to provide guidance on
practices that reduce risk and improve the performance of next-generation
spacecraft by including an examination of NASA's successes and failures in
building spacecraft both before and after implementing the faster, better,
cheaper approach. The RAND study identified the top ten sources of failures
in NASA programs and reported that a significant source of error has been
the failure of NASA to incorporate lessons previously learned and
consistently apply them. As shown in the following table, the reasons for
failure of the Mars Polar Lander and Climate Orbiter are not new; they have
been identified as areas of concern by major program reviews and have
occurred during the development of several other space systems.
Chapter 1: Introduction
Table 1: Reasons for Spacecraft Failures
aWIRE: Wide-Field Infrared Explorer bSOHO: Solar and Heliospheric
Observatory cDC-X: Delta Clipper-Experimental
Source: RAND, used with permission.
Chapter 1: Introduction
NASA Efforts to Apply Lessons Learned from Mars Mishaps
In December 2000, NASA issued a report, Enhancing Mission Success, which
identified specific actions the agency planned to take to apply prior mishap
findings and recommendations broadly in order to improve its approach to
executing programs and projects.4 The development of this report was driven
principally by the multiple mission failures associated with the Mars
Program. As part of its assessment, NASA reviewed a total of 165
recommendations originating from the Mars mishaps reports, the Shuttle
Independent Assessment, and the agency-wide assessment of faster, better,
cheaper projects. As a result, NASA recommended 17 actions that are
integrated into five broad themes: people, technology, risk management,
program and project management, and communications.
One of the 17 recommended actions, which deals with improving
communications, concerns knowledge management5 and the recognition that NASA
needs to do better in capturing, disseminating, and utilizing knowledge.
This includes improving the capture and application of lessons learned from
programs, projects, and missions, with the goal of ensuring that NASA does
not have to keep "relearning" the lessons of the past-�relearning evidenced
by the reoccurrence of similar causes to mission failures or difficulties.
The report also indicated a lack of access to and process for lessons
learned. The recommended action further stated that the continuous capture
and application of project knowledge and lessons learned must become a core
business process within the agency's program and project management
environment.
Lessons Learning: A Use of lessons learned is a principal component of an
organizational
culture committed to continuous improvement. Lessons learnedMechanism to
Learn mechanisms serve to communicate acquired knowledge more effectively
from Successes As and to ensure that beneficial information is factored into
planning, work
processes, and activities. Lessons learned provide a powerful method ofWell
As Mistakes sharing good ideas for improving work processes, facility or
equipment design and operation, quality, safety, and cost-effectiveness.
4 National Aeronautics and Space Administration. Enhancing Mission Success -
A Framework for the Future. A Report by the NASA Chief Engineer and the NASA
Integrated Action Team, December 21, 2000.
5 Knowledge management can be defined as the way that organizations create,
capture, and re-use knowledge to achieve organizational objectives.
Chapter 1: Introduction
Elements of the Lessons The mechanisms or processes used to collect, share,
and disseminate
Learned Process lessons learned may vary, but in general such a process is
comprised of four main elements: collection, verification, storage, and
dissemination. Figure 1 is a generic representation of the lessons learned
process.
Figure 1: Generic Lessons Learned Process
Source: Based on Weber, R., Aha, D., and Becerra-Fernanadez, I. Categorizing
Intelligent Lessons Learned Systems. Intelligent Lessons Learned Systems:
Papers from the AAAI Workshop (Technical Report AIC-00-005). Aha, D.W. and
Weber, R. (Eds.) pp. 63-67. Washington, DC: Naval Research Laboratory, Navy
Center for Applied Research in Artificial Intelligence, 2000.
* The collection process involves the capture of information through
structured and unstructured processes such as mishap or accident reporting,
project critiques, written forms, and meetings. The collection of lessons
may come from as many sources as an organization is willing to solicit.
Lessons learned can be based upon positive experiences that prevent
accidents or save money or on negative experiences that result in
undesirable outcomes. However, if an organization focuses only on failures,
its overall program's effectiveness will be reduced and it will miss
opportunities to improve all its processes.
* The verification process serves to verify the correctness and
applicability of lessons submitted. Domain or subject matter experts may be
involved in coordinating and conducting reviews to determine
Chapter 1: Introduction
whether or not a lesson is relevant across many other projects, is unique to
a particular department or project, or applies globally to the organization
as a whole.
* The storage aspect of lessons learned usually involves incorporating
lessons into an electronic database for the dissemination and sharing of
information. Information should be stored in a manner that allows users to
identify applicable information searches. In addition, each program should
include a keyword and functional category search capability to facilitate
information retrieval.
* The final element, and the most important, is the dissemination of lessons
learned, since lessons are of little benefit unless they are distributed and
used by people who will benefit from them. Dissemination can include the
revision of a work process, training, and routine distribution via a variety
of communication media. Lessons can be "pushed," or automatically delivered
to a user, or "pulled" in situations where a user must manually search for
them. Lessons can also be disseminated with an assigned priority descriptor,
which denotes the risk, immediacy, and urgency of the lessons learned
content.
The Chairman and the Ranking Minority Member, Subcommittee on Space and
Aeronautics, House Committee on Science, requested that we examine whether
NASA has sufficient mechanisms in place to ensure that past lessons learned
are applied effectively towards future mission success. Specifically, we (1)
identified what policies, procedures, and systems NASA has in place for
lessons learning, (2) assessed how effectively these policies, procedures,
and systems facilitate lessons learning, and (3) determined whether further
efforts are needed to improve lessons learning.
We reviewed NASA policies and procedures and interviewed agency staff to
determine how they captured, processed, and used lessons learned in their
programs and projects. We also obtained briefings and documents on the
different mechanisms used by NASA to capture and disseminate lessons
learned. In addition, we conducted site visits at NASA headquarters,
Washington, D.C.; Goddard Space Flight Center, Greenbelt, Maryland; Kennedy
Space Center, Florida; Langley Research Center,
Objectives, Scope,
and Methodology
Chapter 1: Introduction
Hampton, Virginia; and the Jet Propulsion Laboratory, Pasadena,
6
California.
To assess how effectively NASA's policies and procedures facilitate lessons
learning, we conducted a survey of NASA program and project managers. We
issued an interim report in September 2001 on the results of our survey.7
NASA officials generally agreed with the survey results and indicated that
the agency must do a better job of communicating various lessons learning
resources to employees, improving linkages among the sources, and
encouraging training to increase lessons learning. We build on the findings
and results of our survey in this report. The methodology we employed in
conducting this survey included the following steps:
* To obtain the views of NASA program and project managers on lessons
learned processes and systems at NASA, we conducted a survey in June and
July 2001 using a self-administered electronic questionnaire posted on the
World Wide Web. The survey contained four groups of questions on (1) how
NASA personnel collect, access, and use lessons learned; (2) the strengths
and weaknesses of current NASA lessons learned processes, procedures and
systems, including the Lessons Learned Information System; (3) potential
challenges or barriers to sharing lessons learned within NASA; and (4) ways
to improve the lessons learned processes and systems at NASA.
* In designing the questionnaire, we interviewed NASA officials and program
and project managers, as well as other government and industry officials who
had insight into lessons learned processes and systems. We also obtained and
reviewed NASA documents and guidance pertinent to lessons learned. To
further guide the development of appropriate questions, we reviewed current
literature on lessons learned and knowledge management. To validate the
content and structure of the questionnaire, we submitted it to officials at
NASA headquarters and others for review and comment. To verify the clarity,
length of time of administration, and suitability of the questions, we also
pre-tested the questionnaire with selected program and project managers at
Goddard Space Flight Center.
6 The Jet Propulsion Laboratory is a federally funded research and
development center. In our report, we treat the Jet Propulsion Laboratory as
a NASA center in our discussions.
7 U.S. General Accounting Office, Survey of NASA's Lessons Learned Process,
GAO-01-1015R (Washington, D.C.: 2001).
Chapter 1: Introduction
* NASA officials provided us with a list of the e-mail addresses of all NASA
program and project managers as of April 2001. On June 5, 2001, we sent
e-mail messages to each of the 199 NASA managers in this survey population,
notifying them of the survey and asking them to complete the questionnaire.
Subsequently, we discovered one additional NASA manager who was eligible for
our survey, and removed seven managers who were ineligible for the survey
because they were not project or program managers at the time of the survey.
We also eliminated one duplicate listing, resulting in a final survey
population of 192.
* Over the following several weeks, until closing the survey on July 13,
2001, we received a total of 115 useable responses, for an overall response
rate of 60 percent. All of the responses had been submitted using the Web
questionnaire, although one participant had asked to submit a paper version
of the survey, due to problems in accessing the Web survey. Of the 77
non-respondents, 9 provided partial questionnaire responses but had not
indicated that they were finished with the questionnaire. Partial responses
were not included in our survey results.
While we believe that our survey results are generalizable to the population
of NASA program and project managers as described above, the practical
difficulties of conducting any survey may introduce errors into estimates
made from surveys. Although we administered questionnaires to all known
members of the population, and thus our results are not subject to sampling
error, non-response to the entire survey or individual questions can
introduce a similar type of variability or bias into our results to the
extent that those not responding differ from those who do respond in how
they would have answered our survey questions. In addition, population
coverage errors can occur if some members of the population are excluded
from the survey. Measurement errors can arise from how questions are
interpreted by respondents and mistakes made by respondents. Data processing
errors can arise during the handling or analysis of responses. We took steps
in the design, data collection, and analysis phases of our survey to
minimize such errors, such as pre-testing questionnaires before the survey,
following up with those not reachable at original e-mail addresses or
otherwise not immediately responding, and checking for errors in computer
programming used to analyze survey results. In addition, the distribution of
respondents across NASA enterprise areas generally reflected the actual
distribution of the entire population, which was consistent with our belief
that non-response error was not significant.
Chapter 1: Introduction
To identify practices for improving NASA's lessons learning processes and
systems, we reviewed existing literature on lessons learning and knowledge
management practices. In addition, we identified a selected number of
commercial and government organizations that are practitioners of knowledge
management and obtained information related to their experience. In making
our selections, we reviewed literature and spoke with industry and academic
experts to find organizations recognized for their ability to share lessons
learned or effectively manage knowledge. We identified the following
organizations: World Bank, United States Department of Energy, Ford Motor
Company, TRW, Boeing Space and Communications, Northrop Grumman, and
Lockheed Martin.
We discussed with representatives from each organization the processes they
used for sharing information, emphasizing those practices that were critical
for successful lessons learning. During our discussions with the
organizations, we compared and contrasted their practices with those of
NASA. This allowed us to identify practices that NASA could use to improve
its processes and systems. For each organization, we interviewed key
managers and obtained documentation to determine (1) the processes and tools
used to enhance information sharing and (2) the extent to which these
processes and tools affected the organizations' ability to share
information.
Our work was performed from October 2000 through September 2001 in
accordance with generally accepted government auditing standards.
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
NASA uses various mechanisms to communicate lessons garnered from past
programs and projects. Policies and guidelines, programmatic and technical
reviews, mentoring and training programs, the Academy of Program and Project
Leadership, and the Lessons Learned Information System (LLIS) are all
mechanisms employed by NASA for capturing and sharing lessons learned. While
not the only means available for capturing and disseminating lessons
learned, LLIS is the "official" agency-wide repository for such lessons. In
addition to agency-wide mechanisms for collecting and sharing lessons
learned, several NASA centers also operated lessons learned systems
configured to support specific programs, such as the International Space
Station.
To improve the way the agency captures and shares information and lessons
learned, NASA has recently formed a knowledge management team. Knowledge
management is a business strategy used by many organizations that strive to
make more effective use of the experience and expertise of employees within
an organization. Organizations believe that by developing new ways to
capture and share knowledge, they can increase productivity, collaboration,
and innovation.
NASA's policies and procedures require the continuous capture,
dissemination, and utilization of lessons learned. NASA defines a lesson
learned as knowledge or understanding gained by experience. The experience
may be positive, as in a successful test or mission, or negative, as in a
mishap or failure. A lesson must be significant in that it has a real or
assumed impact on operations; valid in that it is factually correct; and
applicable in that it identifies a specific design, process, or decision
that reduces or eliminates the potential for failures and mishaps, or
reinforces a positive result.
NASA's Policies and Procedures Require Lessons Learning
Program and Project Managers Are Directed to Review Lessons Learned
To ensure that lessons learned from previous experiences are used as a
resource, NASA Procedures and Guidelines for Program and Project Management
Processes and Requirements (NPG 7120.5A) directs each program and project
manager to review and apply significant lessons learned from the past
throughout the program or project life cycle. Managers are also directed to
consult LLIS prior to major milestones to gain lessons from past programs
and projects that are documented and collected as a benefit to future
programs and projects. Program and project managers are also directed to
document and submit to the LLIS in a timely manner any significant lessons
learned throughout the life of a project or program. One of the objectives
of the agency's guidance is to
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
protect against the recurrence of past mistakes and provide a mechanism for
sharing best practices.
NASA Mishap Reporting and Investigating Policy (NPD 8621.1G) and NASA
Procedures and Guidelines for Mishap Reporting, Investigating, and
Recordkeeping (NPG 8621.1) also require the development of lessons learned
from the mishaps for possible application to existing or future programs.1
In addition, NASA is supposed to exchange lessons learned and other
pertinent safety information of common interest with other federal agencies,
international partners, and NASA contractors. Since NASA uses contractors to
produce many of its systems, NASA's Procedures and Guidelines for Management
of Government Safety and Mission Assurance Surveillance Functions for NASA
Contracts (NPG 8735.2) requires that lessons learned during the contract
surveillance activities be captured and submitted to LLIS.
NASA Uses Numerous Methods for Lessons Learning
Lessons learned mechanisms are used by many commercial and government
organizations to share and use knowledge derived from experience to (1)
promote the recurrence of desirable outcomes, or (2) preclude the recurrence
of undesirable outcomes. Policies and guidelines, programmatic and technical
reviews, training programs, and the LLIS are all mechanisms employed by NASA
for capturing and sharing lessons learned. While not the only means
available for capturing and disseminating lessons learned, LLIS is NASA's
only agency-wide repository for such lessons.
LLIS is NASA's Principal The purpose of LLIS is to capture and share lessons
learned from Mechanism for Sharing programs, projects, and missions and to
help ensure that NASA does not Lessons Learned have to keep "relearning" the
lessons of the past. LLIS was created in 1995
and subsequently updated to include a Web interface. Program and project
managers are directed to use LLIS to identify and share lessons learned,
thus facilitating the early incorporation of safety, reliability,
maintainability, and quality into the design of flight and ground support
hardware, software, facilities, and procedures. Currently, the system
1 NASA defines a mishap as "an unplanned event that results in injury to
non-NASA personnel caused by NASA operations; damage to public or private
property (including foreign property) caused by NASA operations;
occupational injury or occupational illness to NASA personnel; damage to
NASA property caused by NASA operations; or mission failure."
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
contains over 900 lessons learned on the development and design of
aeronautics and space systems.
Lessons entered in the LLIS database are screened for relevance and to
ensure that they do not contain sensitive or proprietary information.
Initial reviews of lessons are usually conducted by the centers with a final
review conducted by the Office of Safety and Mission Assurance. After a
lesson is entered into the system, it is not reviewed for currency or
relevance. In other words, once lessons are entered, they remain in the
database indefinitely. The Office of Safety and Mission Assurance monitors
the system to determine the number of times it is accessed, a search is
conducted, and/or search results are obtained, but does not collect
information on the number of times lessons are applied to a program or
project.
Administrator for Safety and Mission Assurance is Responsible for LLIS
Center and Program Based Systems Used to Collect and Disseminate Lessons
Learned
NASA's Quality Management System Policy (NPD 8730.3) establishes the Quality
Management Systems used by the agency to identify and control processes to
assure the quality of hardware, software, and services provided.2 According
to this policy, the Associate Administrator for Safety and Mission Assurance
is responsible for ensuring that the LLIS database is maintained and
accessible. Furthermore, the policy states that the associate deputy
administrator and the center directors are responsible for ensuring that
LLIS is used to ensure quality and document, investigate, and apply lessons
learned for all programs and projects.
While NASA's LLIS is supposed to be the NASA-wide system, several NASA
centers maintain their own electronic lessons learned systems. For example,
the Goddard Space Flight Center uses the Flights Programs and Projects
Directorate Lessons Learned Database to collect lessons learned for each
flight project. Once a lesson is approved, it is passed along to the center
data manager for review and eventual submission to LLIS, thereby ensuring
that what has been learned at the Goddard Space Flight Center is accessible
to other facilities. One lesson that was passed along from the Goddard Space
Flight Center system is the benefit of testing deployable devices, such as
the Landsat 7 solar array, early in satellite development. Testing devices
early is a cost-effective tactic for developing complex and critical
mechanism subsystems. By testing the Landsat 7 solar array early,
2 NASA's Quality Management System is implemented in accordance with the
International Organization for Standardization's ISO 9000 Quality Management
System Standards.
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
program officials were able to discover problems that could have resulted in
a program delay and increased costs. Another lesson in the Flights Programs
and Projects Directorate Lessons Learned Database makes a recommendation on
minimizing the effects associated with ongoing construction at
mission-critical facilities. Specifically, NASA officials learned the
importance of developing a strategy to minimize conflicts between the
construction schedule and project requirements. One of the recommendations
for future projects is that construction activities in mission-specific
support areas of a mission-critical facility should be suspended 30 days
prior to launch to ensure that the facility is ready to support mission
operations.
In addition, program-based lessons learned databases have been established
for key NASA programs. For example, the International Space Station and
Phase 1 Lessons Learned Database documents historical lessons learned from
the Phase 1 Program, U.S./Russian Program Office of the International Space
Station Program; the top ten lessons learned during the Skylab program; and
current lessons learned from the International Space Station Program.3 The
International Space Station and Phase 1 Lessons Learned Database is the
official system for submitting new lessons learned in the International
Space Station program. The lessons learned collected in that Lessons Learned
Database are intended to be used as tools for the improvement of
International Space Station operations. NASA officials are deliberating the
feasibility of transferring the lessons learned from the International Space
Station database to the NASA-wide LLIS.
Another mechanism used by NASA to share lessons is training. NASA's Academy
of Program and Project Leadership (APPL) provides numerous courses
throughout the year.4 Many of the classes employ case studies to learn from
successes and failures. According to an academy official, lessons learned
are incorporated into the different courses. Lessons incorporated into the
curriculum are generic and can be applied to all programs (i.e., better
communications) rather than technical issues unique to a particular program.
About 4,000 program management employees
NASA Uses Training for Capturing and Sharing Lessons
3 Skylab was the United States' first experimental space station. The Skylab
program's objectives were to prove that humans could live and work in space
for extended periods, and to expand our knowledge of solar astronomy.
4 The Academy of Program and Project Leadership is the NASA education
organization responsible for training program and project managers.
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
have attended academy courses, and thus they contribute to sharing
information within the organization.
In addition to formal training, APPL engages in development activities,
which play a role in diffusing knowledge throughout NASA. For example, it
hosts a series of forums on "hot topics" to gather and disseminate
information. Another activity involves capturing and sharing the experiences
of program managers. Once a year, APPL sponsors two sessions in which
project managers considered the "best" by their centers are sent to discuss
project management. At these sessions, two or three project managers from
each center share what works and the types of tools that they need to
accomplish their jobs.
Another activity used by APPL is encouraging senior program managers to
share their knowledge through a series of short stories made available
through its Web site. Each story discusses a topic that will help program
managers succeed. One recently completed story dealt with the importance of
saying no. According to a NASA official, many developing project and program
managers are driven by NASA's "can do" attitude and have a hard time knowing
when to say "no" during a project. NASA is also identifying capable
individuals to mentor new project and program managers. In addition,
managers are provided several online tools and training resources, such as
project management tools that can be used to effectively manage a project.
According to NASA officials, periodic revisions to agency policies and
procedures are another way that the agency shares lessons learned. For
example, NASA is currently revising its Program and Project Management
Processes and Requirements (NPG 7120.5A) and will incorporate some of the
issues raised by the Mars Climate Orbiter and Polar Lander mishaps. This
document is intended to provide the basic processes, requirements, and
responsibilities for managing NASA programs and projects.
In response to the Mars Program failures and the recommendations of the NASA
Integrated Action Team, NASA has taken steps to promote the capture and
sharing of knowledge within the agency by developing new policies,
processes, and practices for program and project management. Foremost among
these is the development of a business strategy commonly referred to as
knowledge management. According to NASA officials, knowledge management has
the potential to link people with the information and resources they need to
complete tasks faster, better, and cheaper. Knowledge management can be
defined as the way that
Changes to Policy Also Reflect Lessons Learned
Additional Steps Taken by NASA to Facilitate Lessons Learning
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
organizations create, capture, and re-use knowledge to achieve
organizational objectives. Organizations that manage knowledge effectively
claim higher performance, innovation, and collaboration among employees.
Lesson learning in the context of knowledge management is defined as
"knowledge, both positive and negative, gained through experience, which if
shared, would benefit the work of others."
NASA Created a Knowledge Management Team to Coordinate Knowledge Sharing
Activities
NASA has recently created a knowledge management team, composed of
headquarters and center officials, to advise the agency's chief information
officer (CIO). The team does not act as a central authority for knowledge
management, but rather as a focal point for coordinating knowledge
management activities conducted throughout the agency. Its goal is to
connect the various knowledge management activities ongoing in the agency
and avoid duplication. The team is currently working on efforts to baseline
knowledge management in NASA, gather requirements, and define opportunities
for applying knowledge management.
Knowledge Management Projects Conducted by Selected Centers
Several centers have initiated knowledge management projects in the past few
years. For example, the Ames Research Center is designing and developing a
comprehensive information technology program called Design for
Safety/Engineering for Complex Systems to help programs and projects
agency-wide manage risk and capture and disseminate knowledge and lessons
learned. The Johnson Space Center is piloting a Quality Assurance database,
and the Jet Propulsion Laboratory is developing an "experts" directory as
well as a Technical Questions database. In addition, the Kennedy Space
Center has created a category of knowledge managers in each of its
directorates who are responsible for identifying and recognizing core
knowledge management competencies. Furthermore, the Langley Research Center
has initiated a process to reward scientists if knowledge they have shared
is used by others. The re-use of the knowledge must be documented in order
for the scientist to receive the reward.
CIO-Sponsored Knowledge NASA's CIO supports agency knowledge management
efforts by investing
Management Pilot Projects in information technology projects to facilitate
knowledge sharing. The NASA Knowledge Management Team advises the CIO on
projects submitted by the centers for funding. In fiscal year 2000, the CIO
funded three knowledge management pilot projects totaling $567,000. For
fiscal year 2002, the CIO has a budget of $3 million that can be used to
fund selected knowledge management initiatives. Because knowledge
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
management projects are not centrally managed or funded, there is no
visibility over how much the various NASA centers spend on them.
One of the projects being funded is to develop an agency-wide Web portal to
bring together NASA's online information resources,5 which are currently
dispersed across different agency functional areas, centers, and programs.
The portal will integrate existing resources and provide users with more
direct access to information from across the agency. A second project is
intended to develop a directory of agency expertise, which will allow NASA
personnel to quickly locate others working in a related field or on a
particular project and thus enhance collaboration among distributed groups.
When complete, the directory will be integrated into the NASA Web portal.
The third CIO-funded project will improve the capture and re-use of lessons
learned by augmenting the agency's current LLIS. The pilot project was a
collaborative effort involving the Jet Propulsion Laboratory, Langley
Research Center, and the Air Force Research Laboratory. The Jet Propulsion
Laboratory team was responsible for defining the processes used to create
and capture lessons learned, as well as the functional and system
requirements for the system's redesign. The Langley Center was responsible
for developing the digital library technology for the system. The Air Force
Research Laboratory role was to evaluate the system design.
After spending about $135,000 to develop a system requirements document and
a new system software design, NASA terminated the pilot project. According
to a NASA official, the Office of Safety and Mission Assurance considered
the functional and system requirements document to be inadequate, and their
office lacked the resources to implement the proposed prototype. However,
since the termination of the pilot, the Goddard Space Flight Center has been
tasked with determining ways to improve the LLIS. Officials there have
proposed a subscription service that would enable users to receive lessons
learned tailored to their interests. Other follow-on LLIS improvement
efforts under consideration include an improved searching and storage
capability and the ability to append to, update, and control lessons
configuration over extended periods of time.
5 A portal is a Web site that is a major starting site for users when they
connect to the Internet or Intranet. A portal can include features such as a
topical index, search engine, e-mail, news, and links to other online tools.
Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning and
Knowledge Sharing
Strategic Plan for Knowledge Management
In March 2001, the Knowledge Management Team issued a Strategic Plan for
Knowledge Management that provides a framework for addressing knowledge
management in NASA in terms of people, process, and technology (see table
2). According to the plan, "people" are the most important component of
knowledge management. It calls for recognizing and rewarding people for
sharing knowledge as well as encouraging story telling and establishing
communities of practice to promote knowledge sharing. The plan also
advocates the development of policies and procedures for capturing and
managing knowledge that must be published and understood by users. In
addition, the plan emphasizes the need for better information technology to
facilitate knowledge management. In the context of lessons learning, the
plan suggests providing a subscription service that tailors specific lessons
to user-defined interests and pushing positive lessons to users' desktops
based upon project tasks or organizational characteristics.
Table 2: NASA's Framework for Knowledge Management
People Process Technology
Enable remote Enhance knowledge Enhance system integration collaboration
capture and data mining
Support communities of Manage information Utilize intelligent agents
practice
Exploit expert systems Reward and recognize knowledge sharing
Encourage storytelling
Source: Strategic Plan for Knowledge Management, March 2001.
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
Fundamental Weaknesses Exist in NASA's Processes, Procedures, and Systems
for Lesson Learning
Although NASA recognizes it is more critical than ever to share knowledge
and collect, store, and distribute lessons learned, the agency's current
array of processes, systems, and culture do not facilitate such an
environment. Importantly, our survey revealed that lessons are not routinely
identified and shared by program and project managers, LLIS has not proven
useful to these managers, and there is little incentive for managers to
share. As a result, program and project managers are not maximizing
opportunities to apply lessons learned toward its faster, better, and
cheaper efforts.
NASA's procedures and guidelines require that program and project managers
review and apply lessons learned from the past throughout a program's or
project's life cycle and document and submit any significant lessons learned
in a timely manner. However, a survey we conducted of all NASA program and
project managers revealed fundamental weaknesses agency-wide in the
collection and sharing of lessons learned. While some lessons learning does
take place, our survey identified that lessons are not routinely identified,
collected, or shared by program and project managers. In addition, many
respondents indicated that they are dissatisfied with NASA's lessons learned
processes and systems. Respondents also identified challenges or cultural
barriers to the sharing of lessons learned as well as areas of improvement.
As a consequence, there is no assurance that lessons are being learned and
applied to future mission success. We provided the basic results of our
survey in a September 2001 report.1 The following analysis builds on those
results.
Limited Sharing of Lessons Learned Agency-Wide
To better understand the extent that lessons are being collected and shared
within NASA, we asked program and project managers what they knew about
lessons generated by their own programs and centers as well as by other
centers. As shown in figure 2, program and project managers responded that
while they are very or somewhat knowledgeable about lessons generated by
their own programs and centers, they know less about lessons generated
elsewhere. This fact was further substantiated by survey results which show
that managers primarily identify lessons through program-or center-based
activities such as project reviews or informal discussions with colleagues.
For example, one project manager
1 U.S. General Accounting Office, Survey of NASA's Lessons Learned Process,
GAO-01-1015R (Washington, D.C.: 2001).
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
stated that monthly reviews with center management provided a way to capture
and share lessons learned. The LLIS, NASA's primary method for disseminating
lessons learned agency-wide, was not identified as a primary source for
lessons learning.
Figure 2: Program and Project Managers Knowledge of Lessons Learned
Respondents to our survey also indicated that their principal sources for
identifying lessons learned were (1) system and engineering reviews, (2)
program and project briefings, and (3) informal discussions with colleagues.
According to NASA program officials, regular program and project meetings,
reviews, and briefings provide an opportunity to share information,
including lessons learned. For example, officials at Kennedy Space Center
explained that after each space shuttle launch they gather to discuss what
went well and what could be done better. Program and project managers also
indicated that they maintain informal networks with colleagues where lessons
learning take place.
Project reviews and informal discussions with colleagues are important
mechanisms for lessons learning; however, they do not facilitate the
systematic, agency-wide sharing of lessons. LLIS, which was established for
such agency-wide sharing, is not widely used, according to survey
respondents, even though NASA guidelines state that program and project
managers should consult it prior to major milestones and submit any
significant lessons learned in a timely manner. One reason the system is not
widely used, according to one center official, is that its lessons cover so
many topics that it is difficult to search for an applicable lesson. One of
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
our survey respondents indicated that "It is difficult to weed through all
the irrelevant lessons to get to the few `jewels' that you need to find.
There should be better categories to find relevant lessons." Also,
contractors, who perform much of NASA's work, are unable to query the system
easily. As shown in figure 3, in the past 2 years, 43 percent of program and
project managers have not submitted a lesson to the LLIS compared to 23
percent of managers who have. The survey revealed there is also a low level
of awareness of the system among managers; 27 percent of program and project
managers were not aware of LLIS before our survey.
According to some project officials, accessing LLIS is time consuming and
uses significant project resources. One official said it took about two
weeks to review the lessons in the database. Another said that developing
the process of assessing the applicability of lessons learned from the
database for that project was very labor intensive and costly. For example,
one project reviewed all the lessons learned in LLIS and attached the
appropriate lessons learned as an appendix to the project's Mission
Assurance Plan. In contrast, another project developed a matrix containing
all the lessons learned from LLIS categorized by functional area and
technical discipline. Project teams organized by discipline then reviewed
the lessons learned for applicability to the project. Further, while some
projects formally reviewed lessons learned from the Mars Program failures,
others did not. However, other projects reviewed the lessons learned from
both LLIS and from the Mars Program failures.
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
Figure 3: Many Program and Project Managers Have Not Submitted Lessons to
the Lessons Learned Information System
Dissatisfaction with Lessons Learned Processes and Systems
Our survey also determined that more managers are very or generally
dissatisfied than are satisfied with NASA's lessons learned processes and
systems, including LLIS. This level of dissatisfaction may stem from the
fact that 58 percent of managers stated that current processes and systems
do not allow them to retrieve the right lessons at the right time. As shown
in figure 4, the level of dissatisfaction may also stem from the fact that
53 percent managers stated that they found lessons useful less than 25
percent of the time. One senior center official stated LLIS serves more as a
repository of information that favors technical lessons, and is not
effective for passing along such information.
Also, managers stated that it would be beneficial if more emphasis were
placed on the reporting of positive lessons that are derived from
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
successful program outcomes. For example, a positive lesson noted on the Jet
Propulsion Laboratory's Deep Space 1 program was having the same program
management team from the beginning through the end of the primary mission.
One manager noted that positive lessons can be even more helpful to a
project than negative ones because even if a project avoids negatives,
without positives managers may not follow the most effective and efficient
path to mission success.
Figure 4: Most Program and Project Managers Could Not Identify Helpful
Lessons for Their Program or Project
Barriers Exist to Lessons If lessons learning is to succeed, it is important
that the organizational
Learning culture fosters the value of sharing knowledge based on others'
experiences. Our survey respondents identified several notable cultural
barriers that affect lessons learning within NASA. These included:
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
* perception that there is no benefit from lessons learned;
* lack of time to share knowledge;
* lack of trust;
* intolerance for mistakes.
Our survey found that many program and project managers (39 percent) believe
that implementing lessons learned contributes only some or little to
improving mission effectiveness. This perception was highlighted by one
manager who stated that "Until we can adopt a culture that admits frankly to
what really worked and didn't work, I find many of these tools to be
suspect."
Another critical barrier highlighted by the survey is the lack of time
available for lessons learned. Managers reported that they and their project
staff are very busy conducting assigned project tasks, and little, if any,
time is allotted for lessons learning. One manager noted that "It is time
consuming to agree on correct lessons learned for a project and accurately
describe the issue...in a way that is helpful to a project." Managers also
noted that there is reluctance to share negative lessons for fear that they
might not be viewed as good project managers. According to some respondents,
this appears to stem from a culture that sees lessons learned as negative,
i.e., an admission of failure. For example, one manager noted that "People
are never rewarded for telling about how they screwed-up and caused a
problem/mistake.... This will continue to be a problem until a way is found
to allow and encourage people to talk about their mistakes without feeling
that they are risking their careers." The cultural barriers to lessons
learning identified above present a serious challenge for NASA and the
agency may well be missing fundamental opportunities to share and apply
knowledge toward future mission success.
Suggested Improvements While program and project managers identified a
number of ways for
to Lessons Learning improving lessons learning within NASA, mentoring was
their first choice. One manager noted that the most effective lessons
learned process results from the passing down of lessons from experienced
people to those less experienced. We found that mentoring programs have been
reinvigorated at a few centers in response to lessons learned from the
faster, better, and cheaper changes as well as concerns that management
expertise is being lost because of increasing retirements. Other suggestions
for improvement
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
included "storytelling" or "after-action reviews." For example, NASA's
Academy of Program and Project Leadership asks senior program managers to
share their knowledge through a series of short stories. Each story is 1 to
3 pages and discusses a topic that will help program mangers to succeed. A
recently completed story deals with the importance of knowing when to say
"no" in a project. However, most managers in our survey did not identify the
academy as a principal source for lessons learning. Another notable
suggestion was that senior management should be accountable for the infusion
or engineering of lessons into a project with some kind of a matrix/metric
to validate that the lessons were actually incorporated.
Managers also suggested a number of improvements regarding LLIS. As shown in
table 3, suggested improvements were related to (1) developing a means to
disseminate lessons from users, (2) improving the search capability, and (3)
including more positive lessons. Currently with the LLIS, there is no
automated dissemination of lessons, it is difficult to find relevant
lessons, and there are few positive lessons included.
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
Table 3: Program and Project Manager Observations and Suggestions to Improve
LLIS
Category Specific Suggestion
Dissemination * For non-NASA employees, it is extremely difficult to be
granted access on a project basis. Team members on my project must apply for
and be granted access on an individual basis. The process for access is
tedious to say the least and this discourages routine use of the LLIS.
* Send out lessons of the month.
* Include a bulletin each week in the center newsletters about a lesson
learned.
* Send out alerts.
* Tailor the dissemination or notification to target opportunities for
benefit rather than blanket notification. Too much data inhibits
information.
Searchability * Have Mission Assurance Manager or Systems Engineer review
LLIS on periodic basis & disseminate info where appropriate; this same
individual could review failure/discrepancy reports for submission to LLIS.
* Separate safety and mishap from program, technical, etc, lessons learned.
Provide possible separate systems for each category: safety, technical,
theoretical, Commercial product usage, contracting, project management, etc.
* Better indexing -LLIS search usually does not return a high ratio of
relevant information
* Improved search engine.
* Categorize the lessons into categories like the various technical
disciplines and management disciplines, so that it would be easy to scan the
ones you think are relevant to you.
More Positive Lessons * Need more "positive" lessons learned. They can be
even more helpful to a project than the "negative" ones. Even if a project
avoids "negatives", without "positives" they may not follow the most
effective and efficient path to mission success.
* When NASA has a failure, there are all sorts of reports on what went wrong
from independent review panels (e.g. Mars Reports). It seems there would
definitely be some benefit from similar type reports on projects that were
successful, pointing out what these projects have done right.
* Instead of focusing on the negative, focus on the positive things. There
are some really well run projects. If people can look to great successes,
instead of worrying about failures, I believe that they will try to emulate
those successes. NASA has always been known as a risk taker because of what
we do. We have no one else to look to, but within ourselves. I'm beginning
to see something new at NASA -fear of taking a risk. As long as risks are
calculated, and the project/program buys-in to taking that risk, it's not a
bad thing.
In discussions with NASA officials, we found there was general agreement
with the problems and suggested improvements for lessons learning that were
identified in our survey. Officials indicated that lessons learning has
taken on greater importance in recent years due to the implementation of
more programs and projects under the FBC strategy and the continuing loss of
agency expertise due to attrition. They acknowledge that LLIS has not been
an effective mechanism for agency-wide sharing of lessons. Although the
system is viewed as providing a useful repository for storing lessons,
officials agreed with managers' concerns about the difficulties involved in
searching the system and finding relevant lessons, the inconsistent quality
of information contained in the system, and the lack of lessons about
positive project experiences. However, while program and project managers'
suggested improvements would help increase the
Chapter 3: Lessons Learned Are Not Being Consistently Shared or Applied
Toward Future Program Success
usability of LLIS, they do not target some of the more fundamental problems
hampering NASA's ability to share lessons, such as the cultural barriers
that persist.
Chapter 4: Creating an Environment for Lessons Learning through Knowledge
Management
The limitations in NASA's ability to share lessons learned point toward two
underlying problems: cultural resistance to sharing knowledge and the lack
of an effective strategic framework and management attention for overcoming
such resistance. The same problems often hamper organizations not only in
sharing lessons learned, but also in broader efforts to use knowledge as a
tool for increasing productivity and fostering innovation. We found that
successful industry and government organizations have learned to overcome
such barriers by developing strategic plans that provide a framework for how
knowledge management will be implemented and by securing the management
support and commitment needed to bring these plans to fruition. Moreover,
they employ incentives, processes, and systems designed to overcome
resistance and other cultural impediments to knowledge sharing. Just as
employing these practices enables leading organizations to maintain a
competitive edge, doing so can better position NASA to perform its basic
mission of exploring space, faster, better, and cheaper.
We contacted practitioners of knowledge management in both government and
industry to gain insight relative to how the application of knowledge
management principles could mitigate the fundamental weakness in lessons
learning identified by our survey. This effort included site visits and
interviews of government and industry knowledge management practitioners,
analysis of published case studies, and review of available literature. The
outcome was the identification of common characteristics inherent in the
design and implementation of knowledge management programs operated and
maintained by government and commercial organizations. Literature searches
and reviews of case studies also identified general principles deemed
critical to the success of knowledge management.
Principles of Knowledge Management Could Help Mitigate Fundamental
Weaknesses in Lessons Learning
Knowledge Management Must Be Linked to a Business Plan
According to practitioners of knowledge management, the first and perhaps
most important element of knowledge management is to put in place a
strategic plan that makes a business case for knowledge management. This
involves translating the abstract concept of knowledge management into a
vision, with goals and a roadmap for sharing and using knowledge within the
organization. It also involves developing a collective vision that is long
term and a commitment by senior management that they will see the plan
through.
NASA has developed a strategic plan that identifies knowledge management
goals and objectives the agency hopes to achieve. NASA,
Chapter 4: Creating an Environment for Lessons Learning through Knowledge
Management
however, has not developed a business plan for achieving implementation of
its vision and goals. Such a plan at a minimum should address the following:
* roles and responsibilities;
* knowledge needs and how they it relate to business processes;
* the role of information technology;
* a timetable for implementing for knowledge management;
* resources needed for implementing a knowledge strategy
* cultural barriers to learning;
* metrics needed for tracking and measuring results; and
* training.
Senior Management Support is Key to Knowledge Management's Success
While NASA senior management has shown its support for knowledge sharing and
lessons learning, our survey nonetheless indicates that many program and
project managers believe senior management support is lacking. Over
one-third of survey respondents indicated that the effective utilization of
lessons learning at NASA is inhibited because senior management lacks
commitment to its use. One respondent stated that if management is serious
about capturing and using lessons learned, then senior management must be
seen and heard implementing the policy.
Of the organizations we observed, most had senior management who saw the
strategic value in using knowledge and communicated to their employees that
sharing knowledge is critical to their success.1 At Ford Motor Company, the
chief executive officer serves as a role model for knowledge sharing,
personally writing weekly e-mails to employees with comments on his thoughts
about and experiences of the past week.2 Moreover, Ford executives provide
sponsorship and support for
1 Thomas Davenport and Laurence Prusak, Working Knowledge: How Organizations
Manage What They Know (Boston: Harvard Business School Press,1998).
2 Ford Motor Company 1999 Annual Report.
Chapter 4: Creating an Environment for Lessons Learning through Knowledge
Management
knowledge management. They encourage employee participation and recognize
employees who share information.
An aerospace company believes that one of the best ways to disseminate
lessons learned is to have management use them. This means that for the
dissemination of lessons learned to be considered important by all the
employees in the company, the executive team must understand and value them.
This is accomplished by including as part of its executive training course
items such as teaching managers how to file after action reports.
Management of Knowledge Sharing Activities
Although NASA's policies and procedures require continuous collection and
sharing of lessons learned, NASA has not established a central management
function to coordinate and direct its lessons learning processes and
systems. Lessons learning activities are dispersed across several agency
components. The Office of Safety and Mission Assurance has designated an
administrator to maintain the LLIS; however, responsibility for LLIS is one
of several assigned duties for the administrator. As a result, little effort
is devoted to managing LLIS beyond keeping it operating. NASA has also
established a team to coordinate recent knowledge management initiatives,
but the team lacks authority to shape the direction or provide resources and
tools to improve knowledge sharing.
Our research indicates that organizations that have implemented knowledge
sharing programs often designate a knowledge management steward. For
example, a recent article on knowledge management notes that ". . . a job
title at Philip Morris is knowledge champion, Monsanto has a director of
knowledge management, and Dow Chemical has a director of intellectual assets
management."3 Once such a position has been created, the steward is then
responsible for identifying needs and opportunities for improvement,
obtaining and distributing resources, applying and enforcing policies, and
conducting daily operations. At Ford, a best practices coordinator manages
the knowledge-sharing program. Similarly, at the World Bank, a Knowledge
Management Office leads an organization-wide effort at knowledge sharing.
3 Verna Allee, Linking People, Learning and Performance -12 Principles of
Knowledge Management, American Society for Training and Development, (2001).
Chapter 4: Creating an Environment for Lessons Learning through Knowledge
Management
Investing in Knowledge Sharing Is Viewed as Important by Successful
Organizations
NASA currently invests little in lessons learning. LLIS costs $50,000 to
$100,000 annually to operate. And, according to NASA officials, funding is
insufficient to upgrade LLIS in any significant way. Although there is no
set answer as to the appropriate dollar amount that should be devoted to
knowledge management, our review shows that those organizations that
dedicate resources tend to be more successful at sharing knowledge
effectively. Many of the organizations we contacted reported significant
investments in knowledge management activities. For example, the World Bank
invests about 3 percent of its total administrative budget in knowledge
management. These resources are used for information technology systems,
training, and dedicated staff to facilitate knowledge sharing.
A Corporate Culture That Encourages Knowledge Sharing Is a Key Element for
Success
Effective Knowledge Sharing Requires Adequate Time
Although NASA acknowledges the importance of having an organizational
culture that promotes teamwork and knowledge sharing, our survey indicated
that significant cultural barriers within the agency inhibit an environment
and culture that supports continuous learning. Our research indicates that
practitioners of knowledge management foster an environment and culture that
support continuous learning. Many knowledge management leaders agree that
knowledge management is 90 percent culture and 10 percent technology.4 A
culture of knowledge sharing can be encouraged in many ways.
A common complaint among NASA survey respondents is the lack of time for
sharing lessons learned. Consequently, knowledge-sharing activities at NASA
are seen as simply an additional burden on an already tight schedule. Yet,
if an organization values knowledge sharing it will allow time for knowledge
sharing. Knowledge can only be shared when employees are given adequate
time, as well as established places where they can actually transfer
knowledge.
Formal And Informal Our survey indicated that much of the knowledge sharing
at NASA is done
Knowledge Sharing on an informal basis. While talking informally with
colleagues is one method of sharing information, it is not necessarily the
most efficient way. Such informal exchanges do not allow the information to
be further disseminated and do not necessarily allow the information to be
validated.
4 Directorate of eBusiness & Knowledge Management, Office of the Chief
Information Officer, DOD Knowledge Management Primer, Department of Defense.
Chapter 4: Creating an Environment for Lessons Learning through Knowledge
Management
Information exchanges, however, are important because they allow people to
discuss issues spontaneously. Many of the organizations that we examined
instituted more formal arrangements for sharing information. Ford and the
World Bank have established "communities of practice" as a method for
sharing knowledge. They formed groups of employees who are bound together by
shared expertise and a passion for joint enterprise.5 These communities
serve as knowledge transfer programs because employees can share information
regarding a given issue. At the World Bank there are approximately 120
communities of practice-known as "thematic groups" arranged according to
common interests, such as "urban poverty" or "urban services." The urban
services group, for example, focuses on improving the living conditions in
slums around the world by capturing and sharing past experiences and
adapting them to today's experiences.
Recently, some activities at NASA centers have sought to find ways to
encourage knowledge sharing. For example, scientists at NASA's Langley
Research Center are monetarily rewarded if knowledge they capture and share
is re-used. Nevertheless, NASA has not developed a plan that details the
principles, requirements, and architecture of how to implement its knowledge
sharing strategy. While NASA realizes that rewards and recognition should be
part of its initial priority area for knowledge management, the agency has
not yet initiated any agency wide incentives for knowledge sharing.
Most of the organizations that we examined have developed incentives for
knowledge sharing. Ford rewards employees who submit lessons learned that
are adopted by the company. Managers are also encouraged to share because
they are evaluated annually on the basis of knowledge sharing. Similarly,
the World Bank has made learning and knowledge sharing part of core
behaviors covered in each employee's performance evaluation.
Incentives for Knowledge Sharing
Information Is Deemed Valuable
NASA is less successful at infusing lessons learned because the sharing of
lessons learned is not highly valued by some program and project managers.
Successful knowledge sharing organizations share knowledge because they view
it as critical to their success. At Ford, for example, every plant is
responsible for making a 5 percent productivity increase each year.
Employees refer to this as "task." Ford's knowledge-sharing
5 Wenger, Etienne and William M. Snyder. "Communities of Practice: The
Organizational Frontier." Harvard Business Review. Jan/Feb. 2000:139-145.
Chapter 4: Creating an Environment for Lessons Learning through Knowledge
Management
database (Best Practice Replication system) provides managers with
suggestions for improving efficiency, which in turn will enable managers to
meet "task." Knowledge is shared because it meets a critical need. Moreover,
managers at Ford use the lessons provided by the Best Practice Replication
system because they trust the information, and experts within the
organization have checked the best practice in the database for accuracy.6
This is key to the Ford program.
Information Technology is Important, but Should Not Be the Only Mechanism
for Knowledge Sharing
A pitfall of many knowledge management programs is the assumption that a
database will automatically lead to knowledge sharing. Indeed, NASA's
agency-wide effort at lessons learning is in practice limited to LLIS. At
Ford, its electronic database is supplemented by other interactions. Not
only do employees meet at community of practice meetings, but production
engineers from the various plants also meet quarterly to discuss how each
plant has implemented best practices. The World Bank not only uses a variety
of databases to share information; employees also meet face-to-face at
thematic group meetings and at knowledge fairs.
Well-designed information systems and databases, however, are important to
facilitate knowledge sharing, especially for organizations such as NASA that
have employees located at multiple centers across the country. Organizations
with effective knowledge management have designed their database systems
with heavy end-user involvement. The individuals who are intended to use the
databases should be involved in the development and implementation of the
system if it is to be successful. In addition, organizations often use
dissemination mechanisms to "push" important and relevant information to the
right people rather than have users to search for the information. NASA's
LLIS requires users to search for possible useful lessons. In contrast, at
the Department of Energy users receive automatic e-mail alerts regarding
important lessons learned. Similarly, individuals at Ford receive new best
practices every time they open the database system. Also, we found that
organizations frequently dedicate "gatekeepers" to manage and monitor
knowledge-sharing databases in order to keep the information up to date and
relevant. At Ford, a "focal point" is assigned to manage the best practices
system at each plant. The focal point is usually a production engineer
appointed by the plant manager. They receive the pushed messages and also
enter best
6 Nancy M. Dixon, Common Knowledge: How Companies Thrive by Sharing What
They Know, (Boston: Harvard Business School Press, 2000).
Chapter 4: Creating an Environment for Lessons Learning through Knowledge
Management
practices derived from their own plant. Similarly, at the Department of
Energy a lessons learned coordinator maintains the lessons learned system at
each site.
Chapter 5: Conclusions and Recommendations
The failures of the Mars Polar Lander and Climate Orbiter spacecraft raised
concern that NASA was not learning from its past mistakes and applying
lessons learned toward future mission success. Our survey of NASA program
and project managers and subsequent analysis found that there are weaknesses
in NASA's processes, procedures, and systems for lessons learning and
knowledge sharing. Program and project managers indicate that current
processes and systems do not allow them to retrieve the right lesson at the
right time. Managers also report that they are not very knowledgeable about
lessons generated from other centers and programs. This is partly due to the
fact that NASA's LLIS is not widely used. As a result, there is no assurance
that lessons learned from past mishaps or program successes are being
applied to current programs and projects.
NASA has recognized that it must improve its lessons learning and knowledge
sharing capabilities. The mandate to deliver leaner, more demanding, and
increasingly more complex missions has resulted in a greater dependency on
creating multi-disciplinary teams, building alliances with contractors, and
being able to quickly link to and learn from other agency activities.
Following a series of mission failures and mishaps, the NASA Integrated
Action Team reported that the continuous capture and application of project
knowledge and lessons learned must become a core business process within the
agency's program and project management environment. More importantly, the
team stated that program and project managers must regularly use knowledge
management tools to apply previous lessons learned to their projects.
NASA has taken steps to promote the capture and sharing of lessons and other
knowledge within the agency through knowledge management practices.
According to NASA officials, knowledge management has the potential to
rectify many of the weaknesses identified by our survey through more
effective management of the agency's knowledge resources. Current knowledge
management initiatives by NASA should lead to improvements in knowledge
sharing, but the lack of a coordinated and well-supported effort will limit
knowledge sharing agency-wide. Stronger commitment and efforts are needed to
create an environment that encourages knowledge sharing. To achieve this
environment, NASA must overcome communication barriers and create
opportunities where open and candid communications are the norm and
knowledge sharing is valued. In addition, time and resources must be
dedicated in order for lessons learning and knowledge sharing to occur.
Furthermore, improvements to information technology systems such as LLIS are
needed to facilitate the collection, dissemination, and application of
knowledge.
Chapter 5: Conclusions and Recommendations
Recommendations for Executive Action
NASA needs to strengthen its lessons learning in the context of its overall
efforts to develop and implement an effective knowledge management program.
Improvement of NASA's lessons learning processes and systems can help to
ensure that knowledge is gained from past experiences and applied to future
missions.
We recommend that the NASA administrator strengthen the agency's lessons
learning processes and systems by taking actions in the following areas:
Strategic Planning * Articulate the relationship between lessons learning
and knowledge management through development of an implementation plan for
knowledge management.
Coordination * Designate a lesson learned manager to lead and coordinate all
agency lessons learning efforts.
* Establish functional and technical linkages among the various center-level
and program-level lessons learning systems.
Culture * Develop ways to broaden and implement mentoring and "storytelling"
as additional mechanisms for lessons learning.
* Identify incentives to encourage more collection and sharing of lessons
among employees and teams, such as links to performance evaluations and
awards.
LLIS Enhancement Enhance LLIS by
* coding information and developing an easier search capability to allow
users to identify relevant lessons;
* including more positive lessons;
* providing a means to disseminate key lessons to users; and
* soliciting user input on an ongoing basis.
Chapter 5: Conclusions and Recommendations
Performance Measurement
Agency Comments and Our Evaluation
* Track and report on the effectiveness of the agency's lessons learning
efforts using objective performance metrics.
In written comments on a draft of this report, NASA generally concurred with
our recommendations for improving the agency's lessons learned processes and
systems. NASA stated that it must do a better job of communicating the
various lessons learned sources to employees, improving mechanisms to link
these sources, and ensuring appropriate training for employees in order to
maximize lessons learning. NASA further indicated that it will develop plans
to implement our report recommendations. NASA's comments are reprinted in
appendix I.
Page 46 GAO-02-195 NASA
Page 47 GAO-02-195 NASA
Page 48 GAO-02-195 NASA
Page 49 GAO-02-195 NASA
Appendix II: GAO Contacts and Staff Acknowledgments
GAO Contacts Acknowledgments
(120099)
Allen Li, (202) 512-4841 John Oppenheim, (202) 512-3111
In addition to those named above, Christina Chaplain, Diana Dinkelacker,
James Elgas, Jose Ramos, Carl Ramirez, and Lorene Sarne made key
contributions to this report.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its Web
site daily. The list contains links to the full-text document files. To have
GAO e-mail this list to you every afternoon, go to www.gao.gov and select
"Subscribe to daily e-mail alert for newly released products" under the GAO
Reports heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office P.O. Box 37050 Washington, D.C. 20013
To order by phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Visit GAO's Document GAO Building
Distribution Center Room 1100, 700 4th Street, NW (corner of 4th and G
Streets, NW) Washington, D.C. 20013
Contact:
To Report Fraud,
Web site: www.gao.gov/fraudnet/fraudnet.htm,Waste, and Abuse in E-mail:
[email protected], or Federal Programs 1-(800) 424-5454 or (202) 512-7470
(automated answering system).
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D.C. 20548
*** End of document. ***