Survey of NASA's Lessons Learned Process (05-SEP-01,
GAO-01-1015R).
The National Aeronautics and Space Administration's (NASA)
procedures and guidelines require that program and project
managers review and apply lessons learned from the past
throughout a program's or project's life cycle. Managers must
also document and submit any significant lessons learned in a
timely manner. NASA's principal mechanism for collecting and
sharing lessons learned from programs, projects, and missions
agency wide is the Lessons Learned Information System. The goal
of the system is to prevent NASA from having to "relearn" the
lessons of the past. NASA also shares lessons learned through
revisions to its policies and guidance. Furthermore, lessons
learned from a mishap or operational event are captured in
procedure and process documents. GAO surveyed all of NASA's
program and project managers to obtain their perspectives on
NASA's mechanisms to ensure that past lessons learned from
mission failures are being applied. GAO found fundamental
weaknesses in the collection and sharing of lessons learned in
NASA by program and project managers as well as in the system.
Although some lessons learning does take place, lessons are not
routinely identified, collected, or shared by program and project
managers. In addition, many respondents said that they are
dissatisfied with NASA's lessons learned processes and systems.
Respondents also identified challenges or barriers to the sharing
of lessons learned as well as areas of improvement.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-1015R
ACCNO: A01727
TITLE: Survey of NASA's Lessons Learned Process
DATE: 09/05/2001
SUBJECT: Agency missions
Data collection
Internal controls
Surveys
Program evaluation
Information systems
NASA Lessons Learned Information System
Mars Polar Lander Spacecraft
Climate Orbiter Spacecraft
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-1015R
GAO- 01- 1015R Survey of NASA?s Lessons Learned Process United States
General Accounting Office
Washington, DC 20548
September 5, 2001 The Honorable Dana Rohrabacher Chairman, Subcommittee on
Space
and Aeronautics Committee on Science House of Representatives
The Honorable Bart Gordon Ranking Minority Member, Subcommittee
on Space and Aeronautics Committee on Science House of Representatives
Subject: Survey of NASA?s Lessons Learned Process You asked us to assess
whether the National Aeronautics and Space Administration (NASA) has
adequate mechanisms in place to ensure that past lessons learned from
mission failures are being applied. As part of your request, you asked that
we provide the results of our survey regarding the collection and sharing of
lessons learned by NASA program and project managers. This letter provides
the results. We plan to issue a more detailed report on the effectiveness of
NASA?s lessons learned process early next calendar year.
As part of our assessment of NASA?s effectiveness in applying lessons
learned, we surveyed all of NASA?s program and project managers to obtain
their perspectives on (1) the collection, access, and use of lessons
learned, (2) the strengths and limitations of current NASA?s lessons learned
processes, procedures, and systems, including the Lessons Learned
Information System (LLIS), (3) the challenges or barriers to the sharing of
lessons learned, and (4) suggested areas of improvement. This represented
192 managers overseeing about 240 programs and projects. On August 6 and 8,
2001, we briefed your staff on the results of our survey and this report
summarizes those results. The briefing slides are included as enclosure I.
We conducted our work from September 2000 through July 2001 in accordance
with generally accepted government auditing standards. Enclosure II
describes our objectives, scope and methodology.
GAO- 01- 1015R Survey of NASA?s Lessons Learned Process Page 2 BACKGROUND
NASA?s procedures and guidelines require that program and project managers
review and apply lessons learned from the past throughout a program?s or
project?s life cycle and to document and submit any significant lessons
learned in a timely manner. NASA defines a lesson learned as ?knowledge or
understanding gained by experience. The experience may be positive, as in a
successful test or mission, or negative, as in a mishap or failure. A lesson
must be significant in that it has a real or assumed impact on operations;
valid in that it is factually correct; and applicable in that it identifies
a specific design, process, or decision that reduces or limits the potential
for failures and mishaps, or reinforces a positive result.? The failure of
the Mars Polar Lander and Climate Orbiter spacecraft in 1999 raised concern
that lessons learned from past mishaps and programs were not being
effectively applied towards future mission success.
Lessons learned systems are used by many military, commercial and government
organizations to capture, store, disseminate, and share knowledge gained
from past experiences. NASA?s principal mechanism for collecting and sharing
lessons learned from programs, projects, and missions agency wide is the
LLIS. The goal of LLIS is to ensure that NASA does not have to keep
?relearning? the lessons of the past. NASA also shares lessons learned
through revisions to its policies and guidance. Further, lessons learned
from a mishap or operational event are captured in procedure and process
documents.
SURVEY RESULTS
Our survey highlighted fundamental weaknesses in the collection and sharing
of lessons learned in NASA by program and project managers as well as in the
agency?s LLIS. While some lessons learning does take place, our survey
identified that lessons are not routinely identified, collected, or shared
by program and project managers. In addition, many respondents indicated
that they are dissatisfied with NASA?s lessons learned processes and
systems. Respondents also identified challenges or barriers to the sharing
of lessons learned as well as areas of improvement. Specifically,
respondents indicated the following.
Limited Sharing of Lessons Learned Agency- wide
Lessons learned are of little value unless they are collected and shared
with others. To determine the extent that lessons are being collected and
shared within NASA, we asked program and project managers how knowledgeable
they were of lessons generated by their own programs and centers as well as
by other centers. Program and project managers responded that they are very
or generally knowledgeable of lessons generated by their own programs and
centers but are not very knowledgeable of lessons generated by programs at
other centers. This fact is supported by survey results that show managers
primarily identify lessons through program or centerbased activities such as
project reviews or informal discussions with colleagues. LLIS was not
identified as a primary source for lessons learning.
GAO- 01- 1015R Survey of NASA?s Lessons Learned Process Page 3
LLIS Is Not a Primary Mechanism for Collecting and Sharing Lessons Learned
Our survey identified several reasons why LLIS is not the primary mechanism
used by program and project managers for collecting and sharing of lessons.
One reason is a lack of awareness about the system. About one- third of
program and project managers responded that they first learned of LLIS from
our questionnaire or only became aware of the system within the last 6
months. This level of awareness also corresponds with survey results that
show limited use of LLIS by program and project managers within the last 2
years.
Dissatisfaction with Lessons Learned Processes and Systems
Our survey also determined that more managers are very or generally
dissatisfied with NASA?s lessons learned processes and systems, including
LLIS, than are satisfied. This level of dissatisfaction may stem from the
fact that 58 percent of managers stated that current processes and systems
do not allow them to retrieve the right lessons at the right time. Another
contributing factor to this level of dissatisfaction could be because
program and project managers do not find existing lessons helpful: 53
percent of managers said they found lessons learned useful less than 25
percent of the time.
Barriers Exist to Lessons Learning
If lessons learning is to succeed it is important that an organizational
culture exist that fosters the value of sharing knowledge based on others?
experiences. Our survey determined that many managers do not believe that
implementing lessons learned significantly improves mission effectiveness.
One manager stated that ?Until we can adopt a culture that admits frankly to
what really worked and didn?t work, I find many of these tools to be
suspect.?
Another critical barrier highlighted by the survey is the lack of time
available for lessons learned. One manager noted that ? It is time consuming
to agree on correct lessons learned for a project, accurately describe the
issue? in a way that is helpful to a project.? Finally, many managers noted
in their narrative responses that there is reluctance to share negative
lessons for fear that they might not be deemed to be good project managers.
According to some respondents this appears to stem from a culture that sees
lessons learned as negative, i. e., an admission of failure.
Suggested Improvements to Lessons Learning
While program and project managers identified a number of ways for improving
lessons learning within NASA, mentoring was their first choice. One manager
noted that the most effective lessons learned process results from the
passing down of lessons from experienced people to those less experienced.
Other suggestions for improvement included ?storytelling? or ?post mortems.?
A manager stated that story telling would be a quick way to get the message
distributed and allow persons to ask questions. Another notable suggestion
was that senior management be accountable
GAO- 01- 1015R Survey of NASA?s Lessons Learned Process Page 4
for the infusion or engineering of lessons into a project with some kind of
a matrix/ metric to validate that the lessons was actually incorporated.
Managers also suggested a number of improvements regarding LLIS. Suggested
improvements included (1) inclusion of more positive lessons learned, (2) an
improved search capability, (3) addition of a frequently asked question
section, and (4) addition of information to allow one to determine whether a
lesson is applicable to a specific project. Another suggestion was that
lessons be distributed once a month and an alert used to notify managers
about important lessons.
AGENCY COMMENTS
NASA generally agreed with our draft report. Our efforts to assess the
effectiveness of the NASA?s lessons learning processes and procedures is
continuing and we plan to issue a final report on the results of our work
early in the next calendar year. NASA?s comments in their entirety appear in
enclosure III.
We are sending copies of this letter to the NASA Administrator and
interested congressional committees. The letter is also available on GAO's
home page at http:// www. gao. gov. If you or your staff have any questions
regarding the briefing or this letter, please contact me at (202) 512- 4841,
or John Oppenheim, Assistant Director at (202) 512- 3111. Key contributors
to this assignment were James A. Elgas, Carl M. Ramirez, Jose A. Ramos,
Lorene S. Sarne, and Diana Dinkelacker.
Sincerely yours, Allen Li Director Acquisition and Sourcing Management
Enclosures - 3
ENCLOSURE I ENCLOSURE I
Page 5 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
Slides from the Briefing on the Results of NASA?s Lessons Learned Survey
NASA Lessons Learned Survey Results
August 6, 2001
Briefing to the House Science Committee, Subcommittee on Space and
Aeronautics
1
Purpose of Survey
* Obtain views of National Aeronautics and Space Administration (NASA)
program and project managers regarding:
Collection, sharing, and application of lessons learned Strengths and
limitations of NASA's lessons learned
processes, procedures, and systems, including NASA's Lessons Learned
Information System (LLIS)
Challenges or barriers to the sharing of lessons learned Suggested areas
of improvement
ENCLOSURE I ENCLOSURE I
Page 6 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
2
Survey Scope
Conducted an online survey of 192 program/ project managers in June/ July
2001
Covered all Enterprise areas:
Earth Science
Human Exploration and Development of Space
Aerospace Technology
Space Science
Biological & Physical Research
Identified 244 programs and/ or projects
3
Survey Instrument Development
Obtained relevant information from interviews with agency officials
involved in program and project management
Reviewed lessons learned related documents and guidance
Analyzed literature on lessons learned and knowledge management
ENCLOSURE I ENCLOSURE I
Page 7 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
4
Survey Methodology
NASA Headquarters and others reviewed and commented on the proposed
contents of the questions
Conducted a pretest of the survey questions with program and project
managers at Goddard Space Flight Center to gauge clarity, and length of time
of survey administration
Names/ addresses of NASA program and project managers were provided by
NASA at our request
We believe that our survey results are generalizable to the population of
NASA program and project managers
Practical difficulties of conducting any survey, however, may introduce
errors into estimates made from surveys
5
NASA?s Definition of Lessons Learned
?Knowledge or understanding gained by experience. The experience may be
positive, as in a successful test or mission, or negative, as in a mishap or
failure. A lesson must be significant in that it has real or assumed impact
on operations; valid in that it is factually and technically correct; and
applicable in that it identifies a specific design, process, or decision
that reduces or limits the potential for failures and mishaps, or reinforces
a positive result.?
ENCLOSURE I ENCLOSURE I
Page 8 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
6
NASA Processes, Procedures, and Systems for Lessons Learning
NASA?s Program and Project Management Processes and Requirements (NPG
7120.5A) states that program and project managers:
should review and apply lessons learned from the past throughout the
program/ project life cycle, where appropriate
shall document and submit any significant lessons learned in a timely
manner
NASA?s Procedures and Guidelines for mishap investigating (NPG 8621.1)
requires the identification and submission of lessons learned
7
NASA Has Established Processes, Procedures, and Systems for Lessons Learning
NASA?s Lessons Learned Information System (LLIS) is an on- line, automated
data base system, was established to facilitate the capture and sharing of
lessons learned agency- wide
NASA disseminates lessons through periodic updates to policies and
guidance
Lessons learned through mishaps or other operational events are
incorporated into procedure and process documents
ENCLOSURE I ENCLOSURE I
Page 9 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
8
Characteristics of Survey Respondents
Sixty (60) percent response rate
Project Managers 58%
Program Managers 19%
No Answer 3% Other
20% Biological & Physical Research
6% No answer
6% Earth Science
18% Space Science
33% Human Exploration & Development of Space
9% Aerospace
Technology 28%
Respondents by Enterprise Area Respondent Demographics
9
Summary of Survey Results
The survey identified fundamental weaknesses and improvements in the
collection and sharing of lessons learned:
Limited sharing of lessons learned agency- wide
LLIS is not the primary mechanism for collecting and sharing lessons
learned
There are barriers to lessons learning
Dissatisfaction with lessons learned processes and procedures
Suggested improvements to strengthen lessons learned processes and
procedures
ENCLOSURE I ENCLOSURE I
Page 10 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
10
8 9 .6 4 6 .1 5 1 .3
1 4 .8 7 4 .8
7.8 7 .8 0% 2 0%
4 0% 6 0%
8 0% 1 00%
Own Program Pro g rams and Pro jects in C enter
Pro grams an d Pro jects in o ther Cen ters
No Ans we r N ot Very Knowledgeable
S omewhat Knowledgeable
Very Knowledgable Limited Sharing of Lessons Learned
Agency- wide
Program/ project managers are knowledgeable of lessons learned in their
own programs/ centers, but are less knowledgeable of lessons at other
centers
11
Limited Sharing of Lessons Learned Agency- wide
No systematic method for alerting program and project managers of
important lessons
Sharing of lessons mostly occurs during project review and on an informal
basis
ENCLOSURE I ENCLOSURE I
Page 11 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
12
LLIS is Not the Primary Mechanism for Collecting and Sharing Lessons Learned
Principal sources for identifying lessons learned:
1st Source: Systems and Engineering Reviews
2nd Source: Program/ Project Briefings
3rd Source: Informal Discussions with Colleagues
Twenty- seven (27) percent of program and project managers were not aware
of LLIS before the survey
Only 17. 5 percent of the respondents have applied a lessons learned
identified through LLIS
In the past 2 years only 23 percent of program and project managers have
submitted a lessons learned to LLIS
13
None 43%
1- 2 times 12% 6- 9 times
2% 3- 5 times
6% Do n o t know
30% 10 or more times
3% No answer
4% LLIS is Not the Primary Mechanism for
Collecting and Sharing Lessons Learned
In the last two years 43 percent of program and project managers have not
submitted a lesson to LLIS
ENCLOSURE I ENCLOSURE I
Page 12 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
14
LLIS is Not the Primary Mechanism for Collecting and Sharing Lessons Learned
No answer 4%
10 or m ore times 5%
Do not know 29%
3- 5 times 16% 6- 9 times
7% 1- 2 times
17% None
22%
LLIS is seldom used by programs and projects to look for lessons over the
past two years
15
Barriers Exist to Lessons Learning
Thirty- nine (39) percent of program/ project managers believe that
implementing lessons learned contributes only some or little to improving
mission effectiveness
Managers believe there is too little time for individuals to share lessons
learned:
?It is time consuming to agree on correct lessons learned for a project,
accurately describe the issue and the lesson in a way that is helpful to a
project?
Managers are reluctant to share failures with others:
?[ There is an] unwillingness? to share information or air dirty laundry. If
you made a mistake you might not be deemed to be a good project manager.?
ENCLOSURE I ENCLOSURE I
Page 13 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
16
Dissatisfaction with Lessons Learned Processes and Systems
More respondents are very or generally dissatisfied with NASA?s processes
and systems than are satisfied
Fifty- eight (58) percent stated that processes and systems infrequently
allows them to retrieve the right lessons at the right time
Managers indicate that not enough emphasis is being placed on reporting
positive or successful experiences
17
Suggestions Provided by Managers to Improve Lessons Learning at NASA
Mentoring:
?I believe ?lessons learned? [equals] experience. Experience is gained on
the job, and much of it is passed along informally through mentoring. This
means there must be a base of experienced people who must be allowed to
mentor the newer employees!?
Expanded practice of ?storytelling? or ?post mortems?
ENCLOSURE I ENCLOSURE I
Page 14 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
18
Suggestions Provided by Managers to Improve Lessons Learning at NASA
Develop a capability to disseminate or notify users about important
lessons learned:
?Centers should transmit public (Center- wide) emails containing important
or prominent lessons learned on a regular basis?
19
Suggestions Provided by Managers to Improve Lessons Learning at NASA
Better integration of lessons learned into program/ project planning and
implementation procedures
Allow more time for the lessons learned process:
?If it [lessons learned] is important then make it important. Give it the
processes, procedures, resources, and particularly [the] time, to make it a
positive experience. However, merely making it another requirement to an
already overwhelming work schedule will doom it to the failure it presently
may be experiencing?
ENCLOSURE I ENCLOSURE I
Page 15 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
20
Recognition of the Importance of Lessons Learned
?I am encouraged by the focus and attention the lessons learned process is
getting. I really hope that the follow- through is sustained... rhetoric is
cheap and gets the ?motherhood and apple pie? salute. The real action of
process development and improvement [is done] by actually engineering the
lessons into the processes...[ but this] is costly and hard. A lot of the
work is about culture change to accept a new model that contrasts to ?the
way we've always done it. ?"
Project Manager
ENCLOSURE II ENCLOSURE II
Page 16 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
Survey Objectives, Scope and Methodology To obtain the views of NASA program
and project managers on lessons learned processes and systems at NASA we
conducted a survey in June and July 2001, using a self- administered
electronic questionnaire posted on the World Wide Web. The survey contained
four groups of questions on (1) how NASA personnel collect, access and use
lessons learned; (2) the strengths and weaknesses of current NASA lessons
learned processes, procedures and systems, including the Lessons Learned
Information System; (3) potential challenges or barriers to sharing Lessons
Learned within NASA; and (4) ways to improve the lessons learned process or
system at NASA.
In designing the questionnaire, we interviewed NASA officials, program and
project managers, as well as other government and industry officials who had
insight into lessons learned processes and systems. We also obtained and
reviewed NASA documents and guidance pertinent to lessons learned. To
further guide the development of appropriate questions, we reviewed current
literature on lessons learned and knowledge management. To validate the
content and structure of the questionnaire, we submitted it to officials at
NASA Headquarters and others for review and incorporated their comments
where appropriate. To verify the clarity, length of time of administration,
and suitability of the questions, we also pretested the questionnaire with
selected program and project managers at Goddard Space Flight Center.
NASA officials provided us with a list of the e- mail addresses of 199
personnel that they indicated were representative of all NASA program and
project managers as of April 2001. On June 5, 2001, we sent e- mail messages
to each person in this survey population notifying them of the survey and
asking them to complete the questionnaire. Subsequently, we discovered one
additional NASA manager who was eligible for our survey, and removed 7
managers who were ineligible for the survey because they were not project or
program managers at the time of the survey. We also eliminated one duplicate
listing, resulting in a final survey population of 192.
Over the following several weeks until closing the survey on July 13, 2001,
we received a total of 115 useable responses, for an overall response rate
of 60 percent. All of the responses had been submitted using the web
questionnaire, although one participant had asked to submit a paper version
of the survey, due to problems in accessing the web survey. Of the 77 non-
respondents, 9 provided partial questionnaire responses but had not
indicated that they were finished with the questionnaire.
While we believe that our survey results are generalizable to the population
of NASA program and project managers as described above, the practical
difficulties of conducting any survey may introduce errors into estimates
made from surveys.
ENCLOSURE II ENCLOSURE II
Page 17 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
Although we administered questionnaires to all known members of the
population, and thus our results are not subject to sampling error,
nonresponse to the entire survey or individual questions can introduce a
similar type of variability or bias into our results to the extent that
those not responding differ from those who do respond in how they would have
answered our survey questions. In addition, population coverage errors can
occur if some members of the population are excluded from the survey.
Measurement errors can arise from how questions are interpreted by
respondents and mistakes made by respondents. Data processing errors can
arise during the handling or analysis of responses. We took steps in the
design, data collection and analysis phases of our survey to minimize such
errors, such as pretesting questionnaires before the survey, following up
with those not reachable at original e- mail addresses or otherwise not
immediately responding, and checking for errors in computer programming used
to analyze survey results. In addition, the distribution of respondents
across NASA Enterprise areas generally reflected the actual distribution of
the entire population, which was consistent with our belief that nonresponse
error was not significant.
ENCLOSURE III ENCLOSURE III
Page 18 GAO- 01- 1015R Survey of NASA?s Lessons Learned Process
Comments From the National Aeronautics and Space Administration
(707565)
*** End of document. ***