NASA: Lack of Disciplined Cost-Estimating Processes Hinders
Effective Program Management (28-MAY-04, GAO-04-642).
For more than a decade, GAO has identified the National
Aeronautics and Space Administration's (NASA) contract management
as a high-risk area--in part because of NASA's inability to
collect, maintain, and report the full cost of its programs and
projects. Lacking this information, NASA has been challenged to
manage its programs and control program costs. The scientific and
technical expectations inherent in NASA's mission create even
greater challenges--especially if meeting those expectations
requires NASA to reallocate funding from existing programs to
support proposed new efforts. Because cost growth has been a
persistent problem in a number of NASA programs, GAO was asked to
examine NASA's cost estimating for selected programs, assess
NASA's cost-estimating processes and methodologies, and describe
any barriers to improving NASA's cost-estimating processes. To
conduct GAO's work, GAO analyzed a total of 27 NASA programs--10
of which GAO reviewed in detail.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-642
ACCNO: A10429
TITLE: NASA: Lack of Disciplined Cost-Estimating Processes
Hinders Effective Program Management
DATE: 05/28/2004
SUBJECT: Financial analysis
Financial management
Contract administration
Cost analysis
Cost control
Program management
Data collection
Data integrity
NASA Integrated Financial Management
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-642
United States General Accounting Office
GAO Report to the Committee on Science, House of Representatives
May 2004
NASA
Lack of Disciplined Cost-Estimating Processes Hinders Effective Program
Management
a
GAO-04-642
Highlights of GAO-04-642, a report to the Committee on Science, House of
Representatives
For more than a decade, GAO has identified the National Aeronautics and
Space Administration's (NASA) contract management as a high-risk area-in
part because of NASA's inability to collect, maintain, and report the full
cost of its programs and projects. Lacking this information, NASA has been
challenged to manage its programs and control program costs. The
scientific and technical expectations inherent in NASA's mission create
even greater challenges-especially if meeting those expectations requires
NASA to reallocate funding from existing programs to support proposed new
efforts.
Because cost growth has been a persistent problem in a number of NASA
programs, GAO was asked to examine NASA's cost estimating for selected
programs, assess NASA's cost-estimating processes and methodologies, and
describe any barriers to improving NASA's cost-estimating processes. To
conduct GAO's work, GAO analyzed a total of 27 NASA programs-10 of which
GAO reviewed in detail.
GAO is recommending that NASA take a number of actions to better ensure
that the agency's planned and recently implemented initiatives to improve
its costestimating practices will result in sound cost estimates and
thereby enable NASA to control its programs better.
May 2004
NASA
Lack of Disciplined Cost-Estimating Processes Hinders Effective Program
Management
Considerable change in NASA's program cost estimates-both increases and
decreases-indicates that NASA lacks a clear understanding of how much its
programs will cost and how long they will take to achieve their
objectives. For example, the development cost estimates for more than half
of the 27 programs that GAO reviewed have increased and for some programs
this increase was significant-as much as 94 percent. Cost estimates
changed for each of 10 programs that GAO reviewed in detail. For 8 of the
10 programs, the estimates increased. Although NASA cited specific reasons
for the changes, such as technical problems and funding shortages, the
variability in the cost estimates indicates that the programs lacked the
sufficient knowledge needed to establish priorities, quantify risks, and
make informed investment decisions, and thus predict costs.
Most notably, NASA's basic cost-estimating processes-an important tool for
managing programs-lack the discipline needed to ensure that program
estimates are reasonable. Specifically, GAO found that none of the 10 NASA
programs that GAO reviewed in detail met all of GAO's cost-estimating
criteria, which are based on criteria developed by Carnegie Mellon
University's Software Engineering Institute. Moreover, none of the 10
programs fully met certain key criteria-including clearly defining the
program's life cycle to establish program commitment and manage program
costs, as required by NASA. In addition, only three programs provided a
breakdown of the work to be performed. Without this knowledge, the
programs' estimated costs could be understated and thereby subject to
underfunding and cost overruns, putting programs at risk of being reduced
in scope or requiring additional funding to meet their objectives.
Finally, only two programs have a process in place for measuring cost and
performance to identify risks.
NASA has limited ability to collect the program cost and schedule data
needed to meet basic cost-estimating criteria. For example, as GAO has
previously reported, NASA does not have a system to capture reliable
financial and performance data-key to using effectively the
cost-estimating tools that NASA officials state that programs employ.
Further, without adequate financial and nonfinancial data, programs cannot
easily track an acquisition's progress and assess whether the program can
meet its cost and schedule goals before it incurs significant cost and
schedule overruns. NASA identified other barriers, including limited
cost-estimating staff. According to NASA officials, several initiatives
are under way to remove such obstacles and improve the agency's
cost-estimating practices.
www.gao.gov/cgi-bin/getrpt?GAO-04-642.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Allen Li at (202) 512-4841
or [email protected].
Contents
Letter
Results in Brief
Background
Development Cost Estimates Frequently Changed
Poor Estimating Processes and Methodologies Contributed to
Wide Variations in Baseline Cost Estimates NASA Has Begun to Address
Certain Barriers to Effective
Cost Estimating Conclusions Recommendations for Executive Action Agency
Comments and Our Evaluation
1 2 5 6
12
19 24 24 26
Appendixes
Appendix I: Appendix II:
Appendix III:
Appendix IV: Appendix V:
Appendix VI: Scope and Methodology 30
Assessments of 10 Programs Reviewed in Detail 32 Gravity Probe B 34 Mars
Exploration Rovers 36 Space Infrared Telescope Facility 38 Landsat-7 40
Aqua 42 Aura 44 Fluids and Combustion Facility 46 Hyper-X Program 48
Checkout and Launch Control System 50 Cockpit Avionics Upgrade 52
Summary Descriptions of the 17 Additional Programs 54 Space Science
Enterprise 54 Earth Science Enterprise 56 Space Flight Enterprise 58
Description of Earned Value Management 60
Comments from the National Aeronautics and Space Administration 64
GAO Contact and Staff Acknowledgments 71
Contents
Tables Table 1: Table 2: Table 3: Table 4: Table 5:
Initial and Current Baseline Development Cost Estimates
and Life-Cycle Cost Estimates for 27 NASA Programs 7
Summary of Criteria Used to Assess 10 NASA Programs
Reviewed 14
Summary of Extent 10 NASA Programs Met Assessment
Criteria 16
Summary of the Number of Programs That Met, Partially
Met, or Did Not Meet Criterion 33
Thirty-Two Criteria for Evaluating the Quality of
Management Systems 61
Figure Figure 1: History of Rebaselinings of 10 Programs' Development Cost
Estimates
Contents
Abbreviations
AHMS Phase 1 Advanced Health Management System Phase I
ATP Alternate Turbopump Program
CAIV cost as an independent variable
CALIPSO Cloud-Aerosol Lidar and Infrared Pathfinder
Satellite Observations CARD cost analysis requirements description CAU
Cockpit Avionics Upgrade CLCS Checkout and Launch Control System CPR cost
performance report CRV Crew Return Vehicle DOD Department of Defense EOS
Earth Observing System EVM earned value management FCF Fluids and
Combustion Facility GP-B Gravity Probe B IFMP Integrated Financial
Management Program INTEGRAL International Gamma-Ray Astrophysics
Laboratory IPAO Independent Program Assessment Office MERs Mars
Exploration Rovers MESSENGER Mercury Surface, Space Environment,
Geochemistry, and Ranging NASA National Aeronautics and Space
Administration NMP-EO-1 New Millennium Program Earth Observing-1 OMB
Office of Management and Budget PMA President's Management Agenda SEER
System Evaluation and Estimation of Resources SEI Software Engineering
Institute SIRTF Space Infrared Telescope Facility SOFIA Stratospheric
Observatory for Infrared Astronomy STEREO Solar Terrestrial Relations
Observatory TDRS Tracking and Data Relay Satellite Replenishment TIMED
Thermosphere, Ionosphere, Mesosphere Energetics,
and Dynamics WBS work breakdown structure
Contents
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States General Accounting Office Washington, D.C. 20548
May 28, 2004
The Honorable Sherwood L. Boehlert
Chairman
The Honorable Bart Gordon
Ranking Minority Member
Committee on Science
House of Representatives
The lack of reliable financial and performance information has posed
significant challenges to the National Aeronautics and Space
Administration's (NASA) ability to manage its largest and most costly
programs effectively. For nearly 15 years, NASA contract management has
been on GAO's high-risk list-due in part to NASA's inability to collect,
maintain, and report the full cost of its programs and projects.1 Without
such information, NASA has consistently developed unrealistic cost and
schedule estimates, which, at least in part, are reflected in the cost
growth
and schedule increases in many of its programs.
The demanding scientific and technical expectations inherent in NASA's
mission create even greater challenges for the agency to control program
costs-especially if meeting those expectations requires NASA to
reallocate funding from existing programs to support new efforts. Because
cost growth has been a persistent problem on a number of NASA programs,
you asked us to (1) identify initial cost estimates in selected NASA
programs and any changes in those cost estimates, (2) assess NASA's cost
estimating processes and methodologies, and (3) describe any barriers that
make it difficult for NASA to improve its cost-estimating processes.
Our review focused on 27 of 68 NASA programs in the development phase
as of April 2003 or that completed development in fiscal year 2001 or
2002.
To assess NASA's cost-estimating processes and methodologies, we
conducted a more in-depth review of 10 of the 27 programs, which
generally had the highest development cost estimate within five of NASA's
1 U.S. General Accounting Office, Major Management Challenges and Program
Risks: National Aeronautics and Space Administration, GAO-03-114
(Washington, D.C.: Jan. 2003).
seven Enterprises. 2 Our work was conducted between February 2003 and
March 2004 in accordance with generally accepted government auditing
standards. For a complete description of our scope and methodology, see
appendix I.
Results in Brief Many of the NASA programs that we reviewed cost more and
took longer than was proposed at the time of congressional approval.3
Several factors continue to put NASA projects at risk of increased cost
and schedule delays. Most notably, NASA lacks the basic cost-estimating
processes needed to establish priorities, quantify risks, and make
informed investment decisions for its programs. Further, NASA has limited
ability to collect, analyze, and use program cost and schedule data to
identify and mitigate impediments to program success.
Current baseline development cost estimates for the 27 programs we
reviewed varied considerably from the programs' initial baseline
estimates.4 More than half of the programs' development cost estimates
increased, and for some programs, this increase was significant-as much as
94 percent. In addition, the baseline development estimates for each of 10
programs that we reviewed in detail were rebaselined-some as many as four
times. For 7 of the 10 programs, the new baseline development estimate was
an increase over the previous baseline estimate. Although NASA cited
specific reasons for the cost growth and the recalculated baselines, such
as technical problems and funding shortages, the variability in the cost
estimates and the rebaselinings indicate that the programs lacked
sufficient knowledge needed to make informed acquisition decisions.
2 NASA's Enterprises, listed in the background section of this report,
function as primary business areas for implementing NASA's mission. Each
Enterprise has its own strategic goals, objectives, and implementation
strategies.
3 According to NASA, congressional approval occurs when the Congress
appropriates design and development funds for the program.
4 NASA defines baseline as the technical performance and content,
technology application, schedule milestones, and budget (including
contingency and allowance for program adjustment) that are documented in
the approved program and project plans. Current baseline development cost
estimates are as of April 2003.
Although an important tool for managing programs, NASA's cost-estimating
processes lack the discipline needed to ensure that program estimates are
reasonable. Specifically, we found that none of the 10 NASA programs that
we reviewed in detail met all of the criteria that we selected to assess
NASA's cost-estimating processes. Moreover, none of the 10 programs met
certain key criteria-such as clearly defining the program's life cycle.
NASA procedures and guidelines require programs and projects to be managed
on the basis of life-cycle cost-which the agency clearly defines-and that
such cost be developed to establish the program's commitment.5 In
addition, only three programs provided a complete breakdown of the work to
be performed. Without knowing the full life cycle and the work to be
performed, the programs' estimated costs could be understated and thereby
subject to underfunding and cost overruns, thus putting programs at risk
of being reduced in scope or requiring additional funding to meet their
objectives. Finally, only two programs had a process in place for
measuring cost and performance to identify these potential risks and take
action to avoid them.
NASA faces a number of barriers in meeting the cost-estimating criteria
that we used to assess the 10 programs. For example, although NASA
officials noted that programs are using cost-estimating tools, NASA
generally lacks the data needed to employ these tools effectively. For
more than a decade, we have reported that, despite repeated efforts, NASA
has failed to develop a system to capture reliable financial and
performance information. Most recently, we reported that the agency's
current effort to implement a modern integrated financial management
system will not, as it is being implemented, routinely provide program
managers and other key stakeholders and decision makers-including the
Congress-with the financial information needed to measure program
performance and ensure
5 NASA defines life-cycle cost as the total of the direct, indirect,
recurring, nonrecurring, and other related expenses incurred, or estimated
to be incurred, in the design, development, verification, production,
operation, maintenance, support, and retirement of a system over its
planned life.
accountability.6 According to NASA officials, nonfinancial data, such as
data on technology readiness levels, have also been difficult for the NASA
cost-estimating community to obtain. Without adequate financial and
nonfinancial data, programs cannot easily track an acquisition's progress
and assess whether the program can meet its cost and schedule goals before
the program incurs significant cost and schedule overruns. NASA identified
other barriers, including limited cost-estimating staff. According to NASA
officials, there are several initiatives under way to remove such
obstacles and improve the agency's cost-estimating practices.
We are recommending that NASA take a number of actions to better ensure
that the agency's initiatives result in sound cost-estimating practices
and are integrated into the project approval process. Specifically, we are
recommending that NASA develop an integrated plan that includes specific
actions that ensure that guidance is established on rebaselining and that
programs have a well-defined process in place to measure cost and
performance and identify potential risks. We are also recommending that
NASA establish a framework for developing life-cycle cost estimates.
In its comments on a draft of this report, NASA stated that it concurred
with our recommendations. NASA believes that it has already made progress
toward achieving many of the improvements intended by the recommendations
by developing new guidance, implementing management controls, and
instituting additional levels of project oversight. These reforms to
NASA's project development and implementation processes are, in our view,
positive steps in addressing some of the problems discussed in our report.
However, planned improvements must be integrated and enforced on an agency
wide basis; our recommendations are in line with that thrust. NASA's
detailed comments are included as appendix V.
6 U.S. General Accounting Office, Business Modernization: NASA's
Challenges
in Managing Its Integrated Financial Management Program, GAO-04-255
(Washington, D.C.: Nov. 21, 2003); Business Modernization: Disciplined
Processes Needed
to Better Manage NASA's Integrated Financial Management Program,
GAO-04-118
(Washington, D.C.: Nov. 21, 2003); Business Modernization: NASA's
Integrated
Financial Management Program Does Not Fully Address Agency's External
Reporting
Issues, GAO-04-151 (Washington, D.C.: Nov. 21, 2003); and Information
Technology:
Architecture Needed to Guide NASA's Financial Management Modernization,
GAO-04-43
(Washington, D.C.: Nov. 21, 2003).
Background NASA's programs encompass a broad range of complex and
technical activities-from investigating the composition and resources of
Mars to providing satellite and aircraft observations of Earth for
scientific and weather forecasting. NASA currently funds more than 100
programs and projects in various phases of execution in 7 strategic
Enterprises: Space Science, Earth Science, Biological and Physical
Research, Aeronautics, Space Flight, Education, and Exploration Systems.
Two NASA offices have key responsibilities in ensuring the effective
execution of these programs: the Office of the Chief Financial Officer,
which is responsible for providing oversight and financial management of
agency resources and establishing related policy guidance, and the Office
of Chief Engineer, which is responsible for ensuring development efforts
and mission operations are planned and conducted using sound engineering
practices.
More than two-thirds of NASA's work force is made up of contractors and
grantees, and 90 percent-or roughly $13 billion-of NASA's annual budget is
spent on work performed by its contractors. Since 1990, we have identified
NASA's contract management as a high-risk area. This assessment has been
based in part on our repeated finding that NASA does not have good
cost-estimating processes or the financial information needed to develop
good cost estimates for its programs, making it difficult for NASA to
oversee its contracts and control costs. For example, in July 2002, we
reported that an independent task force convened to assess the management
of the International Space Station concluded that the program's fiscal
year 2002 through fiscal 2006 budget was not credible because of
weaknesses in its cost-estimating processes.7 The task force pointed out
that these problems occurred because NASA had not instituted or had
ignored many of the program's control and contract oversight
procedures-such as preparing a full life-cycle cost estimate-that should
have alerted the agency to the growing cost problem and the need for
mitigating actions. According to the cost analysis team that supported the
task force, NASA's focus on staying within annual budgets instead of
managing total program costs was perhaps the single greatest factor in the
program's cost growth.
NASA's unreliable cost estimates have significant implications for
potential future endeavors, such as those outlined by the President in
January of this
7 U.S. General Accounting Office, Space Station: Actions Under Way to
Manage Cost, but Significant Challenges Remain, GAO-02-735 (Washington,
D.C.: July 17, 2002).
year. Specifically, the President called for a shift in NASA's long-term
focus, envisioning that NASA will retire the shuttle program as soon as
assembly of the International Space Station is completed, planned for the
end of the decade; develop a new crew exploration vehicle as well as
launch human missions to the moon between 2015 and 2020, and build a
permanent lunar base as a stepping stone for more ambitious missions. To
achieve these goals, the President proposed spending $12 billion over the
next 5 years- about $1 billion of which would come from an increase in
NASA's budget, currently $15.4 billion-with the remaining $11 billion
being reallocated from existing NASA programs.
Developing reliable cost estimates has been difficult for agencies across
the federal government. The need for reliable cost estimates is at the
heart of two of the five-governmentwide initiatives in the 2002
President's Management Agenda (PMA); the two are "improved financial
performance" and "budget and performance integration."8 These initiatives
are aimed at ensuring that federal financial systems produce accurate and
timely information to support operating, budget, and policy decisions and
that budgets are performance-based. As part of these initiatives, the
President calls for changes to the budget process to better measure the
real cost and performance of programs. According to the PMA, accomplishing
all of the crosscutting initiatives will matter little without the
integration of agency budgets with performance.
Development Cost Estimates Frequently Changed
As of April 2003, the baseline development cost estimates for the programs
we reviewed varied considerably from the programs' initial baseline
estimates. More than half of the programs' development cost estimates
increased, and for some programs, this increase was significant. The
baseline development cost estimates for each of the 10 programs we
reviewed in detail were rebaselined-that is, recalculated to reflect new
costs, time frames, or resources associated with program changes in
program objectives, deliverables, or scope and plans. Although NASA
provided specific reasons for the increased cost estimates and
rebaselinings-such as delays in the development or delivery of key system
components and funding shortages-it does not have guidance for determining
when rebaselinings are justified. Such criteria are important to
instilling discipline in the cost-estimating process.
8 The other three initiatives are strategic human capital management,
competitive sourcing, and expanded electronic government.
Most of the 27 programs we reviewed experienced a change in their
development costs estimates. While 8 of the 27 programs experienced slight
decreases in their development cost estimates, 17 experienced cost
growth-as much as almost 94 percent. The remaining two programs had no
change. Ten of the 17 programs' cost growth was greater than 25 percent.
Table 1 shows the development cost estimate changes from the initial
baseline to the baseline as of April 2003 and the life-cycle cost estimate
for each of the 27 programs. The 10 programs that we reviewed in detail
are shaded and italicized. (See app. II for assessments of the 10 programs
and app. III for descriptions of the remaining 17 programs.)
Table 1: Initial and Current Baseline Development Cost Estimates and Life-Cycle
Cost Estimates for 27 NASA Programs
Then-year dollars in millions
Baseline development cost estimate
Current (as of April Percent Life-cycle cost estimate Program, by
Enterprise Initial 2003)a change (as of April 2003)a
Space Science Earth Science
Strastospheric Observatory for Infrared
Astronomy (SOFIA) 234.8 373.0 58.9 604.5
Solar Terrestrial Relations Observatory
(STEREO) 404.7 302.1 (25.4) 423.0
Mercury Surface, Space Environment,
Geochemistry and Ranging (MESSENGER) 325 235.1 (27.7) 337.7
Herschel 103.7 72.7 (29.9) 277.6
Thermosphere, Ionosphere, Mesosphere
Energetics and Dynamics (TIMED) 176.8 176.2 (0) 253.5
Solar-B 99.3 80.4 (19.0) 146.4
Rosetta 28.4 40.1 41.2 106.0
International Gamma-Ray Astrophysics
Laboratory (INTEGRAL) 8.2 11.9 45.1 51.2
Terra 1,309.1 1,393.2 6.4 1,451.7
(Continued From Previous Page)
Then-year dollars in millions
Baseline development cost estimate Biological and Physical Research
Current
Percent Life-cycle cost
(as of April estimate
Program, by Enterprise Initial 2003)a change (as of April
2003)a
New Millennium Program Earth
Observing-1
(NMP-EO-1) $111.7 $176.4 57.9 $192.5
SeaWinds 130.2 148.8 14.3 160.1
Cloud-Aerosol Lidar and
Infrared Pathfinder
Satellite Observations 98.0 133.9 36.6 150.9
(CALIPSO)
Jason-1 77.5 87.8 13.3 127.8
Aeronautics
Space Flight
Alternate Turbopump Program (ATP) 1,056.0 764.0 (27.7) 982.0
Advanced Health Management System Phase I
(AHMS Phase 1) 55.0 55.0 (0)
Tracking and Data Relay Satellite Replenishment
(TDRS) 937.0 518.1 (44.7) d
X-38 Crew Return Vehicle (CRV) 792.0 1,025.0 29.4 e
Source: NASA.
Note: The draft NASA Cost Estimating Handbook 2002 defines then-year
dollars as dollars that are escalated into the time period of performance
of a contract. It further states that then-year dollars are sometimes
referred to as escalated costs, inflated costs, or real-year dollars. NASA
normally uses the term-real-year dollars.
aIncludes launch vehicle cost where applicable.
bSIRTF was renamed the Spitzer Space Telescope in December 2003.
cBecause Hyper-X is classified as a test program, there is no life-cycle
cost estimate.
dA life-cycle cost estimate was not developed for the Tracking and Data
Relay Satellite Replenishment program because it is currently in pre-phase
A (conceptual definition). According to a NASA official, a life-cycle cost
estimate will be determined before it enters phase C/D (design,
development, test, and evaluation).
eA life-cycle cost estimate was not developed for the X-38 Crew Return
Vehicle program because the program was cancelled in 2003, and the
program's contracts remained undefinitized at termination- that is, the
final price or estimated cost and fee were not negotiated and mutually
agreed to by NASA and the contractor.
fA life-cycle cost estimate was not developed for the CLCS program because
it was canceled due to excessive cost growth.
The development cost estimates for each of the 10 programs that we
reviewed in detail have been rebaselined-for some programs, as many as
four times-and for 7 of the 10 programs, the cost estimate increased each
time it was rebaselined (see fig. 1).
Figure 1: History of Rebaselinings of 10 Programs' Development Cost
Estimates Then-year dollars in millions
Source: NASA.
aSIRTF was renamed the Spitzer Space Telescope in December 2003.
bThe baseline development estimates for the Aura and Aqua projects were
rebaselined once as a result of a restructuring of the overall Earth
Observing System (EOS) program in 1995 to address affordability issues.
Before EOS' restructuring, the baseline was $524 million for Aura and $1.2
billion for Aqua. However, according to NASA officials, both the Congress
and NASA recognize the revised baseline estimates as the initial baseline
estimates.
cLandsat-7's initial baseline development estimate was established by the
Department of Defense (DOD), which originally had responsibility for
managing the program. A 1994 Presidential Directive later reassigned the
program to a joint NASA, National Oceanic and Atmospheric Administration,
and U.S. Geological Survey program, with NASA having responsibility for
the development and launch of the satellite and development of the ground
system. Landsat-7 also became a part of the EOS program. In 1995, NASA
established a revised initial baseline development estimate for Landsat-7,
which according to NASA officials is recognized by the Congress and NASA
as the initial baseline development estimate. DOD's initial baseline
estimate was not available.
dCLCS was rebaselined twice, but the second rebaselined estimate for CLCS
was not established because NASA terminated the program due to the
program's excessive cost growth.
For the 10 programs we reviewed in detail, NASA cited specific reasons for
changes in the baseline development cost estimates and the recalculated
baselines-many of which were related to technical problems and subsequent
delays in the development or delivery of key system components, and
insufficient funding and reserves, as illustrated in the following
examples:
o Technical problems in the MERs program required a significant redesign
of components and the development of a new landing system. Two of MERs'
three rebaselinings were also the result of inadequate reserves. According
to NASA officials, without the rebaselinings, the development cost "to
go"9 would have drained the program's reserves.
o The increase in CLCS's development cost estimate and rebaselining was
the result of poorly defined requirements and design, software integration
problems, and fundamental changes in the project's management structure
and contractors' approach to the work. The project, which experienced an
almost 94 percent increase in its baseline development cost estimate, was
ultimately terminated.
o The GP-B program-which was rebaselined four times-experienced
significant schedule slippages due to repeated technical problems,
including failures in the probe's heat exchanger, the need for additional
testing, payload electronics delays, and thermal vacuum test failures.
9 According to a NASA project manager, "to go" means from this point
forward to completion of the project, given the current status of the
project and the resources available to complete it.
o Schedule slippages in the SIRTF program-which contributed to increases
in the program's baseline development cost estimate and four rebaselinings
of the estimate-were caused by delays in the delivery of components,
flight software, and the mission operation system as well as launch delays
that resulted from a handling accident involving a global positioning
system payload and concerns of delamination on the launch vehicle's solid
rocket motors.
o Changes in development cost estimates for the CAU program were
primarily the result of the program's expanded scope, which occurred in
October 2002, to produce modification kits that would allow the CAU
upgrade to be installed into the orbiters.
o The Hyper-X program experienced three rebaselinings, and according to
the project manager, the program will be rebaselined again in the near
future. The rebaselinings were due to schedule slippages resulting from
the need to fund an investigation of the problems experienced in the first
Mach 7 flight vehicle-which was destroyed in flight-and related corrective
actions to the second Mach 7 flight.10
Revised contract requirements, funding changes, or the realization that
program goals are not achievable may require a formal rebaselining.
However, NASA has not defined or provided guidance or restrictions on
rebaselining to ensure that programs consistently and appropriately apply
rebaselinings and do not adjust their baseline cost estimates whenever the
estimates become unmanageable. Further, NASA lacks a process for
systematically identifying and assessing programs that are not achieving
their cost, schedule, and performance goals. Such a process has been
employed by the Department of Defense (DOD), which also relies heavily on
contractors to deliver complex, cutting-edge technologies to meet its
mission. Specifically, DOD must report to the Congress programs that incur
a cost growth of 15 percent or more in the program baseline. Moreover, DOD
must justify the continuation of acquisition programs that incur a cost
growth of 25 percent or more in the program baseline by certifying that
specific criteria have been met-including that the new cost estimates are
reasonable.11 Under such a process, 5 of the 10 programs that we reviewed
10 The second Hyper-X flight vehicle flew successfully at Mach 7 speed in
March 2004 (see app. II).
11 10 U.S.C. 2433.
in detail would have been required to report to the Congress, and 4 of the
5 programs would have had to certify that their new cost estimates were
reasonable.
Poor Estimating Processes and Methodologies Contributed to Wide Variations
in Baseline Cost Estimates
NASA has yet to implement a well-defined process for estimating the cost
of its programs-a weakness we and NASA's Inspector General have repeatedly
reported.12 Recognizing the need for such a process, NASA developed a
cost-estimating handbook in 2002-the first such guidance provided to its
cost-estimating community and program and project managers.13 Despite this
effort, the programs we reviewed failed to follow key cost-estimating
processes, including developing and documenting full life-cycle cost
estimates, summarizing estimates according to the current breakdown of
work to be performed, conducting an uncertainty analysis, performing an
independent review of contractors' cost estimates, and later using earned
value management (EVM) to assess progress.14
12 See, for example, GAO-04-118; GAO-04-255; GAO-03-114; U.S. General
Accounting Office, Space Station: Actions Under Way to Manage Cost, but
Significant Challenges Remain, GAO-02-735 (Washington, D.C.: July 17,
2002); NASA Program Costs: Space Missions Require Substantially More
Funding Than Initially Estimated, GAO/NSIAD-93-97 (Washington, D.C.: Dec.
31, 1992); and NASA Office of Inspector General, Final Management Letter
on Failures in Cost Estimating and Risk Management Weaknesses in Prior
Space Launch Initiative Assignment Numbers A-01-049-01and A-01-049-02,
IG-03-023 (Washington, D.C.: Sept. 29, 2003).
13 The cost-estimating handbook is in draft form, but NASA made it
available for official use by its cost-estimating community and program
managers. NASA expected to complete the handbook by May 2004.
14 EVM compares the actual work performed at certain stages of a job to
its actual costs- rather than comparing budgeted and actual costs, the
traditional management approach to assessing progress. By measuring the
value of work that has been completed at certain stages in a job, EVM can
alert program managers, contractors, and administrators of potential cost
overruns and schedule delays before they occur and of problems that need
correcting before they worsen. For a more detailed discussion of EVM, see
appendix IV.
Reflecting Office of Management and Budget (OMB) guidance and best
practices of government and industry leaders, NASA requires that full
life-cycle cost estimates be prepared using full cost accounting,15 that
estimates be summarized according to the current breakdown of work to be
performed, and that major changes be tracked to the life-cycle cost. In
its draft cost-estimating handbook, NASA lists a number steps that are
integral to preparing a reliable life-cycle cost estimate, including
preparing or obtaining a cost analysis requirements description (CARD),16
developing ground rules and assumptions, and developing cost range and
risk assessments.
Carnegie Mellon University's Software Engineering Institute (SEI) 17
echoes the need for reliable cost-estimating processes in managing
software implementations-identifying tasks to be estimated, mapping the
estimates to the breakdown of work to be performed, and identifying and
explaining assumptions are among SEI's requisites for producing reliable
cost estimates. To evaluate the cost-estimating processes of the 10 NASA
programs that we reviewed in detail, we selected 14 criteria based on SEI
checklists (see table 2).18 Many of these criteria are included in NASA's
cost-estimating guidance.
15 The full cost of a project is the sum of all direct costs, service
costs, and general administrative costs. Full cost accounting ties all
NASA agency costs (including civil service personnel costs) to major
projects.
16 A CARD provides a system technical description and programmatic
information to create a common baseline used by the project team to
develop estimates.
17 SEI is a government-funded research organization that is widely
considered an authority on software implementation.
18 SEI developed checklists to help evaluate software costs and schedule.
However, SEI states that these checklists are equally applicable to
hardware and systems engineering projects.
Table 2: Summary of Criteria Used to Assess 10 NASA Programs Reviewed
Criterion Purpose/Significance
The objectives of the estimate are stated The objectives of the program
must be clearly stated in a concise document for the
in writing. cost estimator to use to develop the cost estimate. NASA
guidance states that NASA programs and projects are to be defined as
activities that have defined objectives along with goals and requirements.
The life cycle to which the estimate applies is The life cycle must be
clearly defined to ensure that the full cost of the program-that
clearly defined. is, all direct and indirect costs for planning,
procurement, operations and maintenance, and disposal-are captured. The
draft NASA cost-estimating handbook states that a life-cycle cost estimate
provides "an exhaustive accounting of all resources necessary to develop,
deploy or field, operate, maintain, and dispose of a system over its
lifetime." The handbook defines life cycle as the program's or project's
"total life, beginning with mission feasibility and extending through
operation and disposal or conclusion of the system or program."
The task has been appropriately sized. This criteria asks if the
appropriate metric was used in the development of the estimate, such as
the size of a software product with expected amount of reuse.
The estimated cost and schedule are consistent In other words, estimates
have been validated by relating them back to demonstrated
with demonstrated accomplishments on other performance on completed
projects.
projects.
A written summary of parameter values and their This criterion refers to
the underlying cost-estimating methodology. If a parametric
rationales accompany the estimate. equation was used to generate the
estimate, then the parameters that feed the equation must be provided
along with an explanation of why they were chosen.
Assumptions have been identified and explained. The draft NASA draft
cost-estimating handbook states that assumptions are a critical step in
any estimate and should be clearly prominent in all documentation for the
estimate. Accurate assumptions can prevent inaccurate or misleading
estimates.
A structured process such as a template or This criterion refers to
whether or not the program has established a work breakdown format has
been used to ensure that key factors structure (WBS)-that is a structure
that organizes, defines, and graphically displays have not been
overlooked. the individual work units to be performed. NASA policy
guidance calls for breaking
down work into smaller units to facilitate cost- estimating and project
and contract management as well as to help ensure that all relevant costs
are captured. The guidance requires that a preliminary WBS be developed
during the formulation phase, and that a final WBS be generated following
contractor selection or approval to implement. The guidance further
requires that programs describe the overall WBS structure and the content
of each individual element of the WBS.
Uncertainties in parameter values have been Again this criterion refers to
the underlying cost-estimating methodology. For all major identified and
quantified. cost drivers, an uncertainty analysis should be performed to
assess the risk associated with the cost estimate.
If a dictated schedule has been imposed, an This criterion asks whether a
dictated schedule was imposed on the program, that is, estimate of the
normal schedule has been whether the program was forced to accelerate the
schedule in order to meet compared to the additional expenditures required
requirements. If this occurred, then the impacts to the cost estimate need
to be
a
to meet the dictated schedule. calculated and provided.
If more than one cost model or estimating This criterion checks to ensure
that the primary methodology or cost model approach has been used, any
differences in results are consistent with any secondary methodology (for
example, cross checks) results have been analyzed and explained.
performed.
(Continued From Previous Page)
Criterion Purpose/Significance
Estimators independent of the performing NASA policy guidance states,
"when a project under a program has an estimated organization concurred
with the reasonableness NASA life-cycle cost greater than $150 million, an
independent life-cycle cost of the parameter values and estimating
analysis is required during formulation in conjunction with initiating the
preliminary methodology. design."
Estimates are current. Estimates should be updated whenever changes to
requirements affect cost or schedule. NASA policy guidance requires that
the life-cycle cost estimate be updated prior to each budget submission.
The results of the estimate have been integrated NASA policy guidance
requires that a life-cycle cost be developed to establish a
with project planning and tracking. program/project commitment, assessed
at major reviews, and updated for each budget submission and should use
currently available full cost initiative guidance.
Earned value reporting has been used to NASA policy guidance requires
program and project managers to "ensure that EVM
manage the program. provisions and requirements are included in requests
for proposals and contracts and ensure that an effective surveillance
program is in place to provide assurance that EVM data are valid and that
the contractor's integrated management system remains in compliance with
the EVM criteria." The guidance further requires each program and project
to periodically generate estimates at completion, perform cost and
schedule variance analyses based upon pre-established thresholds, and
prepare corrective action plans where necessary.
Sources: NASA and SEI.
aDoes not apply to all programs.
Despite NASA requirements and OMB and SEI guidance, few of the 10 programs
that we reviewed in detail met even a third of these criteria; only one
met half. Further, none of the programs fully met certain key criteria.
For example, none provided a complete life cycle with definitions or a
complete description of the methodology used to generate the complete cost
estimate, such as data sources and uncertainties. According to the draft
NASA cost-estimating handbook, a reliable life-cycle cost estimate is
critical to making realistic decisions about developing or producing a
system and to determining the appropriate scope or size of a program. NASA
guidance also calls for breaking down the work to be performed into
smaller units to facilitate cost estimating and program and contract
management and to help ensure relevant costs are not omitted. However,
only 3 of the 10 programs provided a complete breakdown of the work to be
performed. Table 3 shows for each program the applicable criteria that
were met, partially met, or not met.19 (See app. II for a program by
program assessment.)
19 If a program provided substantiating evidence for a criterion, we
determined that the program "fully met" the criterion. If partial evidence
was provided for a criterion, we determined the program "partially met"
the criterion. If no evidence was found, then we determined that the
criterion was "not met."
Table 3: Summary of Extent 10 NASA Programs Met Assessment Criteria
Biological
and physical
research Aeronautics Space flight
Space science Earth science
Criteria for
cost estimating GP-B MERs SIRTF Landsat-7 Aqua Aura FCF Hyper-X CLCS CAU
Key: M = Met, P = Partially met, NM = Not met
(Continued From Previous Page)
Biological
and physical
research Aeronautics Space flight
Space science Earth science
Criteria for
cost estimating GP-B MERs SIRTF Landsat-7 Aqua Aura FCF Hyper-X CLCS CAU
Sources: NASA (data), SEI (criteria), GAO (analysis).
Key: M = Met, P = Partially met, NM = Not met
Failing to meet these criteria puts programs at certain risk. For example,
underestimating a program's full life-cycle costs creates the risk that a
program could be underfunded and subject to major cost overruns, which
would ultimately result in the program being reduced in scope or
additional funding being requested and appropriated to ensure the program
meets its objectives. Conversely, overestimating life-cycle costs creates
the risk that a program will be deemed unaffordable and would, therefore,
go unfunded. Without a complete WBS, NASA programs cannot ensure that the
life-cycle cost estimates have captured all relevant costs, which again
can result in underfunding and cost overruns. Further, inconsistent WBS
estimates across programs can create problems of double counting or,
worse, underestimating costs when using historical program costs as a
basis for projecting future costs on similar programs.
Despite the uncertainty inherent in estimating the cost of emerging
technologies, all of the 10 programs we reviewed also failed to conduct an
uncertainty analysis to assess risks associated with the cost estimates.
Instead, the programs expressed their cost estimates as point values-which
implies certainty-not as ranges or numbers with confidence levels.20
Performing an uncertainty analysis, such as a Monte
20 For example, a cost estimate of $1 million could be presented either as
a range of $900,000 to $1.1 million or as $1 million with a confidence
interval of 90 percent, indicating that there is a 10-percent chance that
the cost will exceed the estimate.
Carlo simulation,21 quantifies the amount of cost risk within a program.
Only by quantifying the cost risk can management make informed decisions
about risk mitigation strategies. Quantifying cost risks also provides a
benchmark against which future progress can be measured. Without this
knowledge, NASA may have little specific basis to determine adequate
financial reserves, schedule margins, and technical performance margins to
provide managers the flexibility needed to address programmatic,
technical, cost, and schedule risks, as required by NASA policy.
Seven of the 10 programs also failed to have an independent review of
contractors' cost estimates-as required by NASA. Instead, programs
established their budgets based on contractor proposals-particularly
problematic since many contractors could bid low in order to win the
contract. To ensure contractor costs are realistic, NASA procedures and
guidelines specifically require programs to ensure that independent
reviews are conducted and that these reviews address project life-cycle
costs, risk management plans, as well as technical issues. Without such
reviews, NASA decision makers lacked the benchmarks needed to assess the
reasonableness of the contractors' proposed costs, limiting NASA's ability
to make sound investment decisions and accurately assess contractor
performance.
Finally, only two programs used EVM-an approach used by DOD and leading
companies to provide meaningful assessments of a program's progress by
comparing the value of work performed to its costs, rather than the
traditional management approach of comparing budgeted and actual costs,
which can provide a distorted view of a program's progress. (For a
detailed discussion of EVM, see app. IV.) By using the value of completed
work as a basis for estimating the cost and time needed to complete the
program, EVM can alert program managers to potential problems early in the
program. NASA requires that EVM be used on all significant contracts-that
is, research and development contracts with a total anticipated final
value of $70 million or more, and production
21 A Monte Carlo simulation randomly generates values for uncertain
variables over and over to simulate a model. Without the aid of
simulation, a model will only reveal a single outcome, generally the most
likely or average scenario, but after hundreds or thousands of trials, one
can view the statistics of the results and the certainty of any outcome.
contracts with a total anticipated final value of $300 million or more-
which includes all of the 10 programs we reviewed in detail.22 Although
the program managers for all 10 programs stated that EVM was used in their
projects, only two programs provided cost performance reports, indicating
a true EVM process was in place. The remaining eight programs relied on
NASA Form 533, which captures planned and actual obligations and
expenditures-not the value of the work performed.23 Without a true EVM
process, programs cannot readily determine if a program is at risk of cost
and schedule overruns until it is too late to make programmatic changes to
avoid these risks.
NASA Has Begun to There are several impediments that NASA needs to
overcome to implement
effective cost-estimating practices. These include the lack of
reliableAddress Certain financial data and other performance information;
lack of trained EVM Barriers to Effective staff, data analysis tools, and
incentive for supporting and implementing Cost Estimating EVM; and
ineffective use of cost analysts. NASA has initiated several
measures to begin addressing some of these impediments.
Utility of Cost-Estimating Tools Depends on the Reliability of NASA's
Financial and Performance Data
According to NASA officials, state-of-the-art cost-estimating tools have
been funded and implemented. For example, NASA officials told us that
commercial-off-the-shelf models have been used to estimate hardware and
software acquisition costs and quantify the level of uncertainty
surrounding cost estimates. However, these cost-estimating tools are only
as good as the data they rely on to develop the estimates. For more than a
decade, we have reported that NASA has failed to develop a system to
capture reliable financial and performance information, posing significant
challenges to NASA's ability to estimate and control program costs. Over
the past year alone, we issued numerous reports on NASA's Integrated
Financial Management Program (IFMP)-the agency's third and most recent
effort to implement a modern, integrated financial management system.
Specifically, we found that IFMP-which is under the responsibility of the
Program Executive Officer for IFMP-will not, as it is being implemented,
routinely
22 See Earned Value Management, NASA Policy Directive 9501.3A (Aug. 3,
2002) and Earned Value Management Implementation on NASA Contracts, NASA
Procedural Requirements 9501.3 (Nov. 24, 2002).
23 Form 533 captures financial information that is used as basis for the
financial management and budget activities within projects and NASA-wide.
provide program managers and other key stakeholders and decision
makers-including the Congress-with the financial related information
needed to measure program performance and ensure accountability. For
example, the core financial module (considered the backbone of the system)
does not appropriately capture property, plant, and equipment, as well as
material in its general ledger at the transaction level-which is needed to
provide independent control over these assets. In addition, NASA
implemented the system before it had the capability to capture the full
costs of its programs and projects. According to headquarters officials,
collecting nonfinancial data crucial to cost estimating-such as technology
readiness levels, parts counts, and team and management experience and
skill ratings-has also been difficult.
Use of EVM Has Been Undermined by a Lack of Trained Staff, Data Analysis
Tools, and Incentive
According to headquarters officials, agencywide EVM implementation efforts
began in 1996 and are recognized by NASA management as a key tool in
monitoring and measuring cost trends in higher risk project elements-a
tool that serves as an early warning of the need for cost-risk mitigation
actions to maintain control of program costs. These officials stated that
EVM has been applied to the International Space Station Program24 and with
varying levels of emphasis to other programs and projects at different
NASA centers.25 While all of the program managers for the 10 programs that
we reviewed in detail stated that they used EVM, only 2 of the programs
used a true EVM process.
NASA headquarters officials identified several challenges that have
affected the agency's ability to implement EVM effectively, including a
lack of staff and data analysis tools. According to officials, resource
constraints have prevented the agency from staffing many project offices
with appropriate personnel to fulfill all project functions. In addition,
there has been little or no priority to include a trained EVM analyst,
even if one were available. Headquarters officials also noted that EVM has
been hampered by the lack of a practical automated software data analysis
tool. Without such a tool, analyzing the contractors' EVM cost performance
reports, which contain significant amounts of data, became a cumbersome
undertaking that often resulted in incomplete and untimely analyses,
24 The International Space Station Program was not a part of our review.
25 NASA has nine centers located around the country and owns the Jet
Propulsion Laboratory, which is operated by the California Institute of
Technology.
providing little usefulness to inform management decisions. A lack of
incentive to support EVM has further undermined its use. Some project
managers whom we spoke with are skeptical about the benefits of EVM and
argue that it has failed to help them manage or control program costs.
According to NASA headquarters officials, during proposal and contract
negotiation phases, contractors have also suggested not using EVM as a way
to reduce contract costs. While EVM was included in most contracts for the
10 programs we reviewed in detail-as required by NASA policy-it was used
only in two programs as a cost-estimating tool. In general, EVM has been
viewed by NASA as a financial reporting tool. Consequently, there is
little incentive to use EVM because the data needed to report financial
activity is captured elsewhere, such as in Form 533.
Ineffective Use and Placement of Cost Analysts across the Agency's Cost
Activities also Hinders NASA's Efforts to Improve Its Cost-Estimating
Practices
NASA's efforts to improve its cost-estimating processes have also been
undermined by ineffective use of its limited number of cost-estimating
analysts. For example, headquarters officials state that as projects
entered the formulation phase, they have typically relied on program
control and budget specialists-not cost analysts-to provide the financial
services to manage projects. Yet budget specialists are generally
responsible for obligating and expending funding-not for conducting cost
analyses that underlie the budget or ensuring budgets are based on
reasonable cost estimates-and, therefore, tend to assume that the budget
is realistic. While NASA officials state that its cost-estimating staff is
too limited to be involved in day-to-day project execution activities,
they agreed that the cost analysts could be more effectively used
throughout the life cycle- particularly when projects are rebaselined and
independent cost estimates of project changes must be performed.
In some cases, cost analysts are not appropriately located in the
organization, which may compromise controls NASA has in place to ensure
reasonable cost estimates. For example, some cost analysts at NASA's
centers are located with senior systems engineers in systems management
organizations, while others are not. According to NASA officials, housing
the cost analysts with senior systems engineers has two key benefits.
First, the systems engineers generally conduct systems analyses to help
ensure that a program's requirements are properly established and that the
design and validity meet the requirements. Such analyses can greatly
inform the development of reasonable cost estimates. Second, the systems
engineering offices afford some measures of independence for cost
estimating, which, according to NASA cost-estimating guidance and
procedures, is important to the overall project management process.
However, NASA officials stated that several of its centers' cost analysts
are in the advocacy chain of command-not housed with senior systems
engineers. For example, one center's 15 cost analysts work in the center's
Office of the Chief Financial Officer-which is responsible for directing
the development and execution of the center's budget-not in the systems
management organization, which is independent from the rest of the center.
As a result, the costs analysts' estimates may not be adequately informed
by the systems engineers and may lack the objectivity required to ensure
that the criteria for independence have been met.
Efforts Under Way to Remove Some Barriers and Improve Cost Estimating
NASA has several initiatives under way to improve the agency's
cost-estimating processes. First, NASA has established a Cost Analysis
Division in the Office of the Comptroller to strategically manage analyses
related to directing and funding research, improving cost-estimating
processes and practices, and providing cost-estimating tools and training
throughout the agency. The division also provides, along with the
Independent Program Assessment Office (IPAO), the last independent cost
estimate of projects before the information is released externally. These
efforts are being coordinated through a steering committee composed of the
managers of the cost analysis organizations from each of the centers and
IPAO's deputy director.
NASA is revising the cost sections in its governing procedures and
guidelines and is finalizing its cost-estimating handbook to reflect these
changes.26 These documents will require the routine use of probabilistic
cost risk analysis, a CARD document, cost as an independent variable
(CAIV), and EVM. The CARD supports the project life-cycle cost estimate
and a congressionally required independent cost estimate. Agency officials
note that while there has been some use of CARD in the agency, its first
concentrated and successful use was in the 2001 to 2002 independent cost
estimate for the International Space Station program. According to
headquarters officials, NASA's revised guidance and finalized
cost-estimating handbook will provide direction and guidance for fully
implementing the use of CARDs for major development projects. Although
26 According to NASA officials, revisions of NASA's current governing
program and project guidance-NASA Procedures and Guidelines 7120.5B, NASA
Program and Project Management Processes and Requirements (Nov. 21,
2002)-is expected to be completed by August 2004, and the draft
cost-estimating handbook was expected to be finalized by May 2004.
NASA calls for CAIV to be used routinely and notes that CAIV demonstrates
a commitment to evolutionary acquisition, it has yet to provide guidance
on its implementation. NASA headquarters officials stated that guidance
relating to improvements in the collection of cost data is also being
reflected in its revised governing procedures and guidelines.
With respect to EVM, NASA headquarters officials described several efforts
under way to ensure agencywide implementation of true EVM. For example,
NASA recently revised its EVM policy directives to shift ownership of EVM
responsibilities from NASA's Chief Financial Officer to NASA's Chief
Engineer, to emphasize that EVM is to be considered a project management
tool rather than a financial management tool. NASA officials also noted
that the agency is working to inform managers of the performance
management capabilities available to them through EVM and to emphasize the
importance of providing adequate resources and management support to
ensure successful EVM implementation. Agencywide goals for EVM
implementation include promoting the effective use of EVM and providing
needed training and education for program and project staff. These efforts
and proposed initiatives should help resolve EVM utilization problems.
Finally, NASA officials told us that the agency is planning to hire
additional cost analysts to alleviate understaffing at all of its center
cost analysis offices. The agency envisions a total staff of about 100
cost analysts along with additional support contractors. NASA officials
also stated that it is necessary to ensure centers address the problem of
having cost analysts located in the advocacy chain of command, which could
affect five NASA centers.
Because NASA's initiatives have only recently been implemented or are
still in the drafting or planning stage, we cannot determine to what
degree these efforts will enable NASA to provide reasonable and defensible
cost estimates of its programs and projects.
Conclusions There are numerous scientific and technical challenges
inherent in the successful implementation of many NASA programs.
Nevertheless, the need to choose among competing alternatives within
limited budget resources makes it essential that the agency and the
Congress clearly understand the costs and uncertainties of programs
proposed for authorization and funding. Yet, NASA does not have the
disciplined cost-estimating process needed to make informed acquisition
decisions, nor does the agency have processes and tools for capturing,
monitoring, and managing program costs and schedules within an
implementation plan on a timely basis. This makes it difficult for senior
NASA officials, program and project managers, and other key stakeholders
to measure performance and initiate mitigation measures when needed. Taken
together, the lack of disciplined and established cost-estimating
processes and tools can cause program officials to restructure projects to
available resources rather than develop realistic cost estimates and
implementation plans for projects. As a result, programs may have to be
modified to accommodate emerging technical, cost, and schedule realities.
Ultimately, programs cost more, fail to meet their schedules, or deliver
less than originally envisioned. To help minimize project costs increases
and implementation delays identified in this report, NASA needs to instill
disciplined cost-estimating processes into its project development and
approval activities and to ensure such processes are integrated with its
implementation of an integrated financial management system. Without a
process that prevents programs from proceeding before they have
sufficiently demonstrated that key cost-estimating criteria have been met,
NASA programs will continue to be at risk of cost and schedule overruns.
Recommendations for Executive Action
Improvements to NASA's cost-estimating processes will partly depend on the
agency's ability to address recommendations that we made in November 2003
to help ensure NASA effectively implements a modern, integrated financial
management system.27 Notwithstanding the need to address those
recommendations, to better position NASA to ensure its recent initiatives
result in sound cost-estimating practices agencywide, we are making three
recommendations with minimum suggested courses of action. First, we are
recommending that the NASA Administrator direct the Program Executive
Officer for IFMP, the Chief Financial Officer, and
27 GAO-04-118, GAO-04-151, and GAO-04-43.
the Chief Engineer to develop an integrated plan for improving cost
estimating that, at a minimum, includes specific actions for ensuring that
o guidance is established on rebaselining and that rebaselining is
consistently applied to provide accountability among programs,
o true earned value management is used as an organizational management
tool to bring cost to the forefront in NASA's management decisionmaking
process,
o acquisition and earned value management policies and procedures are
enforced, and
o staff and support for cost-estimating and earned value analyses are
effectively used.
In addition, we recommend that the NASA Administrator direct the Chief
Financial Officer to establish a standard framework for developing
lifecycle cost estimates. At a minimum the framework should require each
program or project to
o base its cost estimates on a full life cycle for the program-including
all direct and indirect costs for operations and maintenance and disposal
as well as planning and procurement-and on a work breakdown structure that
encompass both in-house and contractor efforts,
o prepare a cost analysis requirements description,
o prepare an independent government estimate at each milestone of the
program, and
o conduct a cost risk assessment that identifies the level of uncertainty
inherent in the estimate.
Further, we recommend that the NASA Administrator develop procedures that
would prohibit proposed projects from proceeding through the review and
approval process when they do not address the elements of the recommended
cost-estimating practices.
Agency Comments and Our Evaluation
In written comments on a draft of this report, NASA's Deputy Administrator
stated that the agency concurs with the recommendations, adding that the
recommendations validate and reinforce the importance of activities
underway at NASA to improve cost estimating and program management.
Notwithstanding agreement with our recommendations, the Deputy
Administrator believes NASA has made substantive changes and achieved
significant improvements in its cost-estimating processes. For example,
NASA's comments on a draft of this report cite a 1992 GAO report
(GAO/NSIAD-93-97) that found a median 77 percent increase in NASA program
costs. According to the Deputy Administrator, this contrasts with a 13
percent cost growth in this present study. While there may be improvements
in the percent of cost growth of some projects, such declines in cost
growth are often achieved by rescoping and rebaselining projects to remain
within available resources, as was demonstrated in a number of projects
discussed in this report. We do not believe other examples cited by the
Deputy Administrator, namely termination of the Checkout and Launch
Control System and cost control measures imposed on the International
Space Station, demonstrate that NASA has made substantive changes and
achieved significant improvements in its costestimating processes. Rather,
we believe these examples demonstrate what happens when projects are
undertaken without a full understanding of the potential costs and
management challenges inherent in many of the programs NASA proposes and
then implemented without adequate financial management systems in place.
With regard to our recommendation to develop guidelines for rebaselining
and ensure effective use of earned value management, the Deputy
Administrator cited the development of revised direction on program and
project management and a refocus on risk and cost-risk analysis. NASA also
now requires the establishment of cost thresholds that, if exceeded, will
require a rebaselining review. Further, because much of NASA's work is
performed through grants and contracts, NASA's revised procedures will
emphasize how risk and technical complexity affect contractor performance.
New earned value management and acquisition policies and procedures will
be implemented through program management councils that will review and
approve programs and projects regularly through each step of their
development. Also, a new Cost Analysis Division has been established, and
cost-estimating staff has been added to it and NASA's Independent Program
Assessment Office. NASA also noted the importance of training needed to
match the new requirements.
NASA's Deputy Administrator also concurred with our recommendation to
establish a standard framework for developing life-cycle cost estimates.
According to the Deputy Administrator, NASA's new processes and procedural
requirements document will define the full life-cycle cost to include
development, operations, maintenance, disposal, and all NASA inhouse
direct and indirect costs to eliminate ambiguity and ensure consistency.
NASA's revised cost-estimating handbook will provide further guidance for
life-cycle cost estimates. Also, project managers will be responsible for
developing and maintaining a cost analysis requirements document similar
to a tool DOD uses that will include the equivalent of a project and
technical description; key performance parameters, including documentation
of actual work breakdown structure cost elements; and initial and annual
updates of the life-cycle cost estimates. NASA guidance will also require
periodic independent cost estimates on major programs and approval by the
respective program management council to enter into implementation after
an independent estimate has been completed.
Lastly, NASA's Deputy Administrator concurred with our recommendation to
prohibit proposed projects from proceeding through the review and approval
process when they do not address the elements of the recommended
cost-estimating practices. Accordingly, NASA's forthcoming procedural
requirements will define the authority of the program management councils
that will, according to NASA, enforce the requirements, including the
required information, documentation, and management methods needed for
proceeding through the review and approval process. The Deputy
Administrator also noted the availability of recent management information
system improvements that enhance visibility over project and program
performance. In his general comments, the Deputy Administrator also stated
that NASA had recently taken steps to address issues raised in the draft
report and suggested a report title that would better reflect that
progress.
We agree that NASA has initiated number of reforms to its project
development and implementation processes that, if properly implemented,
would be positive steps to addressing many of the problems noted in this
report. However, we also note that some of these problems have been
long-standing in the projects discussed in this report and in a number of
other projects we and NASA's Office of Inspector General have reviewed.
Furthermore, planned improvements in the past have fallen short of
agencywide implementation. For example, poor or inadequate cost estimates
and management oversight have been central to the problems that plagued
several programs, including those intended to develop new
space transportation and the International Space Station programs. A
reliable financial management structure is central to the success of many
measures noted by the Deputy Administrator in his reply. We recently
reported and testified on the impediments that exist in achieving such a
capability. Finally, we note that contract management has been a
longstanding problem at NASA. In 1990, we identified NASA's contract
management function as an area at high risk. During that time, there was
little emphasis on end results, product performance, and cost control.
NASA found itself procuring expensive hardware that did not work properly.
This report shows that these types of problems still exist. Regarding the
Deputy Administrator's suggestion that we revise the title of our report
to reflect recent progress that NASA has made towards addressing issues
that we raise, we believe NASA's improvements have been properly reflected
in our report's title. We considered the concerns expressed in the Deputy
Administrator's comments, and consistent with our stated position that
NASA's improvements are positive steps but that its problems still
persist, we revised the title accordingly.
Finally, until NASA's integrated financial management system, which is
central to providing effective management and oversight, is fully
implemented, performance assessments relying on cost data may be
incomplete and full costing will be only partially achieved. And until
these problems are resolved and the measures the Deputy Administrator
noted in commenting on a draft of this report are fully implemented and
integrated into the way the agency does business, NASA's contract
management function will continue to be an area of concern.
As agreed with your office, unless you announce its contents earlier, we
will not distribute this report further until 30 days from its date. At
that time, we will send copies to the NASA Administrator and interested
congressional committees. We will make copies available to others upon
request. In addition, the report will be available at no charge on the GAO
Web site at http://www.gao.gov.
If you or your staff have any question concerning this report, please
contact
me at (202) 512-4841 or [email protected]. Key contributors to this report are
acknowledged in appendix VI.
Allen Li Director Acquisition and Sourcing Management
Appendix I
Scope and Methodology
To determine cost estimates in selected NASA programs and any changes in
those estimates, we asked NASA to provide a list of programs that were
currently in the development phase, and programs that had completed
development or were launched in fiscal year 2001or 2002. We also asked
NASA to provide the initial baseline development cost estimate and current
cost estimate for the development phase and life of the program, and the
reasons for changes to initial development cost estimates. NASA identified
68 programs that were currently in development or had completed
development in fiscal years 2001 and 2002. These included planetary
missions and Earth observatory, aeronautical technology, and space flight
systems. From that universe, we selected at least one program (10 in
total) from 5 of NASA's 7 Enterprises. This involved 6 of 9 NASA centers
(and the Jet Propulsion Laboratory) with lead responsibility for one or
more of these programs. Our selection was generally based on programs with
the highest current development cost estimates within an Enterprise. We
compared the initial development cost estimates NASA provided to the
current development cost estimates for the programs. The initial
development estimates generally reflect the projected costs at the time a
new program was first approved by the Congress. The current development
and life-cycle cost estimates reflect the latest estimates provided by
NASA as of April 2003. We also interviewed program officials to obtain
additional information related to NASA's revisions to initially
established baseline development cost estimates, including the rationale
for changes to the cost estimates.
We also analyzed the initial and current development cost estimates for 17
additional NASA programs, later added to the scope of our review, to
ascertain the level of cost growth or decline as those programs progressed
through the development phase.
To assess NASA's cost-estimating processes and methodologies, we used
cost-estimating criteria developed by Carnegie Mellon University's
Software Engineering Institute (SEI) designed to assess the reliability of
project cost and schedule estimates. SEI is a government-funded research
organization that is widely considered an authority on software
implementation. SEI developed checklists with these criteria to help
evaluate software costs and schedule; however, SEI states that these
checklists are equally applicable to hardware and systems engineering
projects. We first analyzed NASA's cost-estimating procedures and
guidelines to determine if they incorporated key components of good
cost-estimating practices advocated by SEI and other experts.
Appendix I Scope and Methodology
Based on that analysis, we selected 14 criteria from two SEI reports1 to
use in assessing NASA's cost-estimating practices for the 10 programs we
selected to review in detail. Our selection of the 14 criteria from the
SEI reports was based, in part, on their commonality with NASA
costestimating procedures and guidelines. Finally, using the
cost-estimating documentation provided by NASA for the 10 programs, we
determined the extent to which the programs met the 14 criteria. If a
program provided substantiating evidence for a criterion, we determined
that the program "fully met" the criterion. If partial evidence was
provided for a criterion, we determined the program "partially met" the
criterion. If no evidence was found, then we determined that the criterion
was "not met." Table 2 describes each of the 14 criteria and the
significance of each criterion.
To identify any barriers that make it difficult to improve any weaknesses
in NASA's cost-estimating processes, we reviewed our recent work on NASA's
efforts to implement a modern integrated financial management system. We
also provided questions to NASA headquarters that asked for information
regarding NASA's ability to use its cost estimates as a management tool
for its programs. We also provided questions related to the SEI criteria,
and NASA's responses to these questions provided further insight into the
agency's cost-estimating management process at the organizational level.
In addition, we interviewed officials in NASA headquarters' Office of the
Chief Financial Officer and Office of the Chief Engineer, and the center
project managers for the 10 programs and other appropriate personnel to
obtain further perspective on this issue.
To accomplish our work, we visited NASA headquarters, Washington, D.C.,
and Goddard Space Flight Center, Maryland. We also contacted officials at
Marshall Space Flight Center, Alabama; Jet Propulsion Laboratory,
California; Kennedy Space Center, Florida; Glenn Research Center, Ohio;
Johnson Space Center, Texas; and Langley Research Center, Virginia.
We conducted our work from February 2003 to March 2004 in accordance with
generally accepted government auditing standards.
1 Software Engineering Institute, A Manager's Checklist for Validating
Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh,
Penn.: Jan. 1995) and Software Engineering Institute, Checklists and
Criteria for Evaluating the Cost and Schedule Estimating Capabilities of
Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Penn.: 1995).
Appendix II
Assessments of 10 Programs Reviewed in Detail
This appendix provides a program by program assessment of the 10 NASA
programs we reviewed in detail. Each assessment provides
o a brief description of the program's mission;
o the status of the program-that is, whether it is in development,
operational, or terminated;
o the year the program was initiated;1
o the fiscal year in which the Congress approved the program-that is,
when full-scale design and development funds were appropriated;
o a comparison of the initial and current (as of April 2003) baseline
development estimates; and
o an assessment of the program's cost-estimating processes,
methodologies, and practices to determine the extent they met the 14
cost-estimating criteria that we used to measure program performance.
(Table 4 shows for each criterion the number of programs that met,
partially met, or did not meet the criterion.)
1 We use the date the program was initiated to refer to the beginning of
the formulation subprocess-the phase of a NASA program that establishes an
affordable project concept and plan to meet mission objectives or
technology goals.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
Table 4: Summary of the Number of Programs That Met, Partially Met, or Did Not
Meet Criterion Number of programs that met criterion
Criterion Met Partially met Not met
The objectives of the estimate are stated in 2 5
writing.
The life cycle to which the estimate applies is 0 10
clearly defined.
The task has been appropriately sized. 1 5
The estimated cost and schedule are consistent
with demonstrated
accomplishments on other projects. 0 7
A written summary of parameter values and their
rationales accompany
the estimate. 0 4
Assumptions have been identified and explained. 2 6
A structured process such as a template or
format has been used to ensure that key
factors have not been overlooked. 3 7
Uncertainties in parameter values have been 0 3
identified and quantified.
If a dictated schedule has been imposed, an estimate of the normal
schedule has been compared to the additional expenditures required to meet
the dictated
aaa
schedule.
If more than one cost model or estimating approach has been used, any
differences
in results have been analyzed and explained. 0 4
Estimators independent of the performing organization concurred with
the
reasonableness of the parameter values and estimating methodology. 3 7
Estimates are current. 3 7
The results of the estimate have been integrated with project planning 6 4
and tracking.
Earned value reporting has been used to manage the program. 2 8
Source: NASA (data), SEI (criteria), GAO (analysis).
aThis criterion did not apply to 5 of the 10 programs we reviewed. For
those 5 programs to which the criterion did apply, none provided evidence
comparing the dictated schedule to the normal schedule.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
SPACE SCIENCE
Gravity Probe B
The mission of the Gravity Probe B (GP-B) space vehicle-launched in April
2004-is to test Einstein's theory of relativity, which states that space
and time are very slightly distorted by the presence of massive objects,
such as Earth. Over approximately 16 months, GP-B will measure very
precisely, the expected tiny changes in the direction of the spin of four
gyroscopes contained in GP-B as it orbits at a 400-mile altitude directly
over the poles. The gyroscopes, free from disturbance, will provide an
almost perfect space-time reference system.
Program Facts
o Status: Development
o Program initiation: Fiscal year 1993
o Program approved by Congress: Fiscal year 1996
o Comparison of initial and current baseline development estimates:
$179.7 million or 33.9 percent increase
Development Baselines
Initial baseline
Baseline as of April 2003
$709
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Estimates used as o Estimate life cycle o Estimate objectives
baselines for program clearly defined stated in writing
tracking o Assumptions identified o Tasks appropriately
and explained sized
Structured format used Estimated costs based
o to ensure all costs o on demonstrated
are
captured programs
o Dictated schedules o Written documentation
show cost impacts of of parameter values
acceleration and rationale
Independent estimates Parameter value
o concur with program o uncertainties
identified and
estimates quantified
o Estimates reflect o More than one cost
changes over time model or estimating
o Earned value reporting approach used
used to manage
program
Sources: NASA (data),
SEI (criteria), GAO
(analysis).
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The GP-B program failed to provide evidence of a documented complete cost
estimate, including the lack of a written objective or description of the
program to be estimated. While the program assessed risk, there was no
evidence of how it determined the impact the risk elements had on the cost
estimates. Our other key findings related to the GP-B program are
summarized as follows:
o The GP-B program failed to provide evidence of a documented complete
cost estimate, including the lack of a written objective or description of
the program to be estimated. While the program assessed risk, there was no
evidence of how it determined the impact the risk elements had on the cost
estimates. Our other key findings related to the GP-B program are
summarized as follows:
o Although a partial list of assumptions was included in the GP-B
cost-estimating documentation, it was not clear to us whether those
assumptions pertained to the entire life- cycle cost estimate.
o A high-level cost breakdown was provided; however, descriptions for key
program elements-including formulation, implementation, operations, launch
vehicle, and tracking and data-were not included. A NASA official stated
that a more detailed work breakdown structure existed, but this breakdown
was not provided.
o The delays in the launch date-originally scheduled for 2000 and pushed
out to 2003 (the actual launch was in April 2004)-causing cost overruns
were documented. However, there was no explanation of how the schedule
delays impacted the cost estimate.
o The GP-B program submitted independent reviews annually between 1997
and 2001 and for 2003. The 1998 independent review stated that program
cost was a challenge for the program. To control costs, the 2001
independent review recommended that the program office (a) constrain the
workforce, (b) replan the schedule, (c) have an independent team look
again at the remaining costs, and (d) develop a contingency plan. However,
none of these reviews provided details on the costestimating methodology.
o NASA submitted the history of programmatic changes from June 1994 to
June 2004, but the associated increases in program costs were not
included.
o GP-B gathers earned value type data using NASA Form 533. However, the
form did not report full-earned value management data. According to a NASA
official, a modified milestone-based earned value management system was
used because the prime contractor resisted the implementation of a
full-earned value management system.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
SPACE SCIENCE
Mars Exploration Rovers
Launched in the summer of 2003, NASA's twin roving exploration
robots-Spirit and Opportunity-landed on opposite sides of Mars in January
2004 in search of answers about the history of water on the red planet.
Over the course of their 90-day mission, the rovers were expected to
perform on-site geological investigations, searching for and
characterizing a wide range of rocks and soils. The robotic geologists
were equipped with mast-mounted cameras that provide 360-degree,
stereoscopic, humanlike views of the terrain; robotic arms capable of
human-like elbow and wrist movements; and a mechanical "fist" with a
microscopic camera and rock hammer.
Source: NASA.
Program Facts
o Status: Operations
o Program initiation: Fiscal year 2000
o Program approved by Congress: Fiscal year 2001
o Comparison of initial and current baseline development estimates:
$109.8 million or 16.7 percent increase
Development Baselines
Initial baseline
Baseline as of April 2003 $657
$767
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Structured format used to ensure all costs are o Estimate objectives
stated in writing o Tasks appropriately sized captured o Estimate life
cycle clearly defined o Written documentation of parameter values
o Independent estimates concur with program o Estimated costs based on
demonstrated and rationale estimates programs o Parameter value
uncertainties identified and
o Estimates are kept current by reflecting o Assumptions identified and
explained quantified changes over time o Dictated schedules show cost
impact of o More than one cost model or estimating
o Estimates used as baselines for program acceleration approach used
tracking o Earned value reporting used to manage program
Sources: NASA (data), SEI (criteria), GAO (analysis).
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The MERs program failed to provide evidence of a documented complete cost
estimate. However, the main reason for the more than $100 million cost
growth was the imposed schedule requirements. As NASA stated, there was a
"less than optimal utilization of project funds...driven by the
significant loss of leverage associated with design heritage from the
Pathfinder mission and the extremely tight schedule which did not have any
resiliency to accommodate the design changes and flight hardware delays."
Our other key findings related to the MERs program are summarized as
follows:
o While the MERs cost-estimating supporting o A NASA official stated
that the MERs program did documentation provided to us collectively
described the program's objectives, no one document clearly and concisely
described the overall objectives-key to developing a reliable life-cycle
cost estimate.
o A high-level breakout of the life-cycle phases of the estimate was
provided, but descriptions of the phases were not included in the
documentation.
o NASA stated that the program relied on design heritage from the
Pathfinder program. This assumption ultimately led to cost and schedule
overruns and rebaselinings.
o Some high-level assumptions about the cost estimate were provided. A
reference to "cost guidelines" was made, but those guidelines were not
provided. One document showed the cost estimate assumptions for the flight
system component of the program.
o MERs was initially given a 3-year development schedule to meet the May
to June 2003 launch window-determined by the relative positions of the
Earth and Mars. To meet this date within the initial budget, NASA planned
to leverage existing technology. However, the program soon discovered this
would not be possible. With the launch date set, the program embarked on
multiple concurrent development efforts to meet the schedule, leading to
cost overruns.
not have an integrated earned value management system. Instead, the
program used NASA form 533, which does not track earned value data since
thereis no measure of the progress of work performed.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
SPACE SCIENCE
Space Infrared Telescope Facility
The Space Infrared Telescope Facility (now called Spitzer), launched in
August 2003, is the fourth and final mission in NASA's Great Observatories
Program-a program designed to see the universe in different kinds of
light. During its planned 2 1/2-year mission, SIRTF aims to detect
infrared heat, which is mostly blocked by the Earth's atmosphere. Infrared
light penetrates gas and dust clouds, allowing scientists to peer into
hidden regions of space, revealing star formations, centers of galaxies,
and newly forming planetary systems. Infrared light also provides
information about cooler objects, such as dim stars, extrasolar planets,
and giant molecular clouds.
Source: NASA.
Program Facts
o Status: Operations
o Program initiation: Fiscal year 1984
o Program approved by Congress: Fiscal year 1998
o Comparison of initial and current baseline development estimates: $139
million or 29.3 percent increase
Development Baselines
Initial baseline
Baseline as of April 2003
$611
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Estimates used as baselines for program tracking
Sources: NASA (data), SEI (criteria), GAO (analysis).
o Estimate objectives stated in writing
o Estimate life cycle clearly defined
o Tasks appropriately sized
o Estimated costs based on demonstrated programs
o Written documentation of parameter values and rationale
o Assumptions identified and explained
o Structured format used to ensure all costs are captured
o Parameter value uncertainties identified and quantified
o More than one cost model or estimating approach used
o Independent estimates concur with program estimates
o Estimates reflect changes over time
o Earned value reporting used to manage program
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The SIRTF program failed to provide supporting documentation-such as
sources of data, estimating approach by work breakdown structure element,
and any uncertainties that may accompany the cost elements-to the evidence
provided on various cost elements and high level explanations of the
methodology used to estimate them. Further, the life-cycle cost estimate
was underestimated because it did not account for various operations and
maintenance costs once SIRTF was launched-despite the prelaunch cost
growth SIRTF experienced due to technical problems such as weight growth,
thermal design issues, telescope heritage problems, and aperture door
risks. Our other key findings related to the SIRTF program are summarized
as follows:
o NASA provided elements of a cost estimate in 1997 non-advocate briefing
slides, but the objectives of the estimates were not concisely or clearly
stated. And while the SIRTF program plan described the overall program
objectives, it did not include the level of detail needed to generate an
estimate.
o The SIRTF 1996 life-cycle cost estimate did not include key life-cycle
costs, such as field support, science, mission operations, flight
operations, and storage. To identify all costs associated with SIRTF's
life cycle, estimates needed to include all costs that support SIRTF's
planned 2 1/2-year mission.
o NASA submitted an example of the parametric model inputs, but provided
no parameters for software. In addition, the 1996 non-advocate review
showed a high-level estimation and validation approach, but did not
include detailed documentation of the sources of data, an estimating
approach by work breakdown structure element, or any uncertainties that
might accompany the cost elements. Finally, the 1997 nonadvocate review
showed detailed estimates for SIRTF subelements, but no supportive
documentation.
o The 1996 and 1997 nonadvocate reviews showed that cost probability
simulations were conducted for certain SIRTF elements, but we found no
evidence that this type of analysis was done for the whole estimate.
o The 1996 nonadvocate review showed that the cost basis had been
validated by independent cost modeling and industry estimates. In the Fall
of 2001, independent reviewers recommended adding 3 to 6 months and $32
million to $55 million to the development program. In October 2002,
independent reviews suggested that an additional $73 million to $130
million would be required for operations activities.
o The development estimate changed four times over the history of the
program. The primary reason for the increases was delays in the schedule.
These changes were tracked in the program plan and task plan.
o Despite the evidence of monthly tracking, we do not view this as
indicative of earned value analysis since there was no measure of the
progress of work performed. Furthermore, there was no description of the
variances' drivers. NASA stated that earned value was applied to the SIRTF
program using a rudimentary Excel-based system, but submitted no reports
from this system.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
EARTH SCIENCE
Landsat-7
Launched in April 1999, Landsat-7 is the latest in a series of earth
observation satellites. Since 1972, Landsat satellites have collected
continuous data on the earth's continental surfaces for land surface
monitoring and global change research. Landsat-7's combination of synoptic
coverage, high spatial resolution, spectral range, and radiometric
calibration is unparalleled and provides digital data in greater
quantities, more quickly, and at lower cost than at any previous time in
Landsat's history.
Source: NASA.
Program Facts
o Status: Operations
o Program initiation: Fiscal year 1992
o Program approved by Congress: Fiscal year 1995
o Comparison of initial and current baseline development estimates: $63
million or 14.1 percent increase
Development Baselines
Initial baseline
Baseline as of April 2003
$446
$509
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Estimate life cycle clearly defined
o Tasks appropriately sized
o Estimated costs based on demonstrated programs
o Structured format used to ensure all costs are captured
o Parameter value uncertainties identified and quantified
o Dictated schedules show cost impacts of acceleration
o Independent estimates concur with program estimates
o Estimates reflect changes over time
o Estimates used as baselines for program tracking
o Earned value reporting used to manage program
o Estimate objectives stated in writing
o Written documentation of parameter values and rationale
o Assumptions identified and explained
o More than one cost model or estimating approach used
Sources: NASA (data), SEI (criteria), GAO (analysis).
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The Landsat-7 program lacked documentation of a complete life-cycle cost
estimate. Our other key findings related to the Landsat-7 program are
summarized as follows:
o None of the Landsat-7 cost-estimating supporting documentation provided
to us described the cost estimate for the complete life cycle (1994-2005).
While we found a high-level work breakdown of costs through 2004, it is
unclear if those costs included all life-cycle costs. For example, in
April 2003, NASA provided us a cost estimate of $509 million for its
portion of the life-cycle cost for Landsat-7. However, in a major review
in 1996, a total life-cycle cost of $848 million was presented. Although
the funding breakout showed the U.S. Geological Survey, National Oceanic
and Atmospheric Administration, and DOD as having funding responsibility
for about $400 million of the $848 million, itis unclear what the true
life-cycle cost of Landsat-7 is from the documentation presented.
o Landsat-7's program plan is highly dependent on heritage from Landsat-6
for management, budget, and schedule. We find this approach questionable
because Landsat 6's failures were never fully understood and NASA's last
significant involvement wason Landsat-5, launched in1984. Further,
internal NASA reviewers warned in 1998 that heritage without original
staff was not heritage and that 20-year old designs could not be
reproduced.
o Although Landsat-7's launch date was accelerated to May 1998-to
accommodate closing of the contractor's facility at Valley Forge-no
corresponding impact to the cost estimate was provided. Ultimately, a
series of design and production problems, uncovered during system level
testing, delayed launch and increased cost by more than $50 million.
o Two independent reviews-in March 1996 and in April 1997-found that
ground system development commitments had not been met and program
contingency funds were critically low. The 1997 review characterized the
integration and testing schedule as optimistic and the May 1998 launch
date as unrealistic. Despite these reviews, it is unclear whether the
assessments concurred with the program cost estimate since we found no
summary of such a comparison in the data.
o NASA submitted a time-phased estimate that compared 1997 and 1998 cost
estimates and provided high-level reasons for the changes and a summary of
contingency cost changes from 1995 to 1998. However, we found no
documentation of the entire life-cycle cost estimate for the initial
program, nor the changes that occurred over time.
o A financial status report for October 1998 to April 1999 compared
planned and actual costs as well as obligations and expenditures and
provided a top-level explanation of cost variances using the work
breakdown structure cited in the document. While the documentation also
provided causes for launch delays, we found no evidence of data that
detailed the corresponding cost impact.
o According to NASA documentation, a project action was initiated in
March 1995 to develop a performance measurement system. Cost and schedule
variances were reported for August 1996, but due to a lack of detail, we
were unable to determine whether this was truly earned value data.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
EARTH SCIENCE
Aqua
Aqua, part of the Earth Observing System (EOS), is expected to provide a
6-year chronology of Earth and its processes. Launched in May 2002, the
Aqua satellite collects information on evaporation from the oceans, water
vapor in the atmosphere, clouds, precipitation, soil moisture, sea and
land ice, and snow cover. Aqua also measures radiative energy fluxes;
aerosols; land vegetation cover; dissolved organic matter and
phytoplankton in the oceans; and air, land, and water temperatures.
Measurements taken by on-board instruments will allow scientists to assess
long-term climate change, identify its human and natural causes, and
advance
Source: NASA. the development of models for long-term forecasting.
Program Facts
o Status: Operations
o Program initiation: Fiscal year 1991
o Program approved by Congress: Fiscal year 1991
o Comparison of initial and current baseline development estimates: $53.1
million or 5.3 percent decrease
Development Baselines
Initial baseline
Baseline as of April 2003
$1006 $952 1000
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Estimate objectives stated in writing
o Estimate life cycle clearly defined
o Assumptions identified and explained
o Structured format used to ensure all costs are captured
o Dictated schedules show cost impacts of acceleration
o Independent estimates concur with program estimates
o Estimates reflect changes over time
o Estimates used as baselines for program tracking
o Earned value reporting used to manage program
o Tasks appropriately sized
o Estimated costs based on demonstrated programs
o Written documentation of parameter values and rationale
o Parameter value uncertainties identified and quantified
o More than one cost model or estimating approach used
Sources: NASA (data), SEI (criteria), GAO (analysis).
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The Aqua program lacked evidence of a well-documented life-cycle cost
estimate. Delays in Aqua's launch, originally scheduled for December 2000,
increased the program's cost to over $49 million, contributing to cost
overruns. Our other key findings related to the Aqua program are
summarized as follows:
o The President's budget submission for fiscal year 2003 indicated that
the $49 million increase in the program baseline included costs for
project support; imaging, sound, and sensor instruments; launch vehicle;
and contingency. However, we found no evidence of operations, support, and
other potential costs. Further, the President's fiscal year 2002 budget
assumed project support and operations costs only for launch plus 120
days, although the program's life cycle is planned for continuance beyond
120 days. Aqua was built to gather data for six years and for full
life-cycle cost estimating; the estimate should represent the lifetime of
operations expected for Aqua.
o We found under the assumptions for the President's 2002 Budget Request
that the costs represented project support through launch plus 120 days.
Furthermore, we found that the budget estimate did not include costs for
mission operations beyond the initial 120 days.
o A high-level work breakdown structure was provided, but it did not
include all costs for the life cycle.
o The effect of a slip in the launch readiness date, which caused a
significant delay, on the cost estimate was not provided.
o While NASA provided evidence of independent reviews of the program, the
most recent one provided was in 2000, and the reviewers concluded that
there would be significant budget shortfalls in fiscal years 2001 and
2002.
o Although evidence of changes to life-cycle costs estimates was
provided, we found no evidence showing that the entire life-cycle cost
estimate was kept current given that the estimate did not include costs
for operations and support.
o We found evidence that the budget estimates were analyzed and presented
to management in a June 11, 2002, monthly status review; however, this
analysis was not indicative of a true earned value management approach.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
EARTH SCIENCE
Aura
Scheduled for launch in June 2004, the Aura satellite is the third in a
series of major Earth-observing satellites to study environment and
climate change. The first and second missions, Terra and Aqua, were
designed to study the land, oceans, and the Earth's radiation budget.
Aura's mission is to study, for at least a 5-year period, the Earth's
ozone, air quality, and climate, focusing exclusively on the composition,
chemistry, and dynamics of the Earth's upper and lower atmospheres.
Source: NASA.
Program Facts
o Status: Development
o Program initiation: Fiscal year 1991
o Program approved by Congress: Fiscal year 1994
o Comparison of initial and current baseline development estimates: $2.1
million or 0.3 percent increase
.
Development Baselines
Initial baseline
Baseline as of April 2003
$763
$765
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Estimate objectives stated in writing
o Estimates reflect changes over time
o Estimate life cycle clearly defined
o Tasks appropriately sized
o Estimated costs based on demonstrated programs
o Assumptions identified and explained
o Structured format used to ensure all costs are captured
o Independent estimators concur with program estimates
o Estimates used as baselines for program tracking
o Earned value reporting used to manage program
o Written documentation of parameter values and rationale
o Parameter value uncertainties identified and quantified
o More than one cost model or estimating approach used
Sources: NASA (data), SEI (criteria), GAO (analysis).
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The Aura program lacked evidence of a life-cycle cost estimate with a
methodology, a complete work breakdown structure, or other supporting
documentation. Without such evidence, we were unable to determine if all
associated program costs were included and if the cost estimate were
reliable. Our other key findings related to the Aura program are
summarized as follows:
o A 2002 budget document included funding for Aura o Although NASA
documented use of earned value development plus 90 days of support after
launch, but it did not include costs for launch vehicle contingencies or
funding for operations.
o Evidence of mass and wattage allocations for Aura power did not clearly
show how these data were used and whether they were used to derive a cost
estimate based on a cost/pound or cost/watt cost-estimating relationship.
o Expected efficiencies gained through the Aqua program's experience were
not clearly documented. Aqua's experience was expected to reduce Aura's
cost, schedule, and technical risks because the majority of Aura's
structural drawings and spacecraft database are common with Aqua's;
similar launch site and vehicle activities are also expected to provide
additional efficiencies.
o Four independent reviews of cost estimates for Aura were conducted-in
July 1998, October 1998, October 1999, and October 2000; however, we found
no evidence of a cost-estimating methodology or reviewer concurrence with
the estimates.
o While we also found some evidence that the program was using program
baselines for program tracking-such as a financial status report that
showed monthly trends for cumulative cost as well as obligation plans
versus actual costs and obligations-the evidence was not convincing enough
to demonstrate that baselines were consistently used for program tracking.
management data for the Aura program in the third quarter of 1998, 1999,
and 2000, officials stated that the program had not used such data during
the past year and a half. While a March 2003 cost performance report
provided earned value management data for one of Aura's four ozone
instruments, evidence that the Aura program as a whole was using earned
value management data on a monthly basis was not provided in the Aura
documentation.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
BIOLOGICAL AND PHYSICAL RESEARCH
Fluids and Combustion Facility
The Fluids and Combustion Facility (FCF) is designed to be a permanent
modular facility for conducting microgravity experiments on the
International Space Station. Through these experiments, scientists hope to
enhance their understanding of gravity's role in a wide range of physical
processes, including materials science, power, propulsion, combustion,
fluid physics, and plasma physics. FCF is to be composed of two racks that
share mutually necessary hardware. The fluids integration rack will be
used to perform investigations for microscopic imaging to particle
tracking. The combustion integration rack will be used to study the
process of combustion in a near weightless environment with the aim of
improving fire safety and increasing fuel efficiency.
Source: NASA.
A flame in gravity (left) and in microgravity (right).
Program Facts
o Status: Development
o Program initiation: Fiscal year 1987
o Program approved by Congress: Fiscal year 2001
o Comparison of initial and current baseline development estimates: $4.8
million or 4 percent decrease
Development Baselines
Initial baseline
Baseline as of April 2003
$119
$114
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Structured format used to ensure all costs are captured
o Estimates used as baselines for program tracking
o Earned value reporting used to manage program
o Estimate objectives stated in writing
o Estimate life cycle clearly defined
o Tasks appropriately sized
o Estimated costs based on demonstrated programs
o Written documentation of parameter values and rationale
o Assumptions identified and explained
o More than one cost model or estimating approach used
o Independent estimates concur with program estimates
o Estimates reflect changes over time
o Parameter value uncertainties identified and quantified
Sources: NASA (data), SEI (criteria), GAO (analysis).
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The FCF program lacked evidence to determine the consistency and
efficiency of the program's estimating. Further, the life-cycle cost
estimates did not provide a clear description of costs included, and the
life cycle began in 1999-12 years after the program was initiated. Our
other key findings related to the FCF program are summarized as follows:
o The February 2001 FCF independent cost estimate- an
estimate-to-complete in real year dollars-clearly defined the objectives
of the estimate and the ground rules and assumptions of what was and what
was not contained in the estimate. Further, it took into accountprevious
development status. However, because detailed initial and life-cycle cost
estimates were not provided, we could not determine if the objectives of
the initial estimates were clearly defined.
o Although the 2001 FCF independent cost estimate clearly defined the
life cycle that the estimate applied to, it did not cover the program from
its initiation in 1987. Instead, development costs were considered sunk
costs in another NASA area. In addition, an estimate of operations support
did not include an analysis or boundaries to back up the estimate.
Finally, the current total life-cycle cost for the FCF program was not
evident in the documents provided.
o All cost estimates for software that were provided and reviewed have
been appropriately sized; however, there were no initial estimates
available for FCF software to review or compare against the preliminary
design review. Further, the independent cost estimate did not map to the
preliminary design review to allow comparisons.
o The government estimate for FCF's prime development contract was broken
down by work breakdown structure element and provided a basis of estimates
for each element that showed the formula of how the cost was derived.
However, supporting documentation did not show how the hours or cost of
materials were estimated or explain the parameters that might have been
used in the estimates.
o While an independent cost assessment was provided with a complete cost
estimate stating assumptions and cost and schedule risks, no conclusions
were provided. The independent assessors did not provide opinions on the
reasonableness of the parameter values and estimating methodology. NASA
provided a list of formal assessments the FCF program had completed, but
the underlying documentation was not provided. Therefore, it was not clear
to us whether or not the performing agencies of those assessments
concurred with the parameter values and methodology.
o NASA provided a budget trace that showed the changes to the program
estimates from the year 2000 to 2003. The budget trace provided the cost
estimates from that period in the program forward. This trace explained
the changes to the estimates at a top level from year to year. However,
there were no details provided to show what specifically caused the
changes in each of the year's estimates.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
AERONAUTICS
Hyper-X Program
The goal of NASA's Hyper-X program is to flight validate key propulsion
and related technologies for air-breathing hypersonic aircraft. The
Hyper-X (X-43A) vehicle, launched in March 2004, flew at Mach 7-greater
than the cruising speed of the SR-71, the world's fastest air-breathing
aircraft, which cruises slightly above Mach 3. The highest speed attained
by NASA's rocket-powered X-15 was Mach 6.7, back in 1967. NASA anticipates
that the technologies exposed by the Hyper-X Program will increase payload
capacities and reduce costs for future air and space vehicles.
Source: NASA.
Program Facts
o Status: Development
o Program initiation: Fiscal year 1996
o Program approved by Congress: Fiscal year 1998
o Comparison of initial and current baseline development estimates: $60
million or 35.9 percent increase
Development Baselines
Initial baseline
Baseline as of April 2003
$167
$227
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Estimates used as baselines for program tracking
Sources: NASA (data), SEI (criteria), GAO (analysis).
o Estimate life cycle clearly defined
o Structured format used to ensure all costs are captured
o Independent estimates concur with program estimates
o Estimates are kept current by reflecting changes over time
o Earned value reporting used to manage program
o Estimate objectives stated in writing
o Tasks appropriately sized
o Estimated costs based on demonstrated programs
o Written documentation of parameter values and rationale
o Assumptions identified and explained
o More than one cost model or estimating approach used
o Parameter value uncertainties identified and quantified
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The Hyper-X program lacked a program plan and a detailed, activity-based
work breakdown structure to manage schedule, funding, and staffing at
multiple sites-which resulted in the program's consistently running over
its budget and not meeting schedule requirements despite the program's
high level of technical expertise needed to complete the project. Our
other key findings related to the Hyper-X program are summarized as
follows:
o The supporting cost-estimating documentation o In a monthly
contractor financial management provided to us for the Hyper-X program
included no definition of the life cycle. Information provided in those
documents consisted of past costs with some projections of post-production
modification and support costs. However, the costs and projections were
stated at a high level and did not include an explanation of where they
occurred in the life cycle.
o Although a partial cost estimate in a work breakdown
structureformatimplied that a structure was established for the program by
1996, the structure was not explicitly provided. The contractor's
financial management report for July 2002 was also provided in a work
breakdown structure format, but no master structure was included in the
data to decipher the extent of the effort. Further, a NASA official stated
that a work breakdown structure was used for both major contracts,
however, it was not used to track costs.
o A nonadvocate review and a cost validation review stated that the
Hyper-X program could not meet program objectives within current program
funding and that additional funding would be required. However, parameter
values and estimating methodology were not discussed in the documents.
o Updated estimates for the Hyper-X program were not provided in the two
aforementioned reviews. However, information in the Hyper-X program status
brief indicated that NASA had to use some updated cost information to
determine that cost overruns were going to occur for the particular system
for which it was proposing alternatives.
report containing the variance analysis, the earned value report, and the
progress report (the only monthly report provided) for the Hyper-X
contract-only actual and planned costs were included; no earned value data
for analysis were provided. In addition, the total value provided in this
report totaled $67.8million-wellbelow theHyper-X program's total cost.
Furthermore, a NASA official stated that EVM was a deliverable on the
contracts, but the contractor used NASA form 533, which does not provide
full EVM data. Also, the data showed that there were no EVM specialists
employed in the Hyper-X program office. As a result, the evidence provided
was insufficient for us to determine whether the Hyper-X program was
implementing earned value throughout the total program.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
SPACE FLIGHT
Checkout and Launch Control System
The Checkout and Launch Control System (CLCS) was intended to replace a
central component in NASA's existing launch processing system for the
space shuttle. The original justification for CLCS was that a substantial
portion of the vendors for the command control and monitor system no
longer provided support. In addition, out-ofdate software and systems were
expected to increase costs. CLCS promised to reduce staff, paperwork, and
operations and maintenance costs by 50 percent. The program was canceled
in September 2002 due to cost overruns, which according to NASA, were
caused by factors such as software development delays based on poorly
defined requirements and design, integration problems, and a lack of
experienced development staff.
Source: NASA.
Program Facts
o Status: Canceled
o Program initiation: Fiscal year 1996
o Program approved by Congress: Fiscal year 1998
o Comparison of initial and current baseline development estimates: $193
million or 93.7 percent increase
Development Baselines
Initial baseline
Baseline as of April 2003
$399
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Assumptions identified and explained
o Independent estimators concur with program estimates
o Estimate objectives stated in writing
o Estimate life cycle clearly defined
o Tasks appropriately sized
o Estimated costs based on demonstrated programs
o Written documentation of parameter values and rationale
o Structured format used to ensure all costs are captured
o More than one cost model or estimating approach used
o Dictated schedules show cost impacts of acceleration
o Estimates reflect changes over time
o Estimates used as baselines for program tracking
o Earned value reporting used to manage program
o Parameter value uncertainties identified and quantified
Sources: NASA (data), SEI (criteria), GAO (analysis).
Appendix II
Assessments of 10 Programs Reviewed
in Detail
The CLCS program lacked complete evidence supporting the cost estimate.
Our other key findings related to the CLCS program are summarized as
follows:
o The description of the program objectives and o A program management
review included a graphic overview provided in the program commitment
agreement was not the description used to generate the cost estimate, and
supporting documentation of the detailed cost estimated referred to in the
1997 non-advocate review briefing was not provided.
o The total life cycle and WBS were not defined in the program's
life-cycle cost estimate.
o The 1997 nonadvocate review identified the analogy to be used as well
as six different projects for parametric estimating. However, no details
on the cost model parameters were documented.
o No evidence was provided to explain how the schedule slip-from June
2001 to June 2005- impacted the cost estimate.
o Various documents discuss various estimating approaches, but no
evidence of the differences in estimating approaches being analyzed was
provided.
o Detailed descriptions of changes in CLCS estimates were not provided,
although NASA stated that changes were tracked and estimates were updated
accordingly.
o A briefing on CLCS software stated that progress was tracked against
the plan for software development, but no documentation on such tracking
at a total program level was provided.
displaying earned value data, and a NASA official stated in an interview
that EVM had been used since 2000. However, cost performance reports or
other supporting documentation showing that EVM had been used were not
provided.
Appendix II
Assessments of 10 Programs Reviewed
in Detail
SPACE FLIGHT
Cockpit Avionics Upgrade
The Cockpit Avionics Upgrade (CAU) project is redesigning the display
formats on the liquid crystal displays of the space shuttle cockpit. The
objective of the redesign is to enhance flight safety by presenting the
crew with flight and vehicle critical information in a user-friendly
format that enhances situational awareness. Because the new display format
uses graphics and color to present complex information, crews are expected
to have better and more rapid decision-making capability under off-nominal
conditions than could be made with the legacy system, enhancing flight
safety and the crew's ability to meet mission objectives.
Source: NASA.
Program Facts
o Status: Development
o Program initiation: Fiscal year 2000
o Program approved by Congress: Fiscal year 2003
o Comparison of initial and current baseline development estimates: $12
million or 2.7 percent increase
Development Baselines
Initial baseline
Baseline as of April 2003
$442
$454
Dollars in millions
Source: GAO.
Cost-Estimating Criteria
Met Partially met Not met
o Estimate objectives stated in writing
o Tasks appropriately sized
o Assumptions identified and explained
o Structured format used to ensure all costs are captured
o Independent estimators concur with program estimates
o Estimates reflect changes over time
o Estimates used as baselines for program tracking
o Earned value reporting used to manage program
Sources: NASA (data), SEI (criteria), GAO (analysis).
o Estimate life cycle clearly defined
o Estimated costs based on demonstrated programs
o Written documentation of parameter values and rationale
o Parameter value uncertainties identified and quantified
o More than one cost model or estimating approach used
Appendix II
Assessments of 10 Programs Reviewed
in Detail
Our assessment results for the CAU program were the highest among the ten
programs we reviewed. However, we noted some weaknesses. The contractor
failed to include full life-cycle costs in its life-cycle cost estimate
and detailed cost information for some work breakdown structure elements
to support the estimate and the analogies to historical known programs.
Further, we found no evidence that the CAU program office conducted its
own cost estimate prior to receiving the contractor's proposal, providing
the office no objective means to assess the realism of the contractor's
estimate before the nonadvocate reviews conducted in July 2001 and October
2002. Our other key findings related to the CAU program are summarized as
follows:
o CAU project costs for certain life-cycle costs through 2008-such as
design, development, and certification and delivery of the hardware and
software; facility costs associated with the upgrade; and development of
operational products associated with the upgrade, as well as reserves-were
documented. However, other projects costs, such as installation costs,
costs to sustain engineering, and operations and support were not. The
sustaining engineering costs after 2008 were assumed to be zero because
the costs would be absorbed through efficiencies provided by the CAU
system. However, the assumption is unproven and could lead to higher costs
after 2008. Further, although NASA requires that each estimate include
specific information-including scope, definitions of terms, ground rules
and assumptions, detailed description of the estimating methodology and
the rationale for the approach, time-phased dollar estimates, pricing
factors, and results of quantitative risk analyses-we found no evidence of
this detailed information. Finally, the CAU program office said that
impacts to the project due to full cost accounting have not been defined.
o The nonadvocate review costs were based on applied parametric
estimating tools using CAU development process and technical descriptions
of products including PRICE-H and NAFCOM for hardware development and SEER
SEM for software development. Detailed assumptions for hardware and
software development were provided, but there was no detail supporting the
cost estimates for a number of elements, including ground facilities and
integrated logistics, to facilitate re-creation of the estimate.
o PRICE-H and NAFCOM cost models were used to estimate hardware
development; however, the documentation did not address whether there were
any differences in model results; therefore, it is unclear whether one
model was used forthe primary estimate and the other was used to validate
that estimate as a crosscheck. For software, we only found evidence of the
SEER-SEM model.
Appendix III
Summary Descriptions of the 17 Additional Programs
In addition to the 10 programs that we reviewed in detail, we analyzed the
initial and current development cost estimates for 17 other NASA programs.
Space Science Enterprise
Thermosphere, Ionosphere, Mesosphere Energetics and Dynamics (TIMED)
NASA's TIMED satellite is conducting the first global study of the Earth's
mesosphere, lower thermosphere, and ionosphere-segments of the Earth's
atmosphere located between 40 and 110 miles above the planet. Initially,
TIMED's mission was to last 2 years, beginning with its launch in December
2001, but NASA extended the satellite's orbital operations through 2006.
TIMED's goal is to improve our understanding of the influences the sun and
humans have on this "gateway region" as well as the effects of its
atmospheric variability on satellites and spacecraft reentering the
Earth's atmosphere.
International Gamma-Ray Astrophysics Laboratory (INTEGRAL)
INTEGRAL is a European Space Agency mission, with Russian and U.S.
involvement. Launched in October 2002, the INTEGRAL satellite is equipped
with two telescopes designed to register elusive gamma rays- some of the
universe's most energetic radiation-and give insight into the most violent
processes in our universe. Through INTEGRAL, scientists plan to study
black holes' interaction with their surroundings, the explosion of
supernovae and their role in forming chemical elements, the nature of
powerful gamma-ray bursts, and transient sources that suddenly change
brightness. U.S. participation consists of co-investigators providing
hardware and software components to the spectrometer and imager
instruments, a co-investigator for the data center, a mission scientist,
and a provision for ground tracking and data collection.
Rosetta Rosetta is a European Space Agency mission whose objectives are to
study the origin of and the relationship between comets and interstellar
material and to improve our knowledge of the origins of the Solar System.
The Rosetta satellite was launched in March 2004 and, after a long cruise
phase, is planned to rendezvous with comet Churyumov-Gerasimenko in 2014.
Plans call for Rosetta to orbit the comet while taking scientific
Appendix III Summary Descriptions of the 17 Additional Programs
measurements and to position a probe on the comet surface to take in-situ
measurements. U.S. involvement includes developing three remote-sensing
instruments and a subsystem for a fourth instrument.
Mercury Surface, Space Currently scheduled to launch during a 15-day
period that opens July 30, Environment, Geochemistry 2004, the MESSENGER
spacecraft is intended to collect images of Mercury. and Ranging Through
these images, NASA scientists hope to determine Mercury's
geological history and the nature of its surface composition, core, poles,
(MESSENGER) exosphere and magnetosphere, and magnetic field. This
information is expected to provide scientists with a better understanding
of how Earth was formed, how it evolved, and how it interacts with the
sun.
Solar Terrestrial Relations Observatory (STEREO)
Through STEREO-an international collaboration involving France, Germany,
the United Kingdom, and the United States-NASA plans to trace the flow of
energy and matter from the sun to Earth by studying the solar origin of
coronal mass ejections, their evolution in the heliosphere, and their
effects on geospace. Twin STEREO observatories, scheduled to be launched
in November 2005, will be used to develop a threedimensional,
time-dependent model of the magnetic topology, temperature, density, and
velocity structure of the ambient solar wind. Because coronal mass
ejections are the prime drivers of major space weather hazards, STEREO is
expected to greatly improve our understanding of the most severe
disturbances of the Sun-Earth system. The observatories will also provide
a continuous data stream for the purpose of real-time space weather
forecasts.
Stratospheric Observatory for Infrared Astronomy (SOFIA)
The SOFIA observatory-a modified Boeing 747 aircraft with a permanently
installed telescope, which NASA plans to begin flying in 2005-will be used
to study different astronomical objects and phenomena, including star
births and deaths; solar system formations; complex molecules in space;
planets, comets, and asteroids in our solar system; nebulae and dust in
galaxies; and black holes at the centers of galaxies. The telescope,
provided through a partnership with the German Aerospace Center, is
designed to provide routine access to nearly all of the visual, infrared,
far-infrared, and submillimeter parts of the spectrum. As such, SOFIA is
expected to extend the range of astrophysical observations significantly
beyond that of previous infrared airborne observatories through increases
in sensitivity and angular resolution. NASA plans to
Appendix III Summary Descriptions of the 17 Additional Programs
incorporate new or upgraded technologies over the aircraft's lifetime to
allow additional scientific exploration. Because most of the instruments
are to be designed and built by graduate students and post-doctoral
scientists in universities throughout the United States, SOFIA will serve
as a training ground for the next generation of instrument builders.
Solar-B Observatory The Solar-B program's objectives are to investigate
the interaction between the Sun's magnetic field and its corona and to
understand the sources of solar variability. Solar-B is a Japanese
Institute of Space and Astronautical Science mission, with significant
U.S. involvement, and follows the Solar-A collaboration among Japan, the
United Kingdom, and the United States. The observatory is designed to
consist of a set of optical, extreme ultraviolet, and X-ray instruments,
and NASA is expected to provide components for each. The Solar-B
observatory is scheduled to be launched on a Japanese M-V rocket out of
Kagoshima, Japan, in September 2006.
Herschel Space Observatory The European Space Agency's Herschel Space
Observatory (formerly the Far Infrared and Submillimetre Telescope, or
FIRST) houses an infrared telescope that is expected to observe virtually
unexplored spectrum wavelengths that cannot be observed from the ground.
Scheduled for launch in February 2007, Herschel is expected to enable
scientists to better understand galaxy formation, evolution in the early
universe, and the nature of active galaxy power sources; star-forming
regions and interstellar medium physics in the Milky Way and other
galaxies; and the molecular chemistry of cometary, planetary, and
satellite atmospheres in our solar system. NASA is providing components
for two of the three instruments that will be flown on Herschel: the
Heterodyne Instrument for Far Infrared and the Spectral and Photometric
Imaging Receiver.
Earth Science Enterprise
Terra Launched in February 2000, Terra is providing measurements that,
according to NASA, are significantly contributing to the understanding of
the total Earth system. Specifically, Terra is collecting 200 gigabytes of
data each day on the earth's physical and radiative properties of clouds,
air-land
Appendix III Summary Descriptions of the 17 Additional Programs
and air-sea exchanges of energy, carbon, and water as well as measurements
of trace gases and volcanology. One of the first operational uses of Terra
was to provide imagery to support the U.S. Forest Service's efforts to
combat forest fires in the western United States. Through Terra, fire
fighters were able to identify the locations of active fires, instead of
locations of smoke, providing them with the data needed to better control
spreading fires. Terra data were also used by the Geography Department of
Dartmouth College in New Hampshire to assist in flood hazard reduction
programs.
New Millennium Program's NASA's New Millennium Program (NMP) is designed
to identify, develop,
Earth Observing-1 (EO-1) and flight-validate key instrument and spacecraft
technologies that can enable new or more cost-effective approaches to
conducting science missions. EO-1-the first NMP mission, launched in
November 2000- includes three land imaging instruments that are expected
to lead to a new generation of lighter weight, higher performance, and
lower cost Landsat-type Earth surface imaging instruments.
Jason-1 The mission of the Jason-1 program, a cooperative effort with the
French Space Agency, is to study the global oceans. Launched in December
2001, the Jason-1 satellite was expected to monitor ocean circulation and
events such as El Nino and ocean eddies and to improve global climate
forecasts and predictions. The Jason-1 satellite was positioned to orbit
the earth in tandem with TOPEX/Poseidon, an earlier generation satellite
launched in 1992, to provide data to the National Oceanic and Atmospheric
Administration.
SeaWinds The SeaWinds satellite, launched in December 2002, is providing
highresolution, ocean surface wind data used for studies of ocean
circulation, climate, and air-sea interaction to understand global climate
changes and weather patterns better. By using long-term wind data in
numerical weather and wave prediction models, SeaWinds is expected to
improve weather forecasts near coastlines and storm warning and
monitoring.
Appendix III Summary Descriptions of the 17 Additional Programs
Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations
(Calipso)
The Calipso satellite, scheduled for launch in 2005, is being designed to
study the effect that aerosols and clouds have on the Earth's radiation
balance, which ultimately controls the temperature of the Earth. Calipso
is expected to provide scientists with data to construct three-dimensional
structures of the atmosphere, enabling new observationally based
assessments of the radiative effects of aerosol and clouds that will
greatly improve our ability to predict future climate change. NASA plans
to fly Calipso in formation with Aqua and CloudSat, a satellite being
designed to measure the vertical structure of clouds from space and
contribute to a better understanding of the role of clouds in the Earth's
climate system. The Calipso program is a cooperative effort with France.
Space Flight Enterprise
X-38 Crew Return Vehicle (CRV)
The X-38 Crew Return Vehicle was cancelled in April 2002, due to its
single purpose design and the potentially high costs identified by an
independent assessment. The purpose of the CRV project was to initiate
work toward an independent U.S. crew return capability for the
International Space Station. As envisioned, CRV was expected to serve as a
back-up to the space shuttle orbiters by providing resupply to the station
or change-out crew, or accommodating safe return for up to seven crew
members who may be ill or injured or in the event that a catastrophic
failure of the station made it unable to support life.
Alternate Turbopump Program (ATP)
ATP's primary objectives were to significantly improve the safety and
operating margins of the high-pressure turbopump in the space shuttle's
main engine and to eliminate the need to remove the turbopump for
postflight maintenance. An alternative turbopump was successfully
implemented in the shuttle launched in April 2002. According to NASA,
ATP's development contract, signed in December 1986, specifically
addressed shortcomings of the previous turbopumps; took advantage of the
latest technologies; and applied lessons learned. The contract called for
the parallel development of two high-pressure turbopumps-one that operates
on oxidization and one on fuel. However, 5 years into the program,
technical problems prompted NASA to end parallel development and
concentrate first on developing the oxidizer turbopump, which was first
flown in July 1995. Although development of the fuel turbopump
Appendix III Summary Descriptions of the 17 Additional Programs
resumed in 1994, extreme high temperatures, pressures, and rotor speeds
resulted in significant design challenges and the design certification
review was not completed until March 2001. The full implementation of the
fuel turbopump into flight was completed beginning with the April 2002
shuttle flight.
Tracking and Data Relay Satellite (TDRS) Replenishment
In December 2002, the TDRS Replenishment project achieved its goal: launch
three geosynchronous satellites to replace the existing aging satellite
constellation, and thereby continue to provide space network tracking,
data, voice, and video services to NASA scientific satellites, the Space
Shuttle program, the International Space Station, and other NASA
customers. According to NASA, the functional and technical performance
requirements for the replacement satellites-launched in June 2000, March
2002, and December 2002-are virtually identical to those of the previous
satellites.
Advanced Health Management System (AHMS) Phase 1
AHMS is expected to provide safe shutdown of the space shuttle main engine
during potentially catastrophic high-pressure turbopump failures through
improved monitoring of engine vibration and anomaly response capabilities.
According to NASA, AHMS modifications include (1) adding a vibration
redline monitor for high pressure turbopumps, (2) doubling memory capacity
and employing radiation tolerant memory, (3) adding an external
communication interface for a potential phase-two health management
computer, and (4) eliminating existing memory retention batteries and
replacing them with nonvolatile memory. While NASA stated the AHMS will be
available for launch in January 2005, the shuttle fleet's return to flight
date is planned for March or April 2005.
Appendix IV
Description of Earned Value Management
Earned value management (EVM) goes beyond the two-dimensional approach of
comparing budgeted costs to actuals. Instead, it attempts to compare the
value of work accomplished during a given period with the work scheduled
for that period. By using the value of completed work as a basis for
estimating the cost and time needed to complete the program, earned value
can alert program managers to potential problems early in the program.
An accurate, valid, and current performance management baseline is needed
to perform useful analyses using EVM. In 1996, in response to acquisition
reform initiatives, the Department of Defense (DOD) adopted 32 criteria
for evaluating the quality of management systems. In general terms, the 32
criteria require contractors to (1) define the contractual scope of work
using a work breakdown structure; (2) identify organizational
responsibility for the work; (3) integrate internal management subsystems;
(4) schedule and budget authorized work; (5) measure the progress of work
based on objective indicators; (6) collect the cost of labor and materials
associated with the work performed; (7) analyze any variances from planned
cost and schedules; (8) forecast costs at contract completion; and (9)
control changes. The criteria have become the standard for EVM and have
been adopted by major U.S. government agencies, industry, and the
governments of Canada and Australia. The full application of EVM system
criteria is appropriate for large cost reimbursable contracts where the
government bears the cost risk. For such contracts, management discipline
prescribed by the criteria is essential. In addition, data from an EVM
system have been proved to provide objective reports of contract status,
allowing numerous indices and performance measures to be calculated. These
can then be used to develop accurate estimates of anticipated costs at
completion, providing early warning of impending schedule delays and cost
overruns.
Table 5 lists the 32 criteria, organized into five basic categories:
organization, planning and budgeting, accounting considerations, analysis
and management reports, and revisions and data maintenance.
Appendix IV
Description of Earned Value Management
Table 5: Thirty-Two Criteria for Evaluating the Quality of Management Systems
Category Criteria
Organization 1. Define the authorized work elements for the program. A
work breakdown structure, tailored for effective internal management
control, is commonly used in this process.
2. Identify the program organizational structure, including the major
subcontractors responsible for accomplishing the authorized work, and
define the organizational elements in which work will be planned and
controlled.
3. Provide for the integration of the company's planning, scheduling,
budgeting, work authorization, and cost accumulation processes with each
other and, as appropriate, the program work breakdown structure and the
program organizational structure.
4. Identify the company organization or function responsible for
controlling overhead (indirect costs).
5. Provide for integration of the program work breakdown structure and the
program organizational structure in a manner that permits cost and
schedule performance measurement by elements of either or both structures
as needed.
Planning and budgeting 6. Schedule the authorized work in a manner that
describes the sequence of work and identifies significant task
interdependencies required to meet the requirements of the program.
7. Identify physical products, milestones, technical performance goals, or
other indicators that will be used to measure progress.
8. Establish and maintain a time-phased budget baseline, at the control
account level, against which program performance can be measured. Budget
for far-term efforts may be held in higher-level accounts until an
appropriate time for allocation at the control account level. Initial
budgets established for performance measurement will be based on either
internal management goals or the external customernegotiated target cost,
including estimates for authorized but undefinitized work. On government
contracts, if an over target baseline is used for performance measurement
reporting purposes, prior notification must be provided to the customer.
9. Establish budgets for authorized work with identification of
significant cost elements (labor and material, for example) as needed for
internal management and for control of subcontractors.
10. To the extent it is practical to identify the authorized work in
discrete work packages, establish budgets for this work in terms of
dollars, hours, or other measurable units. Where the entire control
account is not subdivided into work packages, identify the far term effort
in larger planning packages for budget and scheduling purposes.
11. Provide that the sum of all work package budgets plus planning package
budgets within a control account equals the control account budget.
12. Identify and control level of effort activity by time-phased budgets
established for this purpose. Only that effort which is unmeasurable or
for which measurement is impractical may be classified as level of effort.
13. Establish overhead budgets for each significant organizational
component of the company for expenses that will become indirect costs.
Reflect in the program budgets, at the appropriate level, the amounts in
overhead pools that are planned to be allocated to the program as indirect
costs.
14. Identify management reserves and undistributed budget.
15. Provide that the program target cost goal is reconciled with the sum
of all internal program budgets and management reserves.
Accounting considerations 16. Record direct costs in a manner consistent
with the budgets in a formal system controlled by the general books of
account.
Appendix IV
Description of Earned Value Management
(Continued From Previous Page)
Category Criteria
17. When a work breakdown structure is used, summarize direct costs from
control accounts into the work breakdown structure without allocation of a
single control account to two or more work breakdown structure elements.
18. Summarize direct costs from the control accounts into the contractor's
organizational elements without allocation of a single control account to
two or more organizational elements.
19. Record all indirect costs that will be allocated to the contract.
20. Identify unit costs, equivalent units costs, or lot costs when needed.
21. For EVM, the material accounting system will provide (1) accurate cost
accumulation and assignment of costs to control accounts in a manner
consistent with the budgets using recognized, acceptable, costing
techniques; (2) cost performance measurement at the point in time most
suitable for the category of material involved, but no earlier than the
time of progress payments or actual receipt of material; and (3) full
accountability of all material purchased for the program, including the
residual inventory.
Analysis and management 22. At least on a monthly basis, generate the
following information at the control account and other levels
reports as necessary for management control using actual cost data from,
or reconcilable with, the accounting system: (1) Comparison of the amount
of planned budget and the amount of budget earned for work accomplished.
This comparison provides the schedule variance. (2) Comparison of the
amount of the budget earned and the actual (applied where appropriate)
direct costs for the same work. This comparison provides the cost
variance.
23. Identify, at least monthly, the significant differences between both
planned and actual schedule performance and planned and actual cost
performance, and provide the reasons for the variances in the detail
needed by program management.
24. Identify budgeted and applied (or actual) indirect costs at the level
and frequency needed by management for effective control, along with the
reasons for any significant variances.
25. Summarize the data elements and associated variances through the
program organization and/or work breakdown structure to support management
needs and any customer reporting specified in the contract.
26. Implement managerial actions taken as the result of earned value
information.
27. Develop revised estimates of cost at completion based on performance
to date, commitment values for material, and estimates of future
conditions. Compare this information with the performance measurement
baseline to identify variances at completion important to company
management and any applicable customer reporting requirements, including
statements of funding requirements.
Revisions and data 28. Incorporate authorized changes in a timely manner,
recording the effects of such changes in budgets
maintenance and schedules. In the directed effort prior to negotiation of
a change, base such revisions on the amount estimated and budgeted to the
program organizations.
29. Reconcile current budgets to prior budgets in terms of changes to the
authorized work and internal replanning in the detail needed by management
for effective control.
30. Control retroactive changes to records pertaining to work performed
that would change previously reported amounts for actual costs, earned
value, or budgets. Adjustments should be made only for correction of
errors, routine accounting adjustments, effects of customer or management
directed changes, or to improve the baseline integrity and accuracy of
performance measurement data.
31. Prevent revisions to the program budget except for authorized changes.
32. Document changes to the performance measurement baseline.
Source: Interim Defense Acquisition Guide Book, Appendix 4.
Appendix IV
Description of Earned Value Management
The standard format for tracking earned value is through a cost
performance report (CPR). The CPR is a monthly compilation of cost,
schedule, and technical data, which displays the performance measurement
baseline, any cost and schedule variances from that baseline, the amount
of management reserve used to date, the portion of the contract that is
authorized unpriced work, and the contractor's latest revised estimate to
complete the program. As a result, the CPR can be used as an effective
management tool because it provides the program manager with early warning
of potential cost and schedule overruns.
Using data from the CPR, a program manager can assess trends in cost and
schedule performance. This information is useful because trends tend to
continue and can be difficult to reverse. Studies have shown that once
programs are 15 percent complete, the performance indicators are
indicative of the final outcome. For example, a CPR showing a negative
trend for schedule status would indicate that the program is behind
schedule. By analyzing the CPR, one could determine the cause of the
schedule problem such as delayed flight tests, changes in requirements, or
test problems because the CPR contains a section that describes the
reasons for the negative status. A negative schedule can be a predictor of
later cost problems because additional spending is often necessary to
resolve problems. CPR data also provide the basis for independent
assessments of a program's cost and schedule status and can be used to
project final costs at completion in addition to determining when a
program should be completed.
Examining a program's management reserves is another way that a program
can use a CPR to determine potential issues early on. Management reserves,
which are funds that may be used as needed, provide flexibility to cope
with problems or unexpected events. EVM experts agree that transfers of
management reserves should be tracked and reported because they are often
problem indicators. An alarming situation arises if the CPR shows that the
management reserves are being used at a faster pace than the program is
progressing toward completion. For example, a problem would be indicated
if a program has used 80 percent of its management reserves, but only
completed 40 percent of its work. A program's management reserves should
contain at least 10 percent of the cost to complete a program so that
funds will always be available to cover future unexpected problems that
are more likely to surface as the program moves into the testing and
evaluation phase.
Appendix V
Comments from the National Aeronautics and Space Administration
Appendix V
Comments from the National Aeronautics
and Space Administration
Appendix V
Comments from the National Aeronautics
and Space Administration
Appendix V
Comments from the National Aeronautics
and Space Administration
Appendix V
Comments from the National Aeronautics
and Space Administration
Appendix V
Comments from the National Aeronautics
and Space Administration
Appendix V
Comments from the National Aeronautics
and Space Administration
Appendix VI
GAO Contact and Staff Acknowledgments
GAO Contact Allen Li (202) 512-4841
Acknowledgments Staff making key contributions to this report were Jerry
Herley, Shirley Johnson, Charles Malphurs, Karen Sloan, Madhav Panwar,
Karen Richey, Jennifer Echard, and Deborah Lott.
GAO's Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htmWaste, and Abuse in E-mail:
[email protected] Federal Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Service Requested
*** End of document. ***