NASA: Ares I and Orion Project Risks and Key Indicators to
Measure Progress (03-APR-08, GAO-08-186T).
The National Aeronautics and Space Administration (NASA) is in
the midst of two new development efforts as part of the
Constellation Program--the Ares I Crew Launch Vehicle and the
Orion Crew Exploration Vehicle. These projects are critical to
the success of the overall program, which will return humans to
spaceflight after Space Shuttle retirement in 2010. To reduce the
gap in human spaceflight, NASA plans to launch Ares I and Orion
in 2015--5 years after the Shuttle's retirement. GAO has issued a
number of reports and testimonies that touch on various aspects
of NASA's Constellation Program, particularly the development
efforts underway for the Orion and Ares I projects. These reports
and testimonies have questioned the affordability and overall
acquisition strategy for each project. NASA has revised the Orion
acquisition strategy and delayed the Ares I preliminary design
review based on GAO's recommendations in these reports. In
addition, GAO continues to monitor these projects on an ongoing
basis at the request of members of Congress. Based on this work,
GAO was asked to testify on the types of challenges that NASA
faces in developing the Ares I and Orion vehicles and identify
the key indicators that decision makers could use to assess risks
associated with common trouble spots in development. The
information in this testimony is based on work completed in
accordance with generally accepted government auditing standards.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-08-186T
ACCNO: A81552
TITLE: NASA: Ares I and Orion Project Risks and Key Indicators
to Measure Progress
DATE: 04/03/2008
SUBJECT: Aerospace engineering
Aerospace industry
Cost analysis
Operational testing
Performance measures
Product evaluation
Program evaluation
Program management
Requirements definition
Research and development
Risk assessment
Risk factors
Risk management
Schedule slippages
Space exploration
Strategic planning
Systems analysis
Systems design
Technology
Test facilities
Cost estimates
Program goals or objectives
Ares I Crew Launch Vehicle
NASA Constellation Program
Orion Crew Exploration Vehicle
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-08-186T
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Subcommittee on Space and Aeronautics, Committee on Science
and Technology, House of Representatives:
United States Government Accountability Office:
GAO:
For Release on Delivery Expected at 10:00 a.m. EDT:
Thursday, April 3, 2008:
NASA:
Ares I and Orion Project Risks and Key Indicators to Measure Progress:
Statement of Cristina T. Chaplain, Director:
Acquisition and Sourcing Management:
NASA:
GAO-08-186T:
GAO Highlights:
Highlights of GAO-08-186T, a testimony before the Subcommittee on Space
and Aeronautics, Committee on Science and Technology, House of
Representatives.
Why GAO Did This Study:
The National Aeronautics and Space Administration (NASA) is in the
midst of two new development efforts as part of the Constellation
Program�the Ares I Crew Launch Vehicle and the Orion Crew Exploration
Vehicle. These projects are critical to the success of the overall
program, which will return humans to spaceflight after Space Shuttle
retirement in 2010. To reduce the gap in human spaceflight, NASA plans
to launch Ares I and Orion in 2015�5 years after the Shuttle�s
retirement.
GAO has issued a number of reports and testimonies that touch on
various aspects of NASA�s Constellation Program, particularly the
development efforts underway for the Orion and Ares I projects. These
reports and testimonies have questioned the affordability and overall
acquisition strategy for each project. NASA has revised the Orion
acquisition strategy and delayed the Ares I preliminary design review
based on GAO�s recommendations in these reports. In addition, GAO
continues to monitor these projects on an ongoing basis at the request
of members of Congress. Based on this work, GAO was asked to testify on
the types of challenges that NASA faces in developing the Ares I and
Orion vehicles and identify the key indicators that decision makers
could use to assess risks associated with common trouble spots in
development. The information in this testimony is based on work
completed in accordance with generally accepted government auditing
standards.
What GAO Found:
NASA is currently working toward preliminary design reviews for the
Ares I and Orion vehicles. While this is a phase for discovery and risk
reduction, there are considerable unknowns as to whether NASA�s plans
for these vehicles can be executed within schedule goals and what these
efforts will ultimately cost. This is primarily because NASA is still
in the process of defining many performance requirements. Such
uncertainties could affect the mass, loads, and weight requirements for
the vehicles. NASA is aiming to complete this process in 2008, but it
will be challenged to do so given the level of knowledge that still
needs to be attained. The challenges NASA is facing pose risks to the
successful outcome of the projects. For example:
* Both vehicles have a history of weight issues;
* Excessive vibration during launch threatens system design;
* Uncertainty about how flight characteristics will be impacted by a
fifth segment added to the Ares I launch vehicle;
* Ares I upper stage essentially requires development of a new engine;
* No industry capability currently exists for producing the kind of
heat shields that the Orion will need for protecting the crew
exploration vehicle when it reenters Earth�s atmosphere; and
* Existing test facilities are insufficient for testing Ares I�s new
engine, for replicating the engine�s vibration and acoustic
environment, and for testing the thermal protection system for the
Orion vehicle.
All these unknowns, as well as others, leave NASA in the position of
being unable to provide firm cost estimates for the projects at this
point. Meanwhile, tight deadlines are putting additional pressure on
both the Ares I and Orion projects. Future requirements changes raise
risks that both projects could experience cost and schedule problems.
GAO�s past work on space systems acquisition and the practices of
leading developers identifies best practices that can provide decision
makers with insight into the progress of development at key junctures,
facilitate Congressional oversight, and support informed decision
making. This work has also identified common red flags throughout
development, which decision makers need to keep in mind when assessing
the projects. They include:
Key indicators: Weight growth is often among the highest drivers of
cost growth. Unanticipated software complexity, often indicated by
increases in the number of lines of code, can portend cost and schedule
growth.
Key junctures: The preliminary design review, critical design review,
and production review are key junctures that involve numerous steps and
help focus the agency on realistic accomplishments within reachable
goals. A disciplined approach aligned with key indicators can provide
the knowledge needed to make informed investment decisions at each
review.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-186T]. For more
information, contact Cristina T. Chaplain at (202) 512-4841 or
[email protected].
[End of section]
Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today to discuss challenges that the National
Aeronautics and Space Administration (NASA) faces in developing the
systems to achieve its goals for the President's Vision for Space
Exploration.[Footnote 1] We have been focusing our work primarily on
the Ares I Crew Launch Vehicle and the Orion Crew Exploration
Vehicle[Footnote 2], as they are among the first major efforts
conducted as part of NASA's Constellation Program to support
implementation of the Vision and represent a substantial investment for
NASA. Over $7 billion in contracts has already been awarded--and nearly
$230 billion is estimated to be ultimately spent over the next two
decades. Moreover, NASA is under pressure to develop the vehicles
quickly, as the Space Shuttle's retirement in 2010 means that there
could be at least a 5-year gap in our nation's ability to send humans
to space.
In summary, NASA is currently working toward preliminary design reviews
for the vehicles--a milestone that successful development organizations
use to make hard decisions about whether a program should proceed with
development. While this is a phase for discovery and risk reduction,
there are considerable unknowns as to whether NASA's plans for the Ares
I and Orion vehicles can be executed within schedule goals, as well as
what these efforts will ultimately cost. In fact, we do not know yet
whether the architecture and design solutions selected by NASA will
work as intended. This is primarily because NASA is still in the
process of defining both of the projects' performance requirements and
some of these uncertainties could affect the mass, loads, and weight
requirements for the vehicles. It is also working through significant
technical risks, such as oscillation within the first stage of the Ares
I vehicle, which computer modeling indicates could cause unacceptable
structural vibrations.
NASA is aiming to complete preliminary design reviews for the Ares I
and Orion this year, scheduled for August 2008 Ares I and September
2008 respectively, but it will be challenged in doing so given the
level of knowledge that still needs to be attained. In addition, to
minimize the gap in human spaceflight caused by the shuttle's
retirement, there is a high degree of concurrency within the projects.
Our prior work has shown that concurrent development, especially when
new technologies are involved, increases the risk that significant
problems will be discovered as the systems' designs are integrated that
could result in cost and schedule delays. NASA's schedule leaves little
room for the unexpected. If something goes wrong with the development
of the Ares I or the Orion, the entire Constellation Program could be
thrown off course and the return to human spaceflight delayed.
NASA recognizes the risks involved with its approach and has taken
steps to mitigate some of these risks. It is important that, in
mitigating risks, NASA continually assess the viability of its plans
for the Ares I and Orion. The current state of play requires that NASA
remain open to the possibility that it may need to revisit decisions on
its architecture and design as these vehicles are expected to be in use
for decades to come and decisions made now will have long-term
consequences.
Moreover, with additional significant investment decisions still ahead,
it is important that agency decision makers and Congress maintain clear
insight into the progress the projects are making as well as any
potential problems. This type of oversight is important, not just for
the Ares I and Orion vehicles, but for the entire future exploration
effort--since resources available to fund the Vision are constrained,
as competition for resources increases within the federal government
over the next several decades. In this regard, our work has identified
specific markers that can be used to (1) assess NASA's progress in
closing critical knowledge gaps and (2) identify issues that could
result in cost growth, schedule delays, or decreased performance. In
other words, they can be used to assess whether there is a viable
business case for pressing forward with the projects.
We have issued a number of reports and testimonies that touch on
various aspects of NASA's Constellation Program and in particular the
development efforts underway for the Orion and Ares I projects. These
reports and testimonies have questioned the affordability and overall
acquisition strategy for each project. In July 2006 we recommended that
NASA modify the Orion Crew Vehicle acquisition strategy to ensure the
agency did not commit itself to a long-term contractual obligation
prior to establishing a sound business case. Although initially NASA
disagreed with our recommendation, the agency subsequently revised its
acquisition strategy to address some of the concerns we raised. In
October 2007 we recommended that NASA develop a sound business case
supported by firm requirements, mature technologies, a preliminary
design, a realistic cost estimate, and sufficient funding and time--
before proceeding beyond preliminary design review. NASA concurred with
this recommendation and subsequently slipped the Ares I preliminary
design review from July 2008 to August 2008.
My statement today is based on these products, as well as updated
information based on our continual monitoring of the projects at the
request of members of Congress. To conduct these reviews, we analyzed
relevant project documentation, prior GAO reports, NASA documents, and
contractor information; interviewed program and project officials; and
reviewed NASA's risk management system for the Constellation Program.
Based on this work, my statement will specifically address the
challenges that NASA faces developing the Ares I and Orion vehicles
with regard to requirements definition, technology and hardware gaps,
cost and schedule estimates, and facilities needs. Further, I will
provide key indicators that decision makers could use to assess risks
as the two development efforts move forward. We conducted this
performance audit from October 2007 through April 2008 in accordance
with generally accepted government auditing standards. Those standards
require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions
based on our audit objectives.
Background:
In September 2005, NASA outlined an initial framework for implementing
the President's Vision for Space Exploration in its Exploration Systems
Architecture Study. NASA is now implementing the recommendations from
this study within the Constellation Program, which includes three major
development projects--the Ares I Crew Launch Vehicle, the Orion Crew
Exploration Vehicle, and the Ares V Cargo Launch Vehicle as shown in
figure 1.
Figure 1: Overview of Ares I and Orion Projects:
This figure is a combination of two photographs as part of an overview
of Ares I and Orion Projects. The Ares V Cargo Launch Vehicle and Ares
I Crew Launch Vehicle are pictured.
[See PDF for image]
Source: GAO analysis and presentation of NASA photos and data.
[End of figure]
To reduce cost and minimize risk in developing these projects, NASA
planned to maximize the use of heritage systems and technology. Since
2005, however, NASA has made changes to the basic architecture for the
Ares I and Orion designs[Footnote 3] that have resulted in the
diminished use of heritage systems. This is due to the ability to
achieve greater cost savings with alternate technology and the
inability to recreate heritage technology. For example, the initial
design was predicated on using the main engines and the solid rocket
boosters from the Space Shuttle Program. However, NASA is no longer
using the Space Shuttle Main Engines because greater long-term cost
savings are anticipated through the use of the J-2X engine. In another
example, NASA increased the number of segments on the Ares I first-
stage reusable solid rocket booster from four to five to increase
commonality between the Ares I and Ares V, and eliminate the need to
develop, modify, and certify both a four-segment reusable solid rocket
booster and an expendable Space Shuttle main engine for the Ares I.
Finally, according to the Orion program executive the Orion project
originally intended to use the heat shield from the Apollo program as a
fallback technology for the Orion thermal protection system, but was
unable to recreate the Apollo material.
NASA has authorized the Ares I and Orion projects to proceed with
awarding development contracts. In April 2006, NASA awarded a $1.8
billion contract for design, development, test, and evaluation of the
Ares I first stage to Alliant Techsystems. NASA also awarded a $1.2
billion contract for design, development, test, and evaluation of the
Ares I upper stage engine--the J-2X--to Pratt and Whitney Rocketdyne in
June 2006. NASA is developing the upper stage and the upper stage
instrument unit, which contains the control systems and avionics for
the Ares I, in-house. However, NASA awarded a $514.7 million contract
for design support and production of the Ares I upper stage to the
Boeing Company in August 2007. In August 2006, NASA awarded Lockheed
Martin a $3.9 billion contract to design, test, and build the Orion
crew exploration vehicle.[Footnote 4] According to NASA, the contract
was modified in April 2007, namely by adding 2 years to the design
phase and two test flights of Orion's launch abort system and by
deleting the production of an cargo variant for the International Space
Station. NASA indicates that these changes increased the contract value
to $4.3 billion. Federal procurement data shows that an additional
modification has been signed which increased the value of the contract
by an additional $59 million.
NASA has completed or is in the process of completing key reviews on
both the Ares I and Orion projects. NASA has completed the system
requirements review for each project and is in the midst of finalizing
the system definition reviews.[Footnote 5] At the systems requirements
review, NASA establishes a requirements baseline that serves as the
basis for ongoing design analysis work and systems testing. Systems
definition reviews focus on emerging designs for all transportation
elements and compare the predicted performance of each element against
the currently baselined requirements. Figure 2 shows the timeline for
Ares I and Orion critical reviews.
Figure 2: Timeline for Ares I and Orion Critical Reviews (in fiscal
years):
This figure is a timeline for Ares I and Orion critical reviews in
fiscal years.
Areas I crew launch vehicle;
System requirements review: December 2006;
Preliminary design review: August 2008;
Critical design review: March 2010.
Orion crew exploration vehicle;
System requirements review: March 2007;
Preliminary design review: September 2008;
Critical design review: October 2009.
[See PDF for image]
Source: GAO analysis of NASA data.
[End of figure]
NASA is using its Web-based Integrated Risk Management Application to
help monitor and mitigate the risks with the Ares I and Orion
development efforts and for the overall Constellation Program. The risk
management application identifies and documents risks, categorizes
risks--as high, medium, and low based on both the likelihood of an
undesirable event as well as the consequences of that event to the
project--and tracks performance against mitigation plans. For the Ares
I project, the application is tracking 101 risks, 36 of which are
considered high-risk areas.[Footnote 6] For the Orion project, NASA is
tracking 193 risks, including 71 high-risk areas.[Footnote 7] NASA is
developing and implementing plans to mitigate some of these risks.
Requirements Setting is a Primary Challenge for Both the Ares I and
Orion Projects:
Although project level requirements were baselined at both systems
requirements reviews, continued uncertainty about the systems'
requirements have led to considerable unknowns as to whether NASA's
plans for the Ares I and Orion vehicles can be executed within schedule
goals, as well as what these efforts will ultimately cost. Such
uncertainty has created knowledge gaps that are affecting many aspects
of both projects. Because the Orion vehicle is the payload that the
Ares I must deliver to orbit, changes in the Orion design, especially
those that affect weight, directly affect Ares I lift requirements.
Likewise, the lift capacity of the Ares I drives the Orion design. Both
the Orion and Ares I vehicles have a history of weight and mass growth,
and NASA is still defining the mass, loads, and weight requirements for
both vehicles. According to agency officials, continuing weight growth
led NASA to rebaseline the Orion vehicle design in fall 2007. This
process involved "scrubbing" the Orion Vehicle to establish a zero-
based design capable of meeting minimal mission requirements but not
safe for human flight. Beginning with the zero-based design NASA first
added back the systems necessary to ensure crew safety and then
conducted a series of engineering trade-offs to determine what other
systems should be included to maximize the probability of mission
success while minimizing the system's weight. As a result of these
trade-offs, NASA modified the requirement for nominal landing on land
to nominal landing in water, thereby gaining 1500 lbs of trade space in
the Orion design.
NASA recognizes that continued weight growth and requirements
instability are key risks facing the Orion project and that continued
instability in the Orion design is a risk facing the Ares I project.
The Ares I and Orion projects are working on these issues but have not
yet finalized requirements or design. Our previous work on systems
acquisition work shows that the preliminary design phase is an
appropriate place to conduct systems engineering to support requirement
and resource trade-off decisions. For the Ares I project, this phase is
scheduled to be completed in August 2008, whereas for the Orion
project, it is September 2008--leaving NASA only 4 and 5 months
respectively to close gaps in requirements knowledge. NASA will be
challenged to close such gaps, given that it is still defining
requirements at a relatively high level and much work remains to be
done at the lower levels. Moreover, given the complexity of the Orion
and Ares I efforts and their interdependencies, as long as requirements
are in flux, it will be extremely difficult to establish firm cost
estimates and schedule baselines.
Technology and Hardware Gaps Along with Requirements Uncertainty are
Increasing Risk:
Currently, nearly every major segment of Ares I and Orion faces
knowledge gaps in the development of required hardware and technology
and many are being affected by uncertainty in requirements. For
example, computer modeling is showing that thrust oscillation within
the first stage of the Ares I could cause excessive vibration
throughout the Ares I and Orion. Resolving this issue could require
redesigns to both the Ares I and Orion vehicles that could ultimately
impact cost, schedule, and performance. Furthermore, the addition of a
fifth segment to the Ares I first stage has the potential to impact
qualification efforts for the first stage and could result in costly
requalification and redesign efforts. Additionally, the J-2X engine
represents a new engine development effort that, both NASA and Pratt
and Whitney Rocketdyne recognize, is likely to experience failures
during development. Addressing these failures is likely to lead to
design changes that could impact the project's cost and schedule. With
regard to the Orion project, there is currently no industry capability
for producing a thermal protection system of the size required by the
Orion. NASA has yet to develop a solution for this gap, and given the
size of the vehicle and the tight development schedule, a feasible
thermal protection system may not be available for initial operational
capability to the space station. The table 1 describes these and other
examples of knowledge gaps in the development of the Ares I and Orion
vehicles.
Table 1: Examples of Ares I and Orion Technology and Hardware Gaps:
Ares I Crew Launch Vehicle;
First stage;
Current modeling indicates that thrust oscillation within the first
stage causes unacceptable structural vibrations. There is a possibility
that the thrust oscillation frequency and magnitude may be outside the
design limits of the Ares design requirements. A NASA focus team
studied this issue and has proposed options for mitigation including
incorporating vibration absorbers into the design of the first stage
and redesigning portions of the Orion Vehicle to isolate the crew from
the vibration. Further, it is unknown how the addition of a fifth
segment to the launch vehicle will affect flight characteristics.
Failure to completely understand the flight characteristics of the
modified booster could create a risk of hardware failure and loss of
vehicle control. Ares I relies on hardware adapted from the Space
Shuttle program that may not meet qualification requirements.
Qualification requirements may be difficult to meet due to the new
ascent loads (the physical strain on the spacecraft during launch) and
vibration and acoustic environments associated with the Ares I.
Resulting redesign and requalification efforts could affect cost and
schedule. NASA is currently working to further define the vibration and
acoustic environment.
Ares I Crew Launch Vehicle;
Upper stage;
NASA redesigned the upper stage configuration from two completely
separate propellant tanks to two tanks with one common bulkhead. The
prior configuration employed a simpler design with a lower
manufacturing cost but did not meet mass requirements. The current
common bulkhead design involves a complex and problematic manufacturing
process that challenged earlier development efforts on the Apollo
program. In fact, NASA's Web-based Integrated Risk Management
Application indicated that one of the lessons learned from the Apollo
program was not to use common bulkheads because they are complex and
difficult to manufacture.
Ares I Crew Launch Vehicle;
J-2X upper stage engine;
Although the J-2X is based on the J-2 and J-2S engines used on the
Saturn V and leverages knowledge from the X-33 and RS-68, the number of
planned changes is such that, according to NASA review boards, the
effort essentially represents a new engine development. NASA and Pratt
and Whitney Rocketdyne recognize that some level of developmental
problems are inherent in all new engine development programs. As such,
the project has predicted that the J-2X development will require 29
rework cycles. In addition, the J-2X faces extensive redesign to
incorporate modern controls, achieve increased performance
requirements, and meet human rating standards. The J-2X developers also
face significant schedule risks in developing and manufacturing a
carbon composite nozzle extension needed to satisfy thrust
requirements. According to contractor officials, the extension is more
than 2 feet--i.e., about one-third--wider in diameter than existing
nozzle extensions.
Orion Crew Exploration Vehicle;
Launch abort system;
Technology development of the launch abort system is being conducted
concurrently with design of Orion. Ongoing requirements changes related
to the Orion system and its subsystems or development setbacks could
(1) prevent some test objectives from being adequately demonstrated
during early launch abort system tests, (2) drive the need for
additional testing of the abort system, or (3) lead to design revisions
or changes to the required number of spares. Any of these possibilities
could lead to increased program costs and delays to the flight test
schedule. According to NASA officials, the agency is currently
assessing alternative designs for the launch abort system to address
weight and vibration concerns.
Orion Crew Exploration Vehicle;
Thermal protection system;
The Orion requires the development of a large-scale ablative thermal
protection system. Given the size of the vehicle and the tight
development schedule, a feasible thermal protection system may not be
available in time for the Orion initial operational capability to the
space station. There is currently no industry capability for producing
a thermal protection system of the size required by the Orion.
Furthermore, heat shield design features required by the Orion, namely
the size, have never been proven and must be developed. NASA is
currently conducting an advanced development project to mature
technologies necessary to meet thermal protection system requirements.
Source: GAO analysis of NASA data.
[End of table]
Constellation Cost Estimates Are Preliminary Due to Requirements
Uncertainty:
NASA's preliminary cost estimates for the Constellation Program are
likely to change when requirements are better defined. NASA will
establish a preliminary estimate of life cycle costs for the Ares I and
Orion in support of each project's system definition review. A formal
baseline of cost, however, is not expected until the projects'
preliminary design reviews are completed. NASA is working under a self-
imposed deadline to deliver the new launch vehicles no later than 2015
in order to minimize the gap in human spaceflight between the Space
Shuttle's retirement in 2010 and the availability of new transportation
vehicles. The Constellation Program's budget request maintains a
confidence level of 65 percent (i.e., NASA is 65 percent certain that
the actual cost of the program will either meet or be less than the
estimate) for program estimates based upon a 2015 initial operational
capability. Internally, however, the Ares I and Orion projects are
working toward an earlier initial operational capability (2013), but at
a reduced budget confidence level--33 percent. However, NASA cannot
reliably estimate the money needed to complete technology development,
design, and production for the Ares I and Orion projects until
requirements are fully understood.
NASA has identified the potential for a life cycle cost increase as a
risk for the Orion program. According to NASA's risk database, given
the historical cost overruns of past NASA systems and the known level
of uncertainty in the current Orion requirements, there is a
possibility that Orion's life cycle cost estimate may increase over
time. NASA acknowledges that such increases are often caused by the
unknown impacts of decisions made during development. One factor
currently contributing to cost increases is the addition of new
requirements. NASA is working to formulate the best life cycle cost
estimate possible during development, is identifying and monitoring
costs threats, and is implementing management tools all aimed at
addressing this risk.
Schedule Pressures Add Additional Risks for Ares I and Orion:
There are considerable schedule pressures facing both the Ares I and
Orion projects. These are largely rooted in NASA's desire to minimize
the gap between the retirement of the space shuttle and availability of
the new vehicles. Because of this scheduling goal, NASA is planning to
conduct many interdependent development activities concurrently--
meaning if one activity should slip in schedule, it could have
cascading effects on other activities. Moreover, some aspects of the
program are already experiencing scheduling delays due to the fact that
high-level requirements are still being defined.
Ares I:
The development schedule for the J-2X is aggressive, allowing less than
7 years from development start to first flight, and highly concurrent.
Due to the tight schedule and long-lead nature of engine development,
the J-2X project was required to start out earlier in its development
than the other elements on the Ares I vehicle. This approach has
introduced a high degree of concurrency between the setting of overall
Ares I requirements and the development of the J-2X design and
hardware. Consequently, the engine development is out of sync with the
first stage and upper stage in the flow-down and decomposition of
requirements, an approach our past work has shown to be fraught with
risk. NASA acknowledges that the engine development is proceeding with
an accepted risk that future requirements changes may affect the engine
design and that the engine may not complete development as scheduled in
December 2012. The J-2X development effort represents a critical path
for the Ares I project. Subsequently, delays in the J-2X schedule for
design, development, test, and evaluation would have a ripple effect of
cost and schedule impacts throughout the entire Ares I project.
The schedule for the first stage also presents a potential issue for
the entire Ares I project. Specifically, the critical design review for
the first stage is out of sync with the Ares I project-level critical
design review. NASA has scheduled two critical design reviews for the
first stage. The first critical design review is scheduled for November
2009, 5 months before the Ares I project critical design review. At
this point, however, the project will not have fully tested the first
stage development motors. The second critical design review, in
December 2010, occurs after additional testing of developmental motors
is conducted. By conducting the Ares I critical design review before
the first stage project critical design review, the project could
prematurely begin full-scale test and integration activities a full 9
months before the first stage design has demonstrated maturity. If
problems are found in the first stage design during the later testing,
implementing solutions could result in costly rework and redesign and
delay the overall project schedule.
Orion:
Cost and schedule reporting on the Orion project indicates that the
Orion project's efforts to mature requirements and design and to
resolve weight issues is placing pressure on the Orion schedule.
Specifically, activities aimed at assessing alternate designs to reduce
overall vehicle mass, rework to tooling concepts, and late requirements
definition have contributed to the project falling behind schedule.
Further, the Orion risk system indicates that schedule delays
associated with testing may occur. The current Orion design has high
predicted vibration and acoustic levels. Historically, components
designed and qualified for uncertain vibration and acoustic
environments have resulted in some failures and required subsequent
redesign and retest. Failures during qualification testing of Orion
components may lead to schedule delays associated with redesigning
components.
NASA's Administrator has publicly stated that if Congress provided the
Agency an additional $2 billion that NASA could accelerate the
Constellation program's initial operational capability date to 2013. We
believe that this assessment is highly optimistic. The development
schedule for the J-2X engine, the critical path for the Ares I
development, is already recognized as aggressive, allowing less than 7
years for development. The development of the Space Shuttle Main engine
by comparison took 9 years. Further, NASA anticipates that the J-2X
engine is likely to require 29 rework cycles to correct problems
identified during testing. Given the linear nature of a traditional
test-analyze-fix-test cycle, even large funding increases offer no
guarantee of program acceleration, particularly when the current
schedule is already compressed and existing NASA test facilities are
already maximized.
Test Facilities for Ares I and Orion Insufficient:
According to NASA, at this time, existing test facilities are
insufficient to adequately test the Ares I and Orion systems. Existing
altitude test facilities are insufficient to test the J-2X engine in a
relevant environment. To address this issue, NASA is in the process of
constructing a new altitude test facility at Stennis Space Center for
the J-2X. Also, current facilities are inadequate to replicate the
Orion vibration and acoustic environment. Further, Pratt and Whitney
Rocketdyne--the J-2 X upper stage engine contractor--indicated that
existing test stands that could support J-2X testing will be tied up
supporting the Space Shuttle program until 2010. NASA has taken steps
to mitigate J-2X risks by increasing the amount of component-level
testing, procuring additional development hardware and test facilities,
and working to make a third test stand available to the contractor
earlier than originally planned. NASA has compensated for this schedule
pressure on the Ares I project by adding funds for testing and other
critical activities. But it is not certain that added resources will
enable NASA to deliver the Ares I when expected.
With respect to Orion's thermal protection system, facilities available
from the Apollo era for testing large-scale heat shields no longer
exist. Therefore, NASA must rely on two facilities that fall short in
providing the necessary capability and scheduling to test ablative
materials needed for Orion. Additionally, NASA has no scheduled test to
demonstrate the thermal protection system needed for lunar missions.
NASA is exploring other options, including adding a lunar return flight
test and building a new improved test facility. Due to the scheduled
first lunar flight, any issues identified during such testing would
need to be addressed in the time between the flight test and the first
flight.
Oversight Based on Best Practices and Key Indicators Important for
Program Success:
NASA is poised to invest a significant amount of resources to implement
the Vision over the long term and specifically to develop the Ares I
and Orion projects over the next several years. Accordingly, you asked
us to articulate indicators that Congress could use to assess progress.
Our prior work has shown that investment decisions of this magnitude
need to be based on an established and executable business case and
that there are several key indicators that Congress could be informed
of to assess progress throughout development. These include areas
commonly underestimated in space programs, such as weight growth and
software complexity, as well as indicators used by best practice
organizations to assess readiness to move forward in the development
cycle. Space programs which we have studied in detail in the past have
tended to underestimate cost in some of these areas.
Weight Growth:
Our previous work on government-funded space systems has shown that
weight growth is often not anticipated even though it is among the
highest drivers of cost growth for space systems. Weight growth can
affect the hardware needed to support a system, and, in the case of
launch vehicles, the power or thrust required for the system. As the
weight of a particular system increases, the power or thrust required
for that system will also increase. This could result in the need to
develop additional power or thrust capability to lift the system,
leading to additional costs, or to stripping down the vehicle to
accommodate current power or thrust capability. For example, NASA went
through the process to zero-base the design for the Orion to address
weight concerns. Continual monitoring of system weight and required
power/thrust, as well as margins or reserves for additional growth, can
provide decision makers with an indicator of whether cost increases can
be anticipated.
Software Complexity:
The complexity of software development on a system, often denoted by
the number of lines of code on a system, can also be used as an
indicator to monitor whether a program will meet cost and schedule
goals. In our work on software development best practices, we have
reported that the Department of Defense has attributed significant cost
and schedule overruns on software-intensive systems to developing and
delivering software. Generally, the greater the number of lines of
code, the more complicated the system development. Changes to the
amount of code needed to be produced can indicate potential cost and
schedule problems. Decision makers can monitor this indicator by
continually asking for information on the estimated amount of code
needed on a system and inquiring about any increases in need and their
impact on cost and schedule.
There are other areas, such as the use of heritage systems and
industrial base capability that are commonly underestimated in space
programs as well. However, weight increases and software growth are
more quantifiable and thus useful for oversight purposes.
Indicators that Can be Used to Assess Knowledge Gap at Key Junctures:
Additionally, since the mid-1990s, GAO has studied the best practices
of leading commercial companies. On the basis of this information, and
taking into account the differences between commercial product
development and major federal acquisitions, we have outlined a best
practices product development model--known as a knowledge-based
approach to system development. This type of approach calls for
investment decisions to be made on the basis of specific, measurable
levels of knowledge at critical junctures before investing more money
and proceeding with development.
Importantly, our work has shown the most leveraged decision point is
matching the customer's needs with the developer's resources (time,
dollars, technology, people, etc.) because it sets the stage for the
eventual outcome--desirable or problematic. The match is ultimately
achieved in every development program, but in successful development
programs, it occurs before product development is formally initiated
(usually the preliminary design review). If the knowledge attained at
this and other critical junctures does not confirm the business case on
which the acquisition was originally justified, the best practice
organizations we have studied do not allow the program to go forward.
We have highlighted the three critical junctures at which developers
must have knowledge to make large investment decisions--the preliminary
design review, the critical design review, and the production review--
and the numerous key indicators that can be used to increase the
chances of successful outcomes.
In assessing the Orion and Ares programs, the Congress and NASA
decision-makers can use these indicators in order to reliably gauge
whether there is a sufficient business case for allowing the programs
to proceed forward.
Preliminary design review: Before product development is started, a
match must be made between the customers' needs and the available
resources--technical and engineering knowledge, time, and funding. To
provide oversight at this juncture, NASA could provide Congress with
information to verify that the following have indicators been met:
* All critical technologies are demonstrated to a high level of
technology maturity, that is demonstrated that they can perform in a
realistic or, more preferably, operational environment. A technology
readiness level 6 or 7 would indicate that this has been achieved. One
approach to ensure that technology readiness is reliably assessed is to
use independent testing;
* Project requirements are defined and informed by the systems
engineering process;
* Cost and schedule estimates established for the project are based on
knowledge from the preliminary design using systems engineering tools;
* Additional resources are in place, including needed workforce, and a
decision review is conducted following completion of the preliminary
design review.
A critical enabler for success in this phase of development is
performance and requirements flexibility. Customers and product
developers both need to be open to reducing expectations, deferring
them to future programs, or to investing more resources up front to
eliminate gaps between resources and expectations. In successful
programs we have studied, requirements were flexible until a decision
was made to commit to product development because both customers and
developers wanted to limit cycle time. This makes it acceptable to
reduce, eliminate, or defer some customer wants so that the product's
requirements could be matched with the resources available to deliver
the product within the desired cycle time.
Critical design review: A product's design must demonstrate its ability
to meet performance requirements and be stable about midway through
development. To provide oversight at this juncture, NASA could provide
Congress with information to verify that the following indicators have
been met:
* At least 90 percent of engineering drawings are complete;
* All subsystem and system design reviews have been completed;
* The design meets requirements demonstrated through modeling,
simulation, or prototypes;
* Stakeholders' concurrence that drawings are complete and producible
is obtained;
* Failure modes and effects analysis have been completed;
* Key system characteristics are identified;
* Critical manufacturing processes are identified;
* Reliability targets are established and a growth plan based on
demonstrated reliability rates of components and subsystems is
developed; and:
* A decision review is conducted following the completion of the
critical design review.
Production Review: The developer must show that the product can be
manufactured within cost, schedule, and quality targets and is
demonstrated to be reliable before production begins. To provide
oversight at this juncture, NASA could provide Congress with
information to verify that the following indicators have been met:
* Manufacturing processes have been demonstrated;
* Production representative prototypes have been built;
* Production representative prototypes have been tested and have
achieved reliability goals;
* Production representative prototypes have been demonstrated in an
operational environment through testing;
* Statistical process control data have been collected;
* Critical processes have been demonstrated to be capable and that they
are in statistical control;
* A decision review is conducted following completion of the production
readiness review.
Over the past 2 years, we have recommended that NASA incorporate a
knowledge-based approach in its policies and take steps to implement
this type of approach in its programs and projects.[Footnote 8] NASA
has incorporated some knowledge-based concepts into its acquisition
policies. For example, NASA now requires a decision review between each
major phase of the acquisition life cycle and has established general
entrance and success criteria for the decision reviews. In addition, we
have reported that this type of approach is being embraced by the Ares
I project.
Concluding Observations:
In conclusion, the President's Vision for Space Exploration is an
ambitious effort, not just because there will be technical and design
challenges to building systems needed to achieve the Vision's goals,
but because there are limited resources within which this can be
accomplished. Moreover, the long-term nature of the Vision means that
commitments for funding and to the goals of the Vision will need to be
sustained across presidential administrations and changes in
congressional leadership. For these reasons, it is exceedingly
important that the right decisions are made early on and that decision-
makers have the right knowledge going forward so that they can make
informed investment decisions.
In looking at the first major investments, the Ares I and Orion
projects, it is important to recognize that they are risky endeavors,
largely due to their complexity, scope, and interdependencies. It is
also important to recognize that the desire to minimize the gap in
human space flight adds considerable risk, since it could limit NASA's
ability to study emerging problems and pursue alternative ways of
addressing them. For these reasons, as well as the magnitude of
investment at stake, it is imperative that NASA be realistic and open
about the progress it is making and to be willing to make changes to
the architecture and design if technical problems can not be solved
without overly compromising performance. Additionally, Congress needs
to be well-informed about the extent to which knowledge gaps remain and
what tradeoffs or additional resources are needed to close those gaps
and to support changes if they are determined to be necessary. The
upcoming preliminary design review milestones represent perhaps the
most critical juncture where these assessments can take place and where
hard decisions can be made as to whether the programs should proceed
forward. It may well be the last opportunity to make significant
adjustments before billions of dollars are spent and long term
commitments become solidified.
Mr. Chairman, this concludes my prepared statement. I would be pleased
to answer any questions that you may have at this time.
GAO Contacts and Staff Acknowledgements:
For further questions about this statement, please contact Cristina T.
Chaplain at (202) 512-4841. Individuals making key contributions to
this statement include James L. Morrison, Meredith A. Kimmitt, Lily
Chin, Neil Feldman, Rachel Girshick, Shelby S. Oakley, and John S.
Warren, Jr.
[End of section]
Susan Becker, Acting Manager, [email protected], (202) 512-4800 U.S.
Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548:
Footnotes:
[1] The Vision includes a return to the moon that is intended
ultimately to enable future exploration of Mars and other destinations.
To accomplish this, NASA initially plans to (1) complete its work on
the international Space Station by 2010, fulfilling its commitment to
15 international partner countries; (2) begin developing a new manned
exploration vehicle to replace the space shuttle; and (3) return to the
moon in preparation for future, more ambitious missions.
[2] GAO, NASA: Agency Has Taken Steps Toward Making Sound Investment
Decisions for Ares I but Still Faces Challenging Knowledge Gaps, GAO-
08-51 (Washington, D.C.: Oct. 31, 2007) and GAO, NASA: Long-Term
Commitment to and Investment in Space Exploration Program Requires More
Knowledge, GAO-06-817R (Washington, D.C.: July 17, 2006).
[3] Heritage systems are systems with characteristics similar to the
one being developed. A heritage system is often the one the new program
is replacing.
[4] The actual value of the contract could be greater than $3.9 billion
if NASA exercises options on the contract for production and
sustainment or issues orders against the indefinite delivery/indefinite
quantity portion of the contract.
[5] The system requirements review is intended to examine the function
and performance requirements defined for the system and the preliminary
project plan and ensure that the requirements and the selected concept
will satisfy the mission. The system definition review examines the
proposed system design and the flow-down of that design to all
functional elements of the system. The system requirements review and
system definition review process culminates with key decision point B
wherein NASA determines the project's readiness to move forward.
[6] This is the total number of open risks for the Ares I project as of
March 25, 2008. It does not include risks that have been closed or
risks that NASA considers sensitive.
[7] This is the total number of open risks for the Orion project as of
March 25, 2008. It does not include risks that have been closed or
risks that NASA considers sensitive.
[8] AO, NASA: Implementing a Knowledge-Based Acquisition Framework
Could Lead to Better Investment Decisions and Project Outcomes, GAO-06-
218 (Washington, D.C.: Dec. 21, 2005); GAO, NASA: Long-Term Commitment
to and Investment in Space Exploration Program Requires More Knowledge,
GAO-06-817R (Washington, D.C.: July 17, 2006); and GAO, NASA's James
Webb Space Telescope: Knowledge-Based Acquisition Approach Key to
Addressing Program Challenges, GAO-06-634 (Washington, D.C.: July 14,
2006).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: [email protected]:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, [email protected]:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, [email protected]:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
*** End of document. ***