NASA: Ares I and Orion Project Risks and Key Indicators to	 
Measure Progress (03-APR-08, GAO-08-186T).			 
                                                                 
The National Aeronautics and Space Administration (NASA) is in	 
the midst of two new development efforts as part of the 	 
Constellation Program--the Ares I Crew Launch Vehicle and the	 
Orion Crew Exploration Vehicle. These projects are critical to	 
the success of the overall program, which will return humans to  
spaceflight after Space Shuttle retirement in 2010. To reduce the
gap in human spaceflight, NASA plans to launch Ares I and Orion  
in 2015--5 years after the Shuttle's retirement. GAO has issued a
number of reports and testimonies that touch on various aspects  
of NASA's Constellation Program, particularly the development	 
efforts underway for the Orion and Ares I projects. These reports
and testimonies have questioned the affordability and overall	 
acquisition strategy for each project. NASA has revised the Orion
acquisition strategy and delayed the Ares I preliminary design	 
review based on GAO's recommendations in these reports. In	 
addition, GAO continues to monitor these projects on an ongoing  
basis at the request of members of Congress. Based on this work, 
GAO was asked to testify on the types of challenges that NASA	 
faces in developing the Ares I and Orion vehicles and identify	 
the key indicators that decision makers could use to assess risks
associated with common trouble spots in development. The	 
information in this testimony is based on work completed in	 
accordance with generally accepted government auditing standards.
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-186T					        
    ACCNO:   A81552						        
  TITLE:     NASA: Ares I and Orion Project Risks and Key Indicators  
to Measure Progress						 
     DATE:   04/03/2008 
  SUBJECT:   Aerospace engineering				 
	     Aerospace industry 				 
	     Cost analysis					 
	     Operational testing				 
	     Performance measures				 
	     Product evaluation 				 
	     Program evaluation 				 
	     Program management 				 
	     Requirements definition				 
	     Research and development				 
	     Risk assessment					 
	     Risk factors					 
	     Risk management					 
	     Schedule slippages 				 
	     Space exploration					 
	     Strategic planning 				 
	     Systems analysis					 
	     Systems design					 
	     Technology 					 
	     Test facilities					 
	     Cost estimates					 
	     Program goals or objectives			 
	     Ares I Crew Launch Vehicle 			 
	     NASA Constellation Program 			 
	     Orion Crew Exploration Vehicle			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-186T

   

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Testimony: 

Before the Subcommittee on Space and Aeronautics, Committee on Science 
and Technology, House of Representatives: 

United States Government Accountability Office: 

GAO: 

For Release on Delivery Expected at 10:00 a.m. EDT: 

Thursday, April 3, 2008: 

NASA: 

Ares I and Orion Project Risks and Key Indicators to Measure Progress: 

Statement of Cristina T. Chaplain, Director: 

Acquisition and Sourcing Management: 

NASA: 

GAO-08-186T: 

GAO Highlights: 

Highlights of GAO-08-186T, a testimony before the Subcommittee on Space 
and Aeronautics, Committee on Science and Technology, House of 
Representatives. 

Why GAO Did This Study: 

The National Aeronautics and Space Administration (NASA) is in the 
midst of two new development efforts as part of the Constellation 
Programï¿½the Ares I Crew Launch Vehicle and the Orion Crew Exploration 
Vehicle. These projects are critical to the success of the overall 
program, which will return humans to spaceflight after Space Shuttle 
retirement in 2010. To reduce the gap in human spaceflight, NASA plans 
to launch Ares I and Orion in 2015ï¿½5 years after the Shuttleï¿½s 
retirement. 

GAO has issued a number of reports and testimonies that touch on 
various aspects of NASAï¿½s Constellation Program, particularly the 
development efforts underway for the Orion and Ares I projects. These 
reports and testimonies have questioned the affordability and overall 
acquisition strategy for each project. NASA has revised the Orion 
acquisition strategy and delayed the Ares I preliminary design review 
based on GAOï¿½s recommendations in these reports. In addition, GAO 
continues to monitor these projects on an ongoing basis at the request 
of members of Congress. Based on this work, GAO was asked to testify on 
the types of challenges that NASA faces in developing the Ares I and 
Orion vehicles and identify the key indicators that decision makers 
could use to assess risks associated with common trouble spots in 
development. The information in this testimony is based on work 
completed in accordance with generally accepted government auditing 
standards. 

What GAO Found: 

NASA is currently working toward preliminary design reviews for the 
Ares I and Orion vehicles. While this is a phase for discovery and risk 
reduction, there are considerable unknowns as to whether NASAï¿½s plans 
for these vehicles can be executed within schedule goals and what these 
efforts will ultimately cost. This is primarily because NASA is still 
in the process of defining many performance requirements. Such 
uncertainties could affect the mass, loads, and weight requirements for 
the vehicles. NASA is aiming to complete this process in 2008, but it 
will be challenged to do so given the level of knowledge that still 
needs to be attained. The challenges NASA is facing pose risks to the 
successful outcome of the projects. For example: 

* Both vehicles have a history of weight issues; 

* Excessive vibration during launch threatens system design; 

* Uncertainty about how flight characteristics will be impacted by a 
fifth segment added to the Ares I launch vehicle; 

* Ares I upper stage essentially requires development of a new engine; 

* No industry capability currently exists for producing the kind of 
heat shields that the Orion will need for protecting the crew 
exploration vehicle when it reenters Earthï¿½s atmosphere; and 

* Existing test facilities are insufficient for testing Ares Iï¿½s new 
engine, for replicating the engineï¿½s vibration and acoustic 
environment, and for testing the thermal protection system for the 
Orion vehicle. 

All these unknowns, as well as others, leave NASA in the position of 
being unable to provide firm cost estimates for the projects at this 
point. Meanwhile, tight deadlines are putting additional pressure on 
both the Ares I and Orion projects. Future requirements changes raise 
risks that both projects could experience cost and schedule problems. 

GAOï¿½s past work on space systems acquisition and the practices of 
leading developers identifies best practices that can provide decision 
makers with insight into the progress of development at key junctures, 
facilitate Congressional oversight, and support informed decision 
making. This work has also identified common red flags throughout 
development, which decision makers need to keep in mind when assessing 
the projects. They include: 

Key indicators: Weight growth is often among the highest drivers of 
cost growth. Unanticipated software complexity, often indicated by 
increases in the number of lines of code, can portend cost and schedule 
growth. 

Key junctures: The preliminary design review, critical design review, 
and production review are key junctures that involve numerous steps and 
help focus the agency on realistic accomplishments within reachable 
goals. A disciplined approach aligned with key indicators can provide 
the knowledge needed to make informed investment decisions at each 
review. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-186T]. For more 
information, contact Cristina T. Chaplain at (202) 512-4841 or 
[email protected]. 

[End of section] 

Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today to discuss challenges that the National 
Aeronautics and Space Administration (NASA) faces in developing the 
systems to achieve its goals for the President's Vision for Space 
Exploration.[Footnote 1] We have been focusing our work primarily on 
the Ares I Crew Launch Vehicle and the Orion Crew Exploration 
Vehicle[Footnote 2], as they are among the first major efforts 
conducted as part of NASA's Constellation Program to support 
implementation of the Vision and represent a substantial investment for 
NASA. Over $7 billion in contracts has already been awarded--and nearly 
$230 billion is estimated to be ultimately spent over the next two 
decades. Moreover, NASA is under pressure to develop the vehicles 
quickly, as the Space Shuttle's retirement in 2010 means that there 
could be at least a 5-year gap in our nation's ability to send humans 
to space. 

In summary, NASA is currently working toward preliminary design reviews 
for the vehicles--a milestone that successful development organizations 
use to make hard decisions about whether a program should proceed with 
development. While this is a phase for discovery and risk reduction, 
there are considerable unknowns as to whether NASA's plans for the Ares 
I and Orion vehicles can be executed within schedule goals, as well as 
what these efforts will ultimately cost. In fact, we do not know yet 
whether the architecture and design solutions selected by NASA will 
work as intended. This is primarily because NASA is still in the 
process of defining both of the projects' performance requirements and 
some of these uncertainties could affect the mass, loads, and weight 
requirements for the vehicles. It is also working through significant 
technical risks, such as oscillation within the first stage of the Ares 
I vehicle, which computer modeling indicates could cause unacceptable 
structural vibrations. 

NASA is aiming to complete preliminary design reviews for the Ares I 
and Orion this year, scheduled for August 2008 Ares I and September 
2008 respectively, but it will be challenged in doing so given the 
level of knowledge that still needs to be attained. In addition, to 
minimize the gap in human spaceflight caused by the shuttle's 
retirement, there is a high degree of concurrency within the projects. 
Our prior work has shown that concurrent development, especially when 
new technologies are involved, increases the risk that significant 
problems will be discovered as the systems' designs are integrated that 
could result in cost and schedule delays. NASA's schedule leaves little 
room for the unexpected. If something goes wrong with the development 
of the Ares I or the Orion, the entire Constellation Program could be 
thrown off course and the return to human spaceflight delayed. 

NASA recognizes the risks involved with its approach and has taken 
steps to mitigate some of these risks. It is important that, in 
mitigating risks, NASA continually assess the viability of its plans 
for the Ares I and Orion. The current state of play requires that NASA 
remain open to the possibility that it may need to revisit decisions on 
its architecture and design as these vehicles are expected to be in use 
for decades to come and decisions made now will have long-term 
consequences. 

Moreover, with additional significant investment decisions still ahead, 
it is important that agency decision makers and Congress maintain clear 
insight into the progress the projects are making as well as any 
potential problems. This type of oversight is important, not just for 
the Ares I and Orion vehicles, but for the entire future exploration 
effort--since resources available to fund the Vision are constrained, 
as competition for resources increases within the federal government 
over the next several decades. In this regard, our work has identified 
specific markers that can be used to (1) assess NASA's progress in 
closing critical knowledge gaps and (2) identify issues that could 
result in cost growth, schedule delays, or decreased performance. In 
other words, they can be used to assess whether there is a viable 
business case for pressing forward with the projects. 

We have issued a number of reports and testimonies that touch on 
various aspects of NASA's Constellation Program and in particular the 
development efforts underway for the Orion and Ares I projects. These 
reports and testimonies have questioned the affordability and overall 
acquisition strategy for each project. In July 2006 we recommended that 
NASA modify the Orion Crew Vehicle acquisition strategy to ensure the 
agency did not commit itself to a long-term contractual obligation 
prior to establishing a sound business case. Although initially NASA 
disagreed with our recommendation, the agency subsequently revised its 
acquisition strategy to address some of the concerns we raised. In 
October 2007 we recommended that NASA develop a sound business case 
supported by firm requirements, mature technologies, a preliminary 
design, a realistic cost estimate, and sufficient funding and time-- 
before proceeding beyond preliminary design review. NASA concurred with 
this recommendation and subsequently slipped the Ares I preliminary 
design review from July 2008 to August 2008. 

My statement today is based on these products, as well as updated 
information based on our continual monitoring of the projects at the 
request of members of Congress. To conduct these reviews, we analyzed 
relevant project documentation, prior GAO reports, NASA documents, and 
contractor information; interviewed program and project officials; and 
reviewed NASA's risk management system for the Constellation Program. 
Based on this work, my statement will specifically address the 
challenges that NASA faces developing the Ares I and Orion vehicles 
with regard to requirements definition, technology and hardware gaps, 
cost and schedule estimates, and facilities needs. Further, I will 
provide key indicators that decision makers could use to assess risks 
as the two development efforts move forward. We conducted this 
performance audit from October 2007 through April 2008 in accordance 
with generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 

Background: 

In September 2005, NASA outlined an initial framework for implementing 
the President's Vision for Space Exploration in its Exploration Systems 
Architecture Study. NASA is now implementing the recommendations from 
this study within the Constellation Program, which includes three major 
development projects--the Ares I Crew Launch Vehicle, the Orion Crew 
Exploration Vehicle, and the Ares V Cargo Launch Vehicle as shown in 
figure 1. 

Figure 1: Overview of Ares I and Orion Projects: 

This figure is a combination of two photographs as part of an overview 
of Ares I and Orion Projects. The Ares V Cargo Launch Vehicle and Ares 
I Crew Launch Vehicle are pictured. 

[See PDF for image] 

Source: GAO analysis and presentation of NASA photos and data. 

[End of figure] 

To reduce cost and minimize risk in developing these projects, NASA 
planned to maximize the use of heritage systems and technology. Since 
2005, however, NASA has made changes to the basic architecture for the 
Ares I and Orion designs[Footnote 3] that have resulted in the 
diminished use of heritage systems. This is due to the ability to 
achieve greater cost savings with alternate technology and the 
inability to recreate heritage technology. For example, the initial 
design was predicated on using the main engines and the solid rocket 
boosters from the Space Shuttle Program. However, NASA is no longer 
using the Space Shuttle Main Engines because greater long-term cost 
savings are anticipated through the use of the J-2X engine. In another 
example, NASA increased the number of segments on the Ares I first- 
stage reusable solid rocket booster from four to five to increase 
commonality between the Ares I and Ares V, and eliminate the need to 
develop, modify, and certify both a four-segment reusable solid rocket 
booster and an expendable Space Shuttle main engine for the Ares I. 
Finally, according to the Orion program executive the Orion project 
originally intended to use the heat shield from the Apollo program as a 
fallback technology for the Orion thermal protection system, but was 
unable to recreate the Apollo material. 

NASA has authorized the Ares I and Orion projects to proceed with 
awarding development contracts. In April 2006, NASA awarded a $1.8 
billion contract for design, development, test, and evaluation of the 
Ares I first stage to Alliant Techsystems. NASA also awarded a $1.2 
billion contract for design, development, test, and evaluation of the 
Ares I upper stage engine--the J-2X--to Pratt and Whitney Rocketdyne in 
June 2006. NASA is developing the upper stage and the upper stage 
instrument unit, which contains the control systems and avionics for 
the Ares I, in-house. However, NASA awarded a $514.7 million contract 
for design support and production of the Ares I upper stage to the 
Boeing Company in August 2007. In August 2006, NASA awarded Lockheed 
Martin a $3.9 billion contract to design, test, and build the Orion 
crew exploration vehicle.[Footnote 4] According to NASA, the contract 
was modified in April 2007, namely by adding 2 years to the design 
phase and two test flights of Orion's launch abort system and by 
deleting the production of an cargo variant for the International Space 
Station. NASA indicates that these changes increased the contract value 
to $4.3 billion. Federal procurement data shows that an additional 
modification has been signed which increased the value of the contract 
by an additional $59 million. 

NASA has completed or is in the process of completing key reviews on 
both the Ares I and Orion projects. NASA has completed the system 
requirements review for each project and is in the midst of finalizing 
the system definition reviews.[Footnote 5] At the systems requirements 
review, NASA establishes a requirements baseline that serves as the 
basis for ongoing design analysis work and systems testing. Systems 
definition reviews focus on emerging designs for all transportation 
elements and compare the predicted performance of each element against 
the currently baselined requirements. Figure 2 shows the timeline for 
Ares I and Orion critical reviews. 

Figure 2: Timeline for Ares I and Orion Critical Reviews (in fiscal 
years): 

This figure is a timeline for Ares I and Orion critical reviews in 
fiscal years. 

Areas I crew launch vehicle; 
System requirements review: December 2006; 
Preliminary design review: August 2008; 
Critical design review: March 2010. 

Orion crew exploration vehicle; 
System requirements review: March 2007; 
Preliminary design review: September 2008; 
Critical design review: October 2009. 

[See PDF for image] 

Source: GAO analysis of NASA data. 

[End of figure] 

NASA is using its Web-based Integrated Risk Management Application to 
help monitor and mitigate the risks with the Ares I and Orion 
development efforts and for the overall Constellation Program. The risk 
management application identifies and documents risks, categorizes 
risks--as high, medium, and low based on both the likelihood of an 
undesirable event as well as the consequences of that event to the 
project--and tracks performance against mitigation plans. For the Ares 
I project, the application is tracking 101 risks, 36 of which are 
considered high-risk areas.[Footnote 6] For the Orion project, NASA is 
tracking 193 risks, including 71 high-risk areas.[Footnote 7] NASA is 
developing and implementing plans to mitigate some of these risks. 

Requirements Setting is a Primary Challenge for Both the Ares I and 
Orion Projects: 

Although project level requirements were baselined at both systems 
requirements reviews, continued uncertainty about the systems' 
requirements have led to considerable unknowns as to whether NASA's 
plans for the Ares I and Orion vehicles can be executed within schedule 
goals, as well as what these efforts will ultimately cost. Such 
uncertainty has created knowledge gaps that are affecting many aspects 
of both projects. Because the Orion vehicle is the payload that the 
Ares I must deliver to orbit, changes in the Orion design, especially 
those that affect weight, directly affect Ares I lift requirements. 
Likewise, the lift capacity of the Ares I drives the Orion design. Both 
the Orion and Ares I vehicles have a history of weight and mass growth, 
and NASA is still defining the mass, loads, and weight requirements for 
both vehicles. According to agency officials, continuing weight growth 
led NASA to rebaseline the Orion vehicle design in fall 2007. This 
process involved "scrubbing" the Orion Vehicle to establish a zero- 
based design capable of meeting minimal mission requirements but not 
safe for human flight. Beginning with the zero-based design NASA first 
added back the systems necessary to ensure crew safety and then 
conducted a series of engineering trade-offs to determine what other 
systems should be included to maximize the probability of mission 
success while minimizing the system's weight. As a result of these 
trade-offs, NASA modified the requirement for nominal landing on land 
to nominal landing in water, thereby gaining 1500 lbs of trade space in 
the Orion design. 

NASA recognizes that continued weight growth and requirements 
instability are key risks facing the Orion project and that continued 
instability in the Orion design is a risk facing the Ares I project. 
The Ares I and Orion projects are working on these issues but have not 
yet finalized requirements or design. Our previous work on systems 
acquisition work shows that the preliminary design phase is an 
appropriate place to conduct systems engineering to support requirement 
and resource trade-off decisions. For the Ares I project, this phase is 
scheduled to be completed in August 2008, whereas for the Orion 
project, it is September 2008--leaving NASA only 4 and 5 months 
respectively to close gaps in requirements knowledge. NASA will be 
challenged to close such gaps, given that it is still defining 
requirements at a relatively high level and much work remains to be 
done at the lower levels. Moreover, given the complexity of the Orion 
and Ares I efforts and their interdependencies, as long as requirements 
are in flux, it will be extremely difficult to establish firm cost 
estimates and schedule baselines. 

Technology and Hardware Gaps Along with Requirements Uncertainty are 
Increasing Risk: 

Currently, nearly every major segment of Ares I and Orion faces 
knowledge gaps in the development of required hardware and technology 
and many are being affected by uncertainty in requirements. For 
example, computer modeling is showing that thrust oscillation within 
the first stage of the Ares I could cause excessive vibration 
throughout the Ares I and Orion. Resolving this issue could require 
redesigns to both the Ares I and Orion vehicles that could ultimately 
impact cost, schedule, and performance. Furthermore, the addition of a 
fifth segment to the Ares I first stage has the potential to impact 
qualification efforts for the first stage and could result in costly 
requalification and redesign efforts. Additionally, the J-2X engine 
represents a new engine development effort that, both NASA and Pratt 
and Whitney Rocketdyne recognize, is likely to experience failures 
during development. Addressing these failures is likely to lead to 
design changes that could impact the project's cost and schedule. With 
regard to the Orion project, there is currently no industry capability 
for producing a thermal protection system of the size required by the 
Orion. NASA has yet to develop a solution for this gap, and given the 
size of the vehicle and the tight development schedule, a feasible 
thermal protection system may not be available for initial operational 
capability to the space station. The table 1 describes these and other 
examples of knowledge gaps in the development of the Ares I and Orion 
vehicles. 

Table 1: Examples of Ares I and Orion Technology and Hardware Gaps: 

Ares I Crew Launch Vehicle; 
First stage; 
Current modeling indicates that thrust oscillation within the first 
stage causes unacceptable structural vibrations. There is a possibility 
that the thrust oscillation frequency and magnitude may be outside the 
design limits of the Ares design requirements. A NASA focus team 
studied this issue and has proposed options for mitigation including 
incorporating vibration absorbers into the design of the first stage 
and redesigning portions of the Orion Vehicle to isolate the crew from 
the vibration. Further, it is unknown how the addition of a fifth 
segment to the launch vehicle will affect flight characteristics. 
Failure to completely understand the flight characteristics of the 
modified booster could create a risk of hardware failure and loss of 
vehicle control. Ares I relies on hardware adapted from the Space 
Shuttle program that may not meet qualification requirements. 
Qualification requirements may be difficult to meet due to the new 
ascent loads (the physical strain on the spacecraft during launch) and 
vibration and acoustic environments associated with the Ares I. 
Resulting redesign and requalification efforts could affect cost and 
schedule. NASA is currently working to further define the vibration and 
acoustic environment. 

Ares I Crew Launch Vehicle; 
Upper stage; 
NASA redesigned the upper stage configuration from two completely 
separate propellant tanks to two tanks with one common bulkhead. The 
prior configuration employed a simpler design with a lower 
manufacturing cost but did not meet mass requirements. The current 
common bulkhead design involves a complex and problematic manufacturing 
process that challenged earlier development efforts on the Apollo 
program. In fact, NASA's Web-based Integrated Risk Management 
Application indicated that one of the lessons learned from the Apollo 
program was not to use common bulkheads because they are complex and 
difficult to manufacture. 

Ares I Crew Launch Vehicle; 
J-2X upper stage engine; 
Although the J-2X is based on the J-2 and J-2S engines used on the 
Saturn V and leverages knowledge from the X-33 and RS-68, the number of 
planned changes is such that, according to NASA review boards, the 
effort essentially represents a new engine development. NASA and Pratt 
and Whitney Rocketdyne recognize that some level of developmental 
problems are inherent in all new engine development programs. As such, 
the project has predicted that the J-2X development will require 29 
rework cycles. In addition, the J-2X faces extensive redesign to 
incorporate modern controls, achieve increased performance 
requirements, and meet human rating standards. The J-2X developers also 
face significant schedule risks in developing and manufacturing a 
carbon composite nozzle extension needed to satisfy thrust 
requirements. According to contractor officials, the extension is more 
than 2 feet--i.e., about one-third--wider in diameter than existing 
nozzle extensions. 

Orion Crew Exploration Vehicle; 
Launch abort system; 
Technology development of the launch abort system is being conducted 
concurrently with design of Orion. Ongoing requirements changes related 
to the Orion system and its subsystems or development setbacks could 
(1) prevent some test objectives from being adequately demonstrated 
during early launch abort system tests, (2) drive the need for 
additional testing of the abort system, or (3) lead to design revisions 
or changes to the required number of spares. Any of these possibilities 
could lead to increased program costs and delays to the flight test 
schedule. According to NASA officials, the agency is currently 
assessing alternative designs for the launch abort system to address 
weight and vibration concerns. 

Orion Crew Exploration Vehicle; 
Thermal protection system; 
The Orion requires the development of a large-scale ablative thermal 
protection system. Given the size of the vehicle and the tight 
development schedule, a feasible thermal protection system may not be 
available in time for the Orion initial operational capability to the 
space station. There is currently no industry capability for producing 
a thermal protection system of the size required by the Orion. 
Furthermore, heat shield design features required by the Orion, namely 
the size, have never been proven and must be developed. NASA is 
currently conducting an advanced development project to mature 
technologies necessary to meet thermal protection system requirements. 

Source: GAO analysis of NASA data. 

[End of table] 

Constellation Cost Estimates Are Preliminary Due to Requirements 
Uncertainty: 

NASA's preliminary cost estimates for the Constellation Program are 
likely to change when requirements are better defined. NASA will 
establish a preliminary estimate of life cycle costs for the Ares I and 
Orion in support of each project's system definition review. A formal 
baseline of cost, however, is not expected until the projects' 
preliminary design reviews are completed. NASA is working under a self- 
imposed deadline to deliver the new launch vehicles no later than 2015 
in order to minimize the gap in human spaceflight between the Space 
Shuttle's retirement in 2010 and the availability of new transportation 
vehicles. The Constellation Program's budget request maintains a 
confidence level of 65 percent (i.e., NASA is 65 percent certain that 
the actual cost of the program will either meet or be less than the 
estimate) for program estimates based upon a 2015 initial operational 
capability. Internally, however, the Ares I and Orion projects are 
working toward an earlier initial operational capability (2013), but at 
a reduced budget confidence level--33 percent. However, NASA cannot 
reliably estimate the money needed to complete technology development, 
design, and production for the Ares I and Orion projects until 
requirements are fully understood. 

NASA has identified the potential for a life cycle cost increase as a 
risk for the Orion program. According to NASA's risk database, given 
the historical cost overruns of past NASA systems and the known level 
of uncertainty in the current Orion requirements, there is a 
possibility that Orion's life cycle cost estimate may increase over 
time. NASA acknowledges that such increases are often caused by the 
unknown impacts of decisions made during development. One factor 
currently contributing to cost increases is the addition of new 
requirements. NASA is working to formulate the best life cycle cost 
estimate possible during development, is identifying and monitoring 
costs threats, and is implementing management tools all aimed at 
addressing this risk. 

Schedule Pressures Add Additional Risks for Ares I and Orion: 

There are considerable schedule pressures facing both the Ares I and 
Orion projects. These are largely rooted in NASA's desire to minimize 
the gap between the retirement of the space shuttle and availability of 
the new vehicles. Because of this scheduling goal, NASA is planning to 
conduct many interdependent development activities concurrently-- 
meaning if one activity should slip in schedule, it could have 
cascading effects on other activities. Moreover, some aspects of the 
program are already experiencing scheduling delays due to the fact that 
high-level requirements are still being defined. 

Ares I: 

The development schedule for the J-2X is aggressive, allowing less than 
7 years from development start to first flight, and highly concurrent. 
Due to the tight schedule and long-lead nature of engine development, 
the J-2X project was required to start out earlier in its development 
than the other elements on the Ares I vehicle. This approach has 
introduced a high degree of concurrency between the setting of overall 
Ares I requirements and the development of the J-2X design and 
hardware. Consequently, the engine development is out of sync with the 
first stage and upper stage in the flow-down and decomposition of 
requirements, an approach our past work has shown to be fraught with 
risk. NASA acknowledges that the engine development is proceeding with 
an accepted risk that future requirements changes may affect the engine 
design and that the engine may not complete development as scheduled in 
December 2012. The J-2X development effort represents a critical path 
for the Ares I project. Subsequently, delays in the J-2X schedule for 
design, development, test, and evaluation would have a ripple effect of 
cost and schedule impacts throughout the entire Ares I project. 

The schedule for the first stage also presents a potential issue for 
the entire Ares I project. Specifically, the critical design review for 
the first stage is out of sync with the Ares I project-level critical 
design review. NASA has scheduled two critical design reviews for the 
first stage. The first critical design review is scheduled for November 
2009, 5 months before the Ares I project critical design review. At 
this point, however, the project will not have fully tested the first 
stage development motors. The second critical design review, in 
December 2010, occurs after additional testing of developmental motors 
is conducted. By conducting the Ares I critical design review before 
the first stage project critical design review, the project could 
prematurely begin full-scale test and integration activities a full 9 
months before the first stage design has demonstrated maturity. If 
problems are found in the first stage design during the later testing, 
implementing solutions could result in costly rework and redesign and 
delay the overall project schedule. 

Orion: 

Cost and schedule reporting on the Orion project indicates that the 
Orion project's efforts to mature requirements and design and to 
resolve weight issues is placing pressure on the Orion schedule. 
Specifically, activities aimed at assessing alternate designs to reduce 
overall vehicle mass, rework to tooling concepts, and late requirements 
definition have contributed to the project falling behind schedule. 
Further, the Orion risk system indicates that schedule delays 
associated with testing may occur. The current Orion design has high 
predicted vibration and acoustic levels. Historically, components 
designed and qualified for uncertain vibration and acoustic 
environments have resulted in some failures and required subsequent 
redesign and retest. Failures during qualification testing of Orion 
components may lead to schedule delays associated with redesigning 
components. 

NASA's Administrator has publicly stated that if Congress provided the 
Agency an additional $2 billion that NASA could accelerate the 
Constellation program's initial operational capability date to 2013. We 
believe that this assessment is highly optimistic. The development 
schedule for the J-2X engine, the critical path for the Ares I 
development, is already recognized as aggressive, allowing less than 7 
years for development. The development of the Space Shuttle Main engine 
by comparison took 9 years. Further, NASA anticipates that the J-2X 
engine is likely to require 29 rework cycles to correct problems 
identified during testing. Given the linear nature of a traditional 
test-analyze-fix-test cycle, even large funding increases offer no 
guarantee of program acceleration, particularly when the current 
schedule is already compressed and existing NASA test facilities are 
already maximized. 

Test Facilities for Ares I and Orion Insufficient: 

According to NASA, at this time, existing test facilities are 
insufficient to adequately test the Ares I and Orion systems. Existing 
altitude test facilities are insufficient to test the J-2X engine in a 
relevant environment. To address this issue, NASA is in the process of 
constructing a new altitude test facility at Stennis Space Center for 
the J-2X. Also, current facilities are inadequate to replicate the 
Orion vibration and acoustic environment. Further, Pratt and Whitney 
Rocketdyne--the J-2 X upper stage engine contractor--indicated that 
existing test stands that could support J-2X testing will be tied up 
supporting the Space Shuttle program until 2010. NASA has taken steps 
to mitigate J-2X risks by increasing the amount of component-level 
testing, procuring additional development hardware and test facilities, 
and working to make a third test stand available to the contractor 
earlier than originally planned. NASA has compensated for this schedule 
pressure on the Ares I project by adding funds for testing and other 
critical activities. But it is not certain that added resources will 
enable NASA to deliver the Ares I when expected. 

With respect to Orion's thermal protection system, facilities available 
from the Apollo era for testing large-scale heat shields no longer 
exist. Therefore, NASA must rely on two facilities that fall short in 
providing the necessary capability and scheduling to test ablative 
materials needed for Orion. Additionally, NASA has no scheduled test to 
demonstrate the thermal protection system needed for lunar missions. 
NASA is exploring other options, including adding a lunar return flight 
test and building a new improved test facility. Due to the scheduled 
first lunar flight, any issues identified during such testing would 
need to be addressed in the time between the flight test and the first 
flight. 

Oversight Based on Best Practices and Key Indicators Important for 
Program Success: 

NASA is poised to invest a significant amount of resources to implement 
the Vision over the long term and specifically to develop the Ares I 
and Orion projects over the next several years. Accordingly, you asked 
us to articulate indicators that Congress could use to assess progress. 
Our prior work has shown that investment decisions of this magnitude 
need to be based on an established and executable business case and 
that there are several key indicators that Congress could be informed 
of to assess progress throughout development. These include areas 
commonly underestimated in space programs, such as weight growth and 
software complexity, as well as indicators used by best practice 
organizations to assess readiness to move forward in the development 
cycle. Space programs which we have studied in detail in the past have 
tended to underestimate cost in some of these areas. 

Weight Growth: 

Our previous work on government-funded space systems has shown that 
weight growth is often not anticipated even though it is among the 
highest drivers of cost growth for space systems. Weight growth can 
affect the hardware needed to support a system, and, in the case of 
launch vehicles, the power or thrust required for the system. As the 
weight of a particular system increases, the power or thrust required 
for that system will also increase. This could result in the need to 
develop additional power or thrust capability to lift the system, 
leading to additional costs, or to stripping down the vehicle to 
accommodate current power or thrust capability. For example, NASA went 
through the process to zero-base the design for the Orion to address 
weight concerns. Continual monitoring of system weight and required 
power/thrust, as well as margins or reserves for additional growth, can 
provide decision makers with an indicator of whether cost increases can 
be anticipated. 

Software Complexity: 

The complexity of software development on a system, often denoted by 
the number of lines of code on a system, can also be used as an 
indicator to monitor whether a program will meet cost and schedule 
goals. In our work on software development best practices, we have 
reported that the Department of Defense has attributed significant cost 
and schedule overruns on software-intensive systems to developing and 
delivering software. Generally, the greater the number of lines of 
code, the more complicated the system development. Changes to the 
amount of code needed to be produced can indicate potential cost and 
schedule problems. Decision makers can monitor this indicator by 
continually asking for information on the estimated amount of code 
needed on a system and inquiring about any increases in need and their 
impact on cost and schedule. 

There are other areas, such as the use of heritage systems and 
industrial base capability that are commonly underestimated in space 
programs as well. However, weight increases and software growth are 
more quantifiable and thus useful for oversight purposes. 

Indicators that Can be Used to Assess Knowledge Gap at Key Junctures: 

Additionally, since the mid-1990s, GAO has studied the best practices 
of leading commercial companies. On the basis of this information, and 
taking into account the differences between commercial product 
development and major federal acquisitions, we have outlined a best 
practices product development model--known as a knowledge-based 
approach to system development. This type of approach calls for 
investment decisions to be made on the basis of specific, measurable 
levels of knowledge at critical junctures before investing more money 
and proceeding with development. 

Importantly, our work has shown the most leveraged decision point is 
matching the customer's needs with the developer's resources (time, 
dollars, technology, people, etc.) because it sets the stage for the 
eventual outcome--desirable or problematic. The match is ultimately 
achieved in every development program, but in successful development 
programs, it occurs before product development is formally initiated 
(usually the preliminary design review). If the knowledge attained at 
this and other critical junctures does not confirm the business case on 
which the acquisition was originally justified, the best practice 
organizations we have studied do not allow the program to go forward. 

We have highlighted the three critical junctures at which developers 
must have knowledge to make large investment decisions--the preliminary 
design review, the critical design review, and the production review-- 
and the numerous key indicators that can be used to increase the 
chances of successful outcomes. 

In assessing the Orion and Ares programs, the Congress and NASA 
decision-makers can use these indicators in order to reliably gauge 
whether there is a sufficient business case for allowing the programs 
to proceed forward. 

Preliminary design review: Before product development is started, a 
match must be made between the customers' needs and the available 
resources--technical and engineering knowledge, time, and funding. To 
provide oversight at this juncture, NASA could provide Congress with 
information to verify that the following have indicators been met: 

* All critical technologies are demonstrated to a high level of 
technology maturity, that is demonstrated that they can perform in a 
realistic or, more preferably, operational environment. A technology 
readiness level 6 or 7 would indicate that this has been achieved. One 
approach to ensure that technology readiness is reliably assessed is to 
use independent testing; 

* Project requirements are defined and informed by the systems 
engineering process; 

* Cost and schedule estimates established for the project are based on 
knowledge from the preliminary design using systems engineering tools; 

* Additional resources are in place, including needed workforce, and a 
decision review is conducted following completion of the preliminary 
design review. 

A critical enabler for success in this phase of development is 
performance and requirements flexibility. Customers and product 
developers both need to be open to reducing expectations, deferring 
them to future programs, or to investing more resources up front to 
eliminate gaps between resources and expectations. In successful 
programs we have studied, requirements were flexible until a decision 
was made to commit to product development because both customers and 
developers wanted to limit cycle time. This makes it acceptable to 
reduce, eliminate, or defer some customer wants so that the product's 
requirements could be matched with the resources available to deliver 
the product within the desired cycle time. 

Critical design review: A product's design must demonstrate its ability 
to meet performance requirements and be stable about midway through 
development. To provide oversight at this juncture, NASA could provide 
Congress with information to verify that the following indicators have 
been met: 

* At least 90 percent of engineering drawings are complete; 

* All subsystem and system design reviews have been completed; 

* The design meets requirements demonstrated through modeling, 
simulation, or prototypes; 

* Stakeholders' concurrence that drawings are complete and producible 
is obtained; 

* Failure modes and effects analysis have been completed; 

* Key system characteristics are identified; 

* Critical manufacturing processes are identified; 

* Reliability targets are established and a growth plan based on 
demonstrated reliability rates of components and subsystems is 
developed; and: 

* A decision review is conducted following the completion of the 
critical design review. 

Production Review: The developer must show that the product can be 
manufactured within cost, schedule, and quality targets and is 
demonstrated to be reliable before production begins. To provide 
oversight at this juncture, NASA could provide Congress with 
information to verify that the following indicators have been met: 

* Manufacturing processes have been demonstrated; 

* Production representative prototypes have been built; 

* Production representative prototypes have been tested and have 
achieved reliability goals; 

* Production representative prototypes have been demonstrated in an 
operational environment through testing; 

* Statistical process control data have been collected; 

* Critical processes have been demonstrated to be capable and that they 
are in statistical control; 

* A decision review is conducted following completion of the production 
readiness review. 

Over the past 2 years, we have recommended that NASA incorporate a 
knowledge-based approach in its policies and take steps to implement 
this type of approach in its programs and projects.[Footnote 8] NASA 
has incorporated some knowledge-based concepts into its acquisition 
policies. For example, NASA now requires a decision review between each 
major phase of the acquisition life cycle and has established general 
entrance and success criteria for the decision reviews. In addition, we 
have reported that this type of approach is being embraced by the Ares 
I project. 

Concluding Observations: 

In conclusion, the President's Vision for Space Exploration is an 
ambitious effort, not just because there will be technical and design 
challenges to building systems needed to achieve the Vision's goals, 
but because there are limited resources within which this can be 
accomplished. Moreover, the long-term nature of the Vision means that 
commitments for funding and to the goals of the Vision will need to be 
sustained across presidential administrations and changes in 
congressional leadership. For these reasons, it is exceedingly 
important that the right decisions are made early on and that decision- 
makers have the right knowledge going forward so that they can make 
informed investment decisions. 

In looking at the first major investments, the Ares I and Orion 
projects, it is important to recognize that they are risky endeavors, 
largely due to their complexity, scope, and interdependencies. It is 
also important to recognize that the desire to minimize the gap in 
human space flight adds considerable risk, since it could limit NASA's 
ability to study emerging problems and pursue alternative ways of 
addressing them. For these reasons, as well as the magnitude of 
investment at stake, it is imperative that NASA be realistic and open 
about the progress it is making and to be willing to make changes to 
the architecture and design if technical problems can not be solved 
without overly compromising performance. Additionally, Congress needs 
to be well-informed about the extent to which knowledge gaps remain and 
what tradeoffs or additional resources are needed to close those gaps 
and to support changes if they are determined to be necessary. The 
upcoming preliminary design review milestones represent perhaps the 
most critical juncture where these assessments can take place and where 
hard decisions can be made as to whether the programs should proceed 
forward. It may well be the last opportunity to make significant 
adjustments before billions of dollars are spent and long term 
commitments become solidified. 

Mr. Chairman, this concludes my prepared statement. I would be pleased 
to answer any questions that you may have at this time. 

GAO Contacts and Staff Acknowledgements: 

For further questions about this statement, please contact Cristina T. 
Chaplain at (202) 512-4841. Individuals making key contributions to 
this statement include James L. Morrison, Meredith A. Kimmitt, Lily 
Chin, Neil Feldman, Rachel Girshick, Shelby S. Oakley, and John S. 
Warren, Jr. 

[End of section] 

Susan Becker, Acting Manager, [email protected], (202) 512-4800 U.S. 
Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548: 

Footnotes: 

[1] The Vision includes a return to the moon that is intended 
ultimately to enable future exploration of Mars and other destinations. 
To accomplish this, NASA initially plans to (1) complete its work on 
the international Space Station by 2010, fulfilling its commitment to 
15 international partner countries; (2) begin developing a new manned 
exploration vehicle to replace the space shuttle; and (3) return to the 
moon in preparation for future, more ambitious missions. 

[2] GAO, NASA: Agency Has Taken Steps Toward Making Sound Investment 
Decisions for Ares I but Still Faces Challenging Knowledge Gaps, GAO- 
08-51 (Washington, D.C.: Oct. 31, 2007) and GAO, NASA: Long-Term 
Commitment to and Investment in Space Exploration Program Requires More 
Knowledge, GAO-06-817R (Washington, D.C.: July 17, 2006). 

[3] Heritage systems are systems with characteristics similar to the 
one being developed. A heritage system is often the one the new program 
is replacing. 

[4] The actual value of the contract could be greater than $3.9 billion 
if NASA exercises options on the contract for production and 
sustainment or issues orders against the indefinite delivery/indefinite 
quantity portion of the contract. 

[5] The system requirements review is intended to examine the function 
and performance requirements defined for the system and the preliminary 
project plan and ensure that the requirements and the selected concept 
will satisfy the mission. The system definition review examines the 
proposed system design and the flow-down of that design to all 
functional elements of the system. The system requirements review and 
system definition review process culminates with key decision point B 
wherein NASA determines the project's readiness to move forward. 

[6] This is the total number of open risks for the Ares I project as of 
March 25, 2008. It does not include risks that have been closed or 
risks that NASA considers sensitive. 

[7] This is the total number of open risks for the Orion project as of 
March 25, 2008. It does not include risks that have been closed or 
risks that NASA considers sensitive. 

[8] AO, NASA: Implementing a Knowledge-Based Acquisition Framework 
Could Lead to Better Investment Decisions and Project Outcomes, GAO-06-
218 (Washington, D.C.: Dec. 21, 2005); GAO, NASA: Long-Term Commitment 
to and Investment in Space Exploration Program Requires More Knowledge, 
GAO-06-817R (Washington, D.C.: July 17, 2006); and GAO, NASA's James 
Webb Space Telescope: Knowledge-Based Acquisition Approach Key to 
Addressing Program Challenges, GAO-06-634 (Washington, D.C.: July 14, 
2006).

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability.  

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates."  

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:  

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548:  

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061:  

To Report Fraud, Waste, and Abuse in Federal Programs:  

Contact:  

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470:  

Congressional Relations:  

Ralph Dawn, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548:  

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***