Tactical Aircraft: DOD's Cancellation of the Joint Strike Fighter
Alternate Engine Program Was Not Based on a Comprehensive
Analysis (22-MAY-06, GAO-06-717R).
The Department of Defense (DOD) expects to purchase about 2,400
Joint Strike Fighter (JSF) aircraft, with potential international
sales of 2,000 to 3,500 aircraft. When the number of aircraft
engines and spare parts expected to be purchased is
considered--along with the lifetime support needed to sustain the
engines--the future financial investment will be significant. DOD
implemented the JSF alternate engine development program in 1996
to provide competition between two engine manufacturers in an
effort to achieve cost savings, improve performance, and gain
other benefits. Since then, DOD has invested $1.2 billion in the
alternate engine program, and, in August 2005, it awarded a $2.4
billion contract for system development and demonstration of an
alternate engine. However, in its fiscal year 2007 budget
submission, DOD proposed canceling the alternate engine program.
Concerned whether this decision was based on sound analysis,
Congress asked us to review DOD's rationale for canceling the
program and the analysis supporting it, including the life cycle
savings, benefits, and risks assessed.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-717R
ACCNO: A54540
TITLE: Tactical Aircraft: DOD's Cancellation of the Joint Strike
Fighter Alternate Engine Program Was Not Based on a Comprehensive
Analysis
DATE: 05/22/2006
SUBJECT: Aircraft engines
Defense cost control
Defense procurement
Fighter aircraft
Life cycle costs
Decision making
Competition
Competitive procurement
Risk assessment
F-16 Aircraft
F-22 Aircraft
Joint Strike Fighter
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-717R
* Figure 1: Planned Propulsion Performance Flight Testing and
* PDF6-Ordering Information.pdf
* Order by Mail or Phone
United States Government Accountability Office Washington, DC 20548
May 22, 2006
The Honorable John Warner Chairman Committee on Armed Services United
States Senate
The Honorable Curt Weldon Chairman Subcommittee on Tactical Air and Land
Forces Committee on Armed Services House of Representatives
Subject: Tactical Aircraft: DOD's Cancellation of the Joint Strike Fighter
Alternate Engine Program Was Not Based on a Comprehensive Analysis
The Department of Defense (DOD) expects to purchase about 2,400 Joint
Strike Fighter (JSF) aircraft, with potential international sales of 2,000
to 3,500 aircraft. When the number of aircraft engines and spare parts
expected to be purchased is considered-along with the lifetime support
needed to sustain the engines-the future financial investment will be
significant. DOD implemented the JSF alternate engine development program
in 1996 to provide competition between two engine manufacturers in an
effort to achieve cost savings, improve performance, and gain other
benefits.
Since then, DOD has invested $1.2 billion in the alternate engine program,
and, in August 2005, it awarded a $2.4 billion contract for system
development and demonstration of an alternate engine. However, in its
fiscal year 2007 budget submission, DOD proposed canceling the alternate
engine program. Concerned whether this decision was based on sound
analysis, you asked us to review DOD's rationale for canceling the program
and the analysis supporting it, including the life cycle savings,
benefits, and risks assessed.
To determine DOD's rationale for and analysis supporting the decision to
cancel the alternate engine program, we obtained and discussed data from
the Office of the Under Secretary of Defense for Acquisition, Technology,
and Logistics; the Office of the Director of Program Analysis and
Evaluation; and Air Force and Navy acquisition offices. We also
interviewed officials from the Office of the Director, Operational Test
and Evaluation, and the F-22A engine office, the predecessor engine for
the JSF engine. We reviewed the data, analyses, assumptions, and results
of two prior program management advisory group studies1 and the
justification briefing provided to Congress by the department. The
advisory group studies and briefing were identified as the analytical
underpinnings of the decision. We also met with the Air Force executive
who co-led both advisory groups and was the Air Force's propulsion product
group manager. Finally, the JSF program office stated that it was not
involved in the termination proposal. In performing this review, we used
data and information from that office collected on other assignments. We
performed our review from March to April 2006 in accordance with generally
accepted government auditing standards.
SUMMARY
DOD's decision to cancel the JSF alternate engine program was driven by
the need to identify sources of funding in order to pay for other
priorities within the department. In making the decision, the department
did not conduct a new and comprehensive analysis, but instead relied on
selective elements of two prior studies done in 1998 and 2002. In
supporting the decision to cancel, officials focused only on the potential
up-front savings in engine procurement costs. They did not, however,
consider the full long-term savings that might accrue from competition for
providing support for maintenance and operations over the life cycle of
the engine. Both prior studies had recommended proceeding with the
alternate engine program, despite the lack of significant procurement cost
savings, because of a number of other benefits competition was likely to
provide. Also in supporting the decision to cancel, officials cited
favorable progress made by the primary JSF engine and its predecessor
F-22A engine as reducing operational risks from a single source. However,
the primary JSF engine has completed only a small portion of its ground
tests and has not yet been flown, while the F-22A engine has completed
about 10 percent of its hours needed for system maturity and is not
currently meeting some reliability goals. Further, experts from one early
study concluded that the commonality with the F-22A engine is of limited
benefit for reducing development risk of the JSF engine.
DOD'S RATIONALE FOR TERMINATING THE ALTERNATE ENGINE PROGRAM
The decision for canceling the JSF alternate engine program was driven by
budget needs outside the JSF program. The decision was a consequence of
budget-cutting exercises to meet Office of the Secretary of Defense (OSD)
spending targets and to begin implementing Quadrennial Defense Review
(QDR) decisions. The alternate engine program was deemed to have a lesser
priority than other major DOD activities and programs. DOD officials and
the justification briefing stated that the rationale for canceling the
program was no net cost savings from competing engine buys and minimum
operational risk from relying on a single source.
Advisory groups composed of DOD and foreign partner representatives from
the technical, operational, and financial communities were established
under the aegis of the Assistant Secretary of the Navy for Research,
Development, and Acquisition to review technical and programmatic issues
of the alternate engine program, determine its costs and benefits, and
make recommendation for either continuing or terminating the program.
Funding for the alternate engine was included in the Navy's and Air
Force's initial 2007 budget submissions, according to the services'
acquisition officials. However, because of budget constraints, OSD
directed the military services to identify alternative sources for
reducing the fiscal year 2007 future years defense budget. Both services
proposed the termination of the alternate engine program. DOD officials
estimated that canceling the program would result in savings of about $2
billion over the remaining 8 years of the alternate engine development
program, which could then be used to fund higher-priority programs. In
recent testimony, the Under Secretary of Defense for Acquisition,
Technology, and Logistics stated that the department ultimately concluded
that maintaining two engine suppliers for the JSF program was not the most
efficient use of its resources. Department officials also noted that the
primary engine development program was progressing well, making a second
engine program unnecessary. On the basis of its assessments of the
progress of the primary engine for the JSF, the F-22A engine (which served
as the basis for the primary JSF engine), and past fighter engine
experience, officials deemed operational risks associated with a single
engine supplier acceptable.
DOD'S DECISION TO CANCEL THE ENGINE COMPETITION WAS BASED ON INSUFFICIENT COST,
SAVINGS, AND PERFORMANCE DATA
DOD did not conduct an up-to-date, comprehensive analysis of the total
life cycle costs, savings, and benefits to support its decision to
terminate the JSF's alternate engine development program. Instead, DOD
officials used two prior studies and considered the savings from engine
procurement only, excluding potential life cycle cost savings associated
with supporting, operating, and maintaining the fleet. These officials
also stated that the operational risk from relying on a single supplier is
reduced by favorable test and operational experiences with the primary JSF
and F22A engines. However, this assessment is based on:
o limited ground testing of the primary JSF engine and no actual flight
test results;
o experience with the F-22A engine, which has only completed about 10
percent of the operational flight hours needed to achieve system
maturity and which has still not achieved its reliability goal; and
o comparisons with the F-22A engine, which will likely have different
operational uses than the JSF engine.
DOD's Decision Is Not Supported by a Current and Comprehensive Analysis of Costs
and Benefits
DOD officials stated that the decision to cancel the JSF alternate engine
program is based largely on studies conducted in 1998 and 2002 by program
management advisory groups. These groups recommended that DOD proceed with
the alternate engine program, noting that the recommendation was made
independent of the services' abilities to fund the program. The advisory
groups determined that developing an alternate JSF engine had significant
benefits in the areas of contractor responsiveness, industrial base,
readiness, and international relations. They also reported finding
marginal benefits in the areas of cost savings and additional engine
growth capabilities (ability to add future engine improvements), and no
benefit to reducing development risk without restructuring the program.
Table 1 provides a summary of the program management advisory group study
results.
Table 1: 1998 and 2002 Program Management Advisory Group Study Findings on
the Benefits of an Alternate Engine Program
Beneficial Marginal No value
Factor assessed 1998 2002 1998 2002 1998 2002
Costs ' '
Development risk reduction ' '
Engine growth potential ' '
Fleet readiness ' '
Industrial base ' '
International implications ' '
Other considerations a ' '
Overall ' '
Source: DOD (data); GAO (analysis and presentation). a
Other considerations include contractor responsiveness, improved design
solutions, and competition at the engine subsystem level.
DOD's current conclusion that net savings from an alternate engine program
would be negligible at best was largely based on a break-even analysis in
the 2002 study that calculated how many engines would need to be purchased
at prices reflecting savings from competition to recover the costs
incurred to develop the second source. The analysis used the total
projected costs of the alternate engine development program, an estimated
$2.8 billion at that time.2 However, DOD has now invested about onethird
of that total since the 2002 study. Excluding these sunk costs and basing
the break-even analysis on development costs to go from this point forward
would reduce the engine costs that would need to be recovered through cost
savings associated with ongoing competition between engine suppliers. For
example, on the basis of the estimated development costs to go and using
the same assumptions and data as the 2002 study, we estimate that
achieving 20 percent savings from competition would allow a break-even
point to occur at about 1,700 engines-not 2,500, as projected in the 2002
study. An earlier break-even point in the purchase of engines would
increase the potential for savings over the life of the program.
Officials indicated that DOD's decision did not consider all the costs and
savings over the projected 30-year life cycle of the weapon system. A life
cycle cost-benefit analysis would consider all the potential costs and
savings associated with the competition, including operations and support,
over the expected life of the system. DOD did not analyze the potential
impact that a second supplier and supply chain could have on long-term
support costs for buying spares, engine parts, and maintenance services.
Given that a large percentage of a fighter aircraft's total life cycle
costs are incurred after it has been acquired and fielded, potential
savings from competition could be significant. Competition could also
yield savings through reliability improvements. The 1998 study stated that
a 10 percent improvement in
Dollars are fiscal year 2002 constant for comparison with the 2002 DOD
break-even analysis.
reliability could allow the user to omit one heavy maintenance cycle of
the engine, saving $3 billion in operating and support costs, and
concluded that the costs for developing the alternate engine would likely
be recovered through production and operational cost savings. The 2002
study did not quantify potential operations and support savings, but
supported the earlier study's conclusions on the benefits from
competition.
Finally, DOD officials indicated that the decision to cancel the engine
competition did not fully consider the other, less quantifiable benefits
that were strongly considered by the program management advisory groups in
recommending the continuation of the program in 1998 and 2002. These
studies concluded an alternative engine program would
o maintain the industrial base for fighter engine technology,
o enhance readiness,
o instill contractor incentives for better performance,
o ensure an operational alternative if the current engine developed
problems, and
o enhance international participation.
Many of these were important benefits realized by past competitions such
as that for the Air Force's F-16 engines. While these benefits are
difficult to quantify, the Air Force engine manager who co-led both
advisory group studies explained that they are valuable when trying to
manage significant numbers of fighter-type engines to ensure combat
readiness. He told us that problems are magnified when trying to manage a
single engine system, which can require substantial manpower and extra
hours to keep aircraft flying when engine problems occur. In his opinion,
the benefits of a dual-source engine would outweigh the costs. He stated
that he had not seen anything that would change this conclusion since the
last advisory group study was conducted.
More Engine Performance Data Are Needed to Reduce Operational Risks
Despite DOD officials' assertions that testing to date has reduced risks,
there has not been sufficient testing to demonstrate that the primary JSF
engine will perform as expected. At the time of the decision to cancel the
alternate engine, the primary JSF engine had only completed about 4,600
hours of ground tests-about one-third of the hours planned-and had not yet
been flight-tested in a JSF aircraft. The first flight of the conventional
takeoff and landing variant aircraft is not expected until October 2006.
First flight of the short takeoff and vertical landing variant is
scheduled for early 2008, and the carrier variant in early 2009. The first
dedicated operational testing that will measure the JSF's operational
effectiveness and suitability is scheduled for 2011. Specific propulsion
flight testing starts out slowly and begins to increase significantly
beginning in 2009 (see fig. 1).
Figure 1: Planned Propulsion Performance Flight Testing and First Flights
for the Three JSF Variants
The maturity of the JSF engine is also reflected in the program's contract
strategy, in which initial production orders for the JSF engines will be
on a cost reimbursement basis. This type of contract is used when the
uncertainties involved in contract performance do not permit costs to be
estimated with sufficient accuracy to use any type of fixed-price
contract. This places the risk from the uncertainties with the buyer-in
this case, DOD. The program plans to transition to fixed-price-type
contracts for the engine when processes stabilize and the system matures,
sometime before full-rate production begins in 2013. This will shift more
risk to the contractor.
DOD officials also cite the good performance to date of the F-22A engine
as reducing the risk from relying on a single source, but this argument
has several qualifications. First, although the JSF primary engine is a
derivative of the F-22A engine, the F-22A has completed only about 20,000
operational engine hours; this represents about 10 percent of the 200,000
hours considered sufficient for system maturity. F-22A program engine
officials noted that while overall performance has been good, the F22A
engine is currently not meeting several reliability goals. For example,
the engine's mean time between maintenance actions was expected to be 100
hours on average at its initial service release in 2002. However, as of
April 2006, the engine was experiencing an average of 60 hours between
maintenance actions. The program projects that at system maturity in 2010,
the F-22A engine mean time between maintenance actions will be about 100
hours, but that is only 50 percent of its performance requirement for
200-hour mean time between maintenance actions. Officials also cited four
other reliability goals, two that are currently being met and two that are
not being met.
Second, the two aircraft have different missions and operational concepts
that may produce different stresses on the engines. The single-engine JSF
aircraft is being designed to rapidly transition between different
air-to-surface and air-to-air missions while still airborne. The JSF
aircraft design has three variants, each with a different operational
concept. In contrast, the dual-engine F-22A will primarily be an
air-to-air fighter that will fly at high speed and high altitude. Both
test and engine officials stated that the operational environment for the
JSF may put more stress on the plane's engine than the operational
environment for the F-22A puts on its engine. According to engine
officials, the fact that the JSF relies on a single engine for its
performance magnifies any potential problems that it may incur and
increases the maintenance needed to sustain its readiness.
Third, the 1998 study stated there was limited commonality between the
F-22A engine and the JSF engine configurations and for that reason there
was limited reduction in development risk achieved from F-22A experience.
The study stated the development risk for the JSF engine was commensurate
with a new fighter engine, and of particular concern were the high
temperature and short takeoff and vertical landing integration
requirements. Engine and acquisition officials we talked to had differing
views on the degree of commonality between the two engines and impacts on
development risk. We note that the development effort on the JSF primary
engine is expected to cost $5.8 billion, indicating a substantial
development effort.
CONCLUSIONS
The relative advantages and disadvantages of the JSF alternate engine
program can change significantly depending upon the factors assessed and
considered. In deciding to terminate the program in 2007, DOD did not
conduct a current and comprehensive study of the costs and benefits of the
alternate program. It relied on selected elements of two older studies
that reviewed the JSF alternate engine program in 1998 and 2002. It
focused on the estimated savings to be accrued from the reduced price to
buy engines based on competition between two sources. It did not consider
the benefits, including potential cost savings, that might be derived from
competition during the life of the JSF program-future engine buys, spare
parts, maintenance, reliability improvements, support improvements,
industrial base benefits, and other longer-term factors. The two prior
studies both concluded that these benefits would be substantial and
sufficient reason to continue the program. In addition, the decision was
based on a break-even analysis for the total investment cost (sunk costs
as well as costs to go) of the alternate engine program. Sunk costs should
be excluded from the break-even analysis upon which the decision is based,
thereby lowering the number of engines required to break even. In deciding
to cancel the competition, DOD determined that it could not afford this
program, given other needs in DOD and the government. It had to find money
for other budget priorities, and the alternate engine program was not
accorded a high enough priority. The question remains whether a more
current, comprehensive, and independent study including all costs,
benefits, and risks-not just up-front procurement costs-would result in a
different answer that would cause DOD to reconsider its decision to cancel
the alternate engine program and instead afford it a higher priority and
cause it to continue the JSF engine competition.
AGENCY COMMENTS AND OUR EVALUATION
DOD provided us with written comments on a draft of this report. The
comments appear in the enclosure to this letter.
In summary, DOD believes the report is misleading in a number of respects
in that there are many important issues that deserve more thoughtful and
balanced consideration than the information presented in our report. It
highlights these in its comments, which include its beliefs that (1) data
showing savings from competition do not exist and (2) certain higher costs
would exist if the competitive alternate engine program continued. We
agree that there is a mix of factors that can increase and decrease costs
as well as influence readiness and the industrial base. However, as
pointed out in DOD comments, there is currently a paucity of empirical
data about the costs and benefits of this program; therefore we think the
issues raised by DOD in its letter need to be considered within an overall
and comprehensive analysis of the life cycle costs and benefits of a
competitive alternate JSF engine program.
Our tasking, from both the Senate Armed Services Committee and the House
Armed Services Tactical Air and Land Forces Subcommittee, was to review
DOD's rationale for canceling the program and the analysis supporting that
decision. It was not our tasking to perform the analysis for DOD. Our
conclusions were that DOD had not conducted a current and comprehensive
study of the costs and benefits of an alternate engine program over the
entire life of the JSF. Instead of undertaking a new study, DOD relied on
selective elements of two older studies, both of which concluded that
significant benefits, beyond price savings in the acquisition program,
justified continuing the alternate engine program. DOD supported and
funded the alternate engine program in prior years but now believes the
program is not cost-beneficial and presents a low risk if canceled. We
found that the affordability pressures caused by other more pressing
demands on the DOD budget this year caused DOD to look for sources of
funding. As a result, DOD viewed the alternate engine program as a lesser
priority within the agency.
We are sending copies of this letter to the Honorable Donald H. Rumsfeld,
Secretary of Defense; the Honorable Michael W. Wynne, Secretary of the Air
Force; and the Honorable Donald C. Winter, Secretary of the Navy. We will
make copies available to other interested parties upon request. The letter
is also available at no charge on the GAO Web site at http://www.gao.gov.
Please contact me at (202) 512-4841 if you or your staffs have any
questions. Contact points for our offices of Congressional Relations and
Public Affairs may be found on the last page of this report. Other major
contributors to this letter were Mike Hazard, Matt Lea, Bruce Fairbairn,
and Gary Middleton.
Enclosure
Enclosure
Comments from the Department of Defense
Enclosure Enclosure
(120548)
*** End of document. ***