BNUMBER: B-272418; B-272418.2; 272418.3
DATE: October 30, 1996
TITLE: Hughes Missile Systems Company
**********************************************************************
DOCUMENT FOR PUBLIC RELEASE
A protected decision was issued on the date below and was subject to a
GAO Protective Order. This version has been redacted or approved by
the parties involved for public release.
Matter of:Hughes Missile Systems Company
File: B-272418; B-272418.2; 272418.3
Date:October 30, 1996
Mark D. Colley, Esq., David P. Metzger, Esq., Laura E. Gasser, Esq.,
Craig A. Holman, Esq., Laura A. Eilers, Esq., Melinda C. Burrows,
Esq., and Bradley D. Wine, Esq., Holland & Knight, and David L.
Fowler, Esq., for the protester.
William A. Roberts III, Esq., Lee P. Curtis, Esq., Jerone C. Cecelic,
Esq., J. Eric Andre, Esq., and Terry Petrie, Esq., Howrey & Simon, and
Bucky P. Mansuy, Esq., for Lockheed Martin Integrated Systems, Inc.,
David V. Anthony, Esq., Gregory A. Smith, Esq., Mary Ita Snyder, Esq.,
Stanley R. Soya, Esq., and John E. Benedict, Esq., Piper & Marbury,
and Stephen E. Seele, Esq., for McDonnell Douglas Aerospace
Corporation, intervenors.
Gregory Petkoff, Esq., John A. Dodds, Esq., Marian E. Sullivan, Esq.,
Capt. Rebecca Pearson, and Gary J. Rosnick, Esq., Department of the
Air Force, for the agency.
David A. Ashen, Esq., and John M. Melody, Esq., Office of the General
Counsel, GAO, participated in the preparation of the decision.
DIGEST
1. Protest against evaluation of past performance is denied where
information reasonably available to the procuring agency supported the
report of the agency administering the protester's contract to the
effect that protester was deficient in its cost/schedule performance
on that contract.
2. Protest that agency did not adequately consider the complexity of
the products being developed and/or procured when evaluating past
cost/schedule performance and arriving at an overall rating, and
instead placed undue emphasis on program complexity, is denied where
offerors were advised prior to the closing date for receipt of
proposals that agency viewed program similarity as more important than
product similarity.
DECISION
Hughes Missile Systems Company protests the Department of the Air
Force's award of contracts to Lockheed Martin Integrated Systems, Inc.
(LM) and McDonnell Douglas Aerospace Corporation (MD), under request
for proposals (RFP) No. F08626-96-R-0002, for the definition and
development of the Joint Air-to-Surface Standoff Missile (JASSM).
Hughes challenges the past performance, cost and technical
evaluations.
We deny the protest.
BACKGROUND
The JASSM missile will be an autonomous, precision strike standoff
weapon, to be carried aboard a number of types of aircraft, which can
be launched from beyond the range of enemy air defenses and will
provide the capability to strike heavily defended, high value targets.
Offerors were advised that the agency considered the need for such a
standoff attack capability to be urgent and compelling. The
solicitation contemplated the award of two cost-plus-fixed-fee
contracts for a 24-month Program Definition and Risk Reduction (PDRR)
phase, which would include priced options for a follow-on
cost-plus-incentive-fee Engineering, Management and Development (EMD)
phase, to be exercised on the basis of a downselect competition
between the two PDRR contractors. The downselect contractor will then
commence production, which is expected to include 10 annual production
lots. Offerors were required to furnish cost and pricing information
for the PDRR and EMD phases and cost estimates for the production
phase; during the downselect competition, the PDRR contractors will
have the opportunity to update their EMD phase pricing and to submit
cost and pricing information for the production quantities.
Award was to be made to the offerors whose proposals were most
advantageous to the government under three broad criteria: (1) past
performance--including past technical performance (with factors for
product performance, computer software, and aircraft integration) and
past affordability performance (with factors for manufacturing and
cost/schedule), which were of equal weight--which was equal in
importance to the aggregate of (2) technical performance (key
performance parameters, other requirements, and integrated master plan
and schedule) and (3) affordability, which were of equal weight. The
affordability criterion included four factors: (1) average unit
procurement price (AUPP) for production lots 1-5 (the most important
factor); (2) mission/cost effectiveness--including the expected number
of missiles to kill the specified JASSM target set, total cost (number
of missiles times most probable AUPP) to kill the JASSM target set,
average cost to kill a JASSM target, and number of aircraft sorties
(missions)--which was equal in importance to (3) total contract price
for the PDRR and EMD phases; and (4) AUPP for production lots 6-10
(the least important factor).
Five proposals--including Hughes's, LM's, and MD's--were received by
the closing time on April 29, 1996. All were included in the
competitive range. Following written and oral discussions with
offerors, the Air Force requested PDRR and EMD "price" proposals.
Based upon the evaluation of offerors' past performance and their
written and oral submissions, the source selection advisory council
emphasized the following considerations in its report to the source
selection authority (SSA): (1) while LM had the strongest past
performance and MD's performance was "consistently good," Hughes's
performance was characterized by a weakness with respect to ability to
perform to cost/schedule; (2) all offerors' proposed missiles exceeded
the minimum key performance parameters; and (3) while Hughes's offer
had the best mission/cost effectiveness rating, with the lowest AUPP
and average cost per target, LM's and MD's proposed and evaluated AUPP
were below the agency's objective unit price of $[DELETED] and their
offers had lower development (PDRR and EMD) costs. The specific
results of the evaluation were as follows:
Hughes LM MD
Past Performance
Product Exceptional (Blue)Acceptable Acceptable
Software Acceptable (Green)Acceptable Acceptable
Aircraft IntegrationAcceptableExceptionalExceptional
ManufacturingAcceptable Exceptional Acceptable
Cost/ScheduleMarginal (Yellow)Acceptable Acceptable
Technical
Key PerformanceExceptional
/Moderate Risk Exceptional
/Moderate Risk Exceptional
/Moderate Risk
Other RequirementsExceptional
/Moderate Risk Exceptional
/Moderate Risk Exceptional
/Moderate Risk
Plan/ScheduleAcceptable
/Moderate Risk Exceptional
/Moderate Risk Exceptional
/Moderate Risk
Affordability
Most Probable
AUPP
(Lots 1-5) $[DELETED] $[DELETED] $[DELETED]Mission/Cost
Effectiveness
Total Missiles
Sorties
$ per TargetThreshold/
Objective
[DELETED]
[DELETED]
$[DELETED]/
$[DELETED] Threshold/
Objective
[DELETED]
[DELETED]
$[DELETED]/
$[DELETED] Threshold/
Objective
[DELETED]
[DELETED]
$[DELETED]/
$[DELETED]
Total Contract
(PDRR and EMD)$[DELETED]$[DELETED] $[DELETED]
Most Probable
AUPP
(Lots 6-10) $[DELETED] $[DELETED] $[DELETED]
Based upon the evaluation record, the SSA concluded that LM's and MD's
offers were most advantageous. Specifically, the SSA noted that only
LM and MD were rated as consistently good-to-excellent performers and,
in particular, only those two were rated acceptable with respect to
past cost/schedule performance. The SSA considered their good
performance in the cost/schedule area to be "a very key discriminator
because the schedule on this program is of paramount importance to the
user due to the critical operational shortfall this system fills." In
contrast, the SSA noted that while Hughes's past performance was
generally good, it was characterized by a significant weakness with
respect to cost/schedule performance as a result of schedule problems
experienced on the Advanced Medium Range Air-to-Air Missile (AMRAAM)
program and the Tomahawk (Cruise Missile) Baseline Improvement Program
(TBIP), and significant cost overruns experienced on the TBIP program.
The SSA found the TBIP cost and schedule problems to be "particularly
troubling as this program is of similar scope to JASSM." In addition,
the SSA noted that only LM's and MD's proposals were rated as superior
(exceptional) with moderate risk under all three technical performance
factors. The SSA specifically concluded that MD's proposed design was
"the lowest risk approach due to their use of a proven off-the-shelf
[DELETED] and existing warheads," while LM's design was "of somewhat
higher risk, principally due to the [DELETED]." In contrast, the SSA
considered Hughes's design to have "the highest technical risk of
these three contractors, driven" by their proposal of a [DELETED]
target seeker and new ([DELETED]) warhead.
In the affordability area, the SSA viewed the fact that LM's and MD's
proposals had the lowest total contract prices for the PDRR and EMD
phases to be more significant than Hughes's "roughly [DELETED]
percent" advantage with respect to estimated AUPP for production lots
1-5, noting that he "did not consider this to be a significant
difference at the beginning of the PDRR phase" and that LM's higher
AUPP was based on a [DELETED]. As for Hughes's advantage with respect
to average cost per target killed, the SSA noted that this was
primarily due to their proposal to develop a new, more powerful
warhead, which made unnecessary for many targets the greater accuracy
afforded by a target seeker and thereby enabled Hughes to propose a
higher proportion of less expensive non-seeker missiles; he considered
this approach to be characterized by higher risk than MD's offer of an
off-the-shelf, proven warhead. The SSA concluded that LM's and MD's
superior past performance, superior technical proposals and lowest
total contract price overshadowed Hughes's evaluated advantage with
respect to AUPP and average cost per target.
Upon learning of the resulting June 17 awards to LM and MD, Hughes
filed this protest with our Office, challenging numerous aspects of
the evaluation. We discuss its primary arguments below.
PAST PERFORMANCE
Background
Hughes challenges the evaluation of past performance. As noted above,
past performance was the single most important evaluation area, with a
weight of approximately 50 percent. Past performance was evaluated
based on a maximum of three contracts/programs for each of the five
past performance factors--product performance, computer software,
aircraft integration, manufacturing and cost/schedule. The RFP
generally stated that "[t]he Government will evaluate past performance
using the most recent and relevant performance data" and that
"[w]henever possible, the Government will gather past performance
[data] on contracts/programs which are similar to the JASSM program in
product types, program phase, complexity, scope, competitive
environment, and contract type." In addition, for each of the five
past performance factors, the solicitation specifically described the
prior contracts/programs that would be considered similar. For
example, with respect to cost/schedule performance, the evaluation of
which is at issue here, the solicitation described a similar
contract/program as one involving "a complex missile program that used
streamlined acquisition initiatives, such as limited government
oversight or small industry-Government teams." Potential offerors
were notified (prior to the issuance of the final RFP) of the
contracts/programs selected for evaluation.
The solicitation stated that the agency would evaluate past
performance under the selected contracts using information from such
sources as Contractor Performance Assessment Reports (CPAR), the
Defense Contract Management Command (DCMC), program offices, other
service organizations and the offerors. It required each offeror to
identify in its offer "data sources and points of contact used to
substantiate his strengths, what he has done to overcome any previous
problems encountered . . . and any mitigating factors that precluded
the offeror from meeting his contract cost, schedule or performance
requirements." Although the solicitation limited the past performance
proposal to 20 pages, potential offerors were advised prior to
issuance of the solicitation that they could furnish data to DCMC
supporting their past performance proposals. (Although advised by the
Air Force on three occasions and by DCMC on a fourth that it could
furnish additional supporting data, Hughes, unlike the other four
offerors, declined the opportunity afforded it to submit additional
data prior to the closing date.) In addition, where the agency
questioned an offeror's past performance in a particular area on a
particular contract, it raised the matter during discussions and
afforded the offeror an opportunity to respond in writing and orally.
(While the agency's points for negotiation (PFN) included limits on
the length of responses, Hughes was afforded an additional 10 pages
for responding to the PFNs concerning its past performance.)
TBIP
As noted above, in evaluating Hughes's past cost/schedule performance,
the agency received reports that Hughes experienced schedule problems
on the AMRAAM air-to-air[1] and TBIP cruise missile programs and
significant cost overruns on the TBIP program. These reports led the
agency to evaluate Hughes's overall cost/schedule performance as
marginal (yellow).
Hughes challenges this evaluation to the extent it is based on
Hughes's TBIP performance, primarily asserting that the agency
undertook only a superficial investigation in this regard and,
furthermore, unreasonably concluded that cost growth on the TBIP
program would inevitability result in schedule delays, when, according
to Hughes, this has not proven to be the case.
We will review an evaluation of an offeror's performance risk to
ensure that it was reasonable and consistent with the stated
evaluation criteria. See Dragon Servs., Inc., B-255354, Feb. 25,
1994, 94-1 CPD para. 151. An evaluation of past performance may be based
on the agency's reasonable perception of inadequate prior performance,
even where the contractor disputes the agency's interpretation of the
facts. See Cessna Aircraft Co., B-261953.5, Feb. 5, 1996, 96-1 CPD para.
132; Rockwell Int'l Corp., B-261953.2; B-261953.6, Nov. 22, 1995, 96-1
CPD para. 34; Pannesma Co. Ltd., B-251688, Apr. 19, 1993, 93-1 CPD para. 333.
The Air Force reasonably determined that Hughes's cost/schedule
performance on the TBIP program had been materially deficient. The
record shows that the Air Force relied on information received from
the Navy TBIP program office (including documentation generated by
Hughes) and from Hughes, which reasonably indicated that Hughes had in
fact experienced cost and schedule problems on the TBIP program.
Specifically, we note that in its April 29 response to the Air Force's
request for an evaluation of Hughes's performance, the Navy TBIP
program office rated both Hughes's cost and schedule performance as a
"2" on a scale of 1-4, with "4" indicating an ability to perform with
little or no government oversight and "1" indicating an inability to
perform; the TBIP program office reported that Hughes had demonstrated
the ability to execute program plans, including implementation of any
changes or recovery initiatives, within cost and schedule only "with
substantial Government oversight."[2] The agency also obtained
specific information and documents from the TBIP program office which
supported the Navy's determination that Hughes had encountered
significant cost/schedule problems in its performance. For example,
although awarded the TBIP cost contract in the amount of $226 million
in September 1994, Hughes encountered sufficient cost growth in the
first year of the contract that in July 1995 it requested Navy
approval for an Over Target Baseline (OTB) of $294.8 million
(ultimately $298.7 million). Indeed, the TBIP program office
furnished Air Force evaluators information indicating that in February
1996 Hughes had identified the "'potential' for additional $42.2M
[million] cost growth," and that based on Hughes's April 1996 briefing
it appeared that the OTB was "becoming obsolete." Hearing transcript
(Tr.) at 1218-1219. (In a July 1996 "TBIP Baseline Status Review,"
Hughes indicated an estimate at completion (EAC) of $381 million for
the TBIP program.) In this regard, the TBIP program manager testified
at the hearing held on this protest that Navy source selection
officials had concluded during the TBIP source selection that Hughes
had significantly underestimated the cost of performance and that, as
a result, the Navy had added an additional $70 million to the TBIP
budget beyond the $226 million proposed by Hughes. Tr. at 1019, 1094.
Further, the information available to the Air Force at the time it was
evaluating Hughes's TBIP performance (April/May 1996) reasonably
indicated that Hughes was encountering performance delays that could
ultimately delay delivery of the TBIP improvements. For example,
Hughes was unable to hold the preliminary design review (PDR) on the
TBIP contract in September 1995, when originally scheduled, and
ultimately was not ready for PDR until February 1996. In the view of
the TBIP program office, Hughes's delay in this regard created the
risk that Hughes would be unable to meet the required date for initial
operating capability (IOC). Tr. at 1013-1015, 1125-1127. In
addition, by letter dated February 26, 1996, Hughes advised the TBIP
program office that the funds allocated to the TBIP contract--which
was incrementally funded on an annual basis and contained a
contractual limitation on the funds available for any given
period--would be insufficient at current rates of expenditure to
continue performance for the last approximately 3� months of the
current contract period. Further, in an April 1996 briefing of the
TBIP program office, Hughes warned the Navy that "TBIP Program IOC
will slip 12 months and FOC [full operating capability] 15 months."
Although Hughes referred to only a 10-month slip in IOC in a May 1996
electronic mail to the TBIP contracting officer, it also referred to a
"15 month replan schedule," while the TBIP program manager testified
that he anticipated a 15-month delay in IOC (and not merely the 10- or
12-month delay forecast by Hughes). Tr. At 1054-1055. Further still,
the TBIP program office furnished Air Force evaluators information
indicating that Hughes was proposing to stop work on several system
capabilities. Tr. at 1218-1219. The fact (as pointed out by Hughes)
that the schedule problems had not yet caused Hughes to miss any
contractually established performance dates did not diminish the
seriousness of Hughes's schedule problems; the only date established
by the TBIP contract was IOC at the conclusion of the contract in the
year 2000 and, again, as noted in an April 1996 briefing chart
furnished to the TBIP program office, Hughes was advising contracting
officials that it assumed that "TBIP Program IOC will slip 12 months."
Tr. at 1013.
When questioned by the Air Force during discussions about its
performance on the TBIP contract, Hughes's response led the agency to
believe that Hughes was experiencing serious cost/schedule problems
for which it bore significant responsibility. In this regard, the Air
Force's PFN in this area noted that there was an "approximate 32%
($72M [million]) [cost overrun] plus another $42M problem identified
by contractor in latest financial review," and a "[p]otential 3�
months schedule delay due to lack of funding plus another 12-month
slip presented by contractor in latest financial review." In its
response, Hughes appeared to concede some responsibility for the cost
overruns, stating that:
"[a]dmittedly, Hughes's bid reflected specific optimistic
assumptions in its cost estimate, in particular related to the
[TBIP] seeker subcontract and engineering labor. In addition,
it is Hughes's position that added work comprises $41.9M of
the $72M cost overrun cited in the PFN."
(Hughes's response also noted that the Navy did not agree with its
position.)
The Air Force reasonably viewed Hughes's statement as an admission
that it had underestimated its costs when competing for the contract
and that the contract was at least $30.1 million over cost
irrespective of any work allegedly added by the Navy. Tr. at 1211,
1235. The agency's conclusion in this regard is consistent with a
Hughes summary of the TBIP cost history obtained by the agency, in
which Hughes attributed the cost growth to an "aggressive bidding
strategy," as well as to "requirements clarifications/evolution." It
is also consistent with the TBIP program manager's position that
Hughes "was largely responsible for the cost overruns" on the TBIP
contract. Tr. at 1019-1020. Further, in its PFN response, Hughes did
not deny the existence of these potential 3�- and 12-month schedule
slips. In our view, the Air Force could (and did) reasonably
attribute the potential 3�-month funding-based delay in the first
instance to Hughes's failure to properly cost its proposed TBIP
contract effort. Although Hughes attributed the "potential 12 month
slip" to the cancellation of another missile program under which its
seeker subcontractor was developing the target seeker Hughes proposed
for TBIP, the information available to the Air Force from the Navy
indicated that the choice of the seeker to propose had been up to
Hughes, and suggested that, in any case, Hughes had unduly delayed in
selecting a replacement seeker. Tr. at 1037, 1045-1046, 1056, 1273.
Having reasonably determined that Hughes was experiencing serious cost
and schedule problems on the TBIP contract for which it bore
significant responsibility, and in view of Hughes's 9-month schedule
delay in its performance on the AMRAAM program, the Air Force
reasonably evaluated Hughes's past cost/schedule performance as
marginal. Further, given the fact that its cost/schedule problems had
occurred on programs that, like the JASSM program, involved missiles,
and the fact that the schedule on the JASSM program was of paramount
importance "due to the critical operational shortfall" the system was
to fill, the SSA, in his cost/technical tradeoff, reasonably viewed
Hughes's deficient past cost/schedule performance as a significant
weakness.
Product Similarity
Hughes more generally argues that the evaluation of its past
cost/schedule performance reflects an overall problem in the
evaluation of past performance. Essentially, Hughes contends that the
agency unreasonably failed to consider the complexity of the products
being developed and/or procured when evaluating past performance and
arriving at an overall rating. It specifically contrasts in this
regard the evaluation of its past cost/schedule performance on the
TBIP program with the evaluation of LM's performance on the Joint
Direct Attack Munition (JDAM) program.
The Air Force examined LM's cost/schedule performance on three
programs that it had determined would furnish useful data as to LM's
likely JASSM cost/schedule performance: (1) JDAM, an inertial
guidance/global positioning system tail kit added to bombs to guide
them to their target; (2) the Hellfire II program, an upgrade to the
Hellfire air-launched anti-tank missile, with a semiactive laser
seeker for guidance; and (3) the Low Altitude Navigation and Targeting
Infrared System for Night (LANTIRN) program, consisting of a
navigation pod and a targeting pod mounted on aircraft and primarily
used for targeting laser-guided air-to-ground missiles. Although LM
experienced schedule delays during the development and initial
production of LANTIRN and Hellfire II, and cost overruns of
approximately 60 percent during the development of Hellfire II, the
Air Force concluded that these problems were overshadowed by the facts
that (1) major problems no longer existed on the LANTIRN program and
LM was performing ahead of schedule, resulting in a LANTIRN program
manager rating of "4," the top rating, (2) there had been no schedule
problems on the JDAM program, and (3) LM was performing or had
performed without cost problems during the Hellfire II production
phase and generally on the LANTIRN and JDAM programs. In particular,
with respect to JDAM, the agency noted that the JDAM program manager
reported that LM had never experienced any cost/schedule problems and,
indeed, had performed ahead of schedule in some instances
notwithstanding a "very challenging" schedule; according to the JDAM
program manager, LM's JDAM performance was "absolutely superb." As a
result, the Air Force rated LM's past cost/schedule performance
acceptable.
Noting the importance of LM's JDAM performance in LM's acceptable
rating, and pointing out that unlike TBIP or AMRAAM, JDAM includes
neither an engine, power supply nor warhead, Hughes contends that the
agency has failed to account for the fact that Hughes's marginal
cost/schedule performance rating was earned for products that were
considerably more complex than LM's JDAM (or even LANTIRN). According
to Hughes, the agency's approach resulted in an overemphasis on
program similarity at the expense of product similarity.
Hughes's position is without merit. The Air Force determined that
LM's performance on the JDAM program would furnish a reliable
predictor of its likely cost/schedule performance on JASSM based
primarily on similarities in program complexity, not product
complexity. Tr. 721-747. This determination was consistent with the
agency's approach to past performance as announced prior to the due
date for receipt of initial proposals. Specifically, worksheets
provided to potential offerors along with the draft RFP (for purposes
of soliciting offeror input into the selection of the contracts to be
evaluated) advised that the Air Force would evaluate the relevance of
contracts for consideration under the cost/schedule factor using seven
criteria: (1) product similarity, with the greatest weight to be
accorded to contracts/programs involving cruise missiles, and lesser
(but still significant) weight to be given to guided missiles and
guided bombs; (2) program phase; (3) system complexity; (4)
subcontractor integration; (5) contract type; (6) contract
environment; and (7) whether the contract/program involved a
streamlined acquisition approach. At most, only two of these factors
(product similarity and system complexity), accounting for only 35 of
105 rating points, related to product complexity. (Moreover, even for
those factors, it was clear that the agency may consider a guided bomb
like JDAM to be somewhat similar to a cruise missile like TBIP since,
although a full 20 points were available under the product similarity
factor for a cruise missile, 12 points were available for a guided
bomb.) In general, the worksheets indicated the agency's position
that for purposes of evaluating past cost/schedule performance,
program similarity would be more significant than product similarity.
Tr. at 748, 773-775. Given the agency's previously announced position
in this regard, the fact that the agency's approach resulted in
programs for products of dissimilar complexity being considered
similarly relevant for purposes of serving as a predictor of future
cost/schedule performance on JASSM based on similarities in program
complexity is not objectionable.
TARGET X
Hughes challenges the evaluation of mission/cost effectiveness under
the affordability factor. The solicitation provided for evaluation of
the probable mission/cost effectiveness of the proposed JASSM missile
based on consideration of the expected number of missiles required to
kill the specified JASSM target set, total cost (number of missiles
times most probable AUPP) to kill the JASSM target set, average cost
to kill a JASSM target, and number of aircraft sorties (missions)
required to kill the target set. In this regard, the ultimate overall
JASSM target set adopted by the Air Force and furnished to offerors
for use in preparing their designs included 20 target types: a
threshold target set of 17 target types and 686 targets, and an
objective target set comprised of all 20 target types and a total of
717 targets. In other words, 3 of the target types and 31 of the
targets were simply objective targets. While the solicitation
provided for the entire JASSM target set, including objective targets,
to be taken into account in evaluating mission/cost effectiveness,
only threshhold target types were considered in the evaluation of
compliance with key technical performance parameters. One of the
objective target types, code-named Target X for purposes of this
protest, was a relatively large target that was most vulnerable to
warheads emphasizing blast effectiveness--e.g., through use of a
thinner warhead casing and more explosive--over penetrating
capability.[3] Target X was not included in the initial draft target
set but, rather, was added in October 1995, approximately 5 months
before the final solicitation was issued.
The prescribed approach adopted to measure missile effectiveness
against most of the target types, including Target X, was the
Department of Defense's Joint Munitions Effectiveness Manual (JMEM)
Open End Methods model. Tr. at 35, 167-168. For a particular warhead
against a particular target, this program calculates a single shot
probability of damage (Pd), that is, the percentage of the target that
a single warhead is likely to destroy. The solicitation provided that
"[t]he offeror shall use only Government provided methodology for
computing Pd."
Prior to issuance of the final solicitation, Hughes, MD, and LM
advised the Air Force that destroying Target X would be difficult.
Hughes advised the agency that the Target X class of targets "drives
the need for" high explosive, and that without its proposed new
[DELETED] warhead, which offered additional blast effectiveness, "it
would cost you another thousand missiles" to destroy the Target X
class of targets. Tr. at 517-518. Likewise, MD cautioned the agency
that Target X "sucks up warheads like crazy," and that if it relied on
a penetrator type of warhead against Target X, it would require
significantly more missiles. Notwithstanding these warnings, the Air
Force retained Target X in the JASSM target set.
Upon evaluating the effectiveness of offerors' proposed warheads,
however, the Air Force noted that the JMEM methodology predicted that
large numbers of missiles would be required to destroy the Target X
targets. Although this was true for the warheads proposed by all of
the offerors, it was particularly true for LM's warhead, which
apparently [DELETED]. Indeed, the agency's initial evaluation
indicated that approximately [DELETED] more LM missiles would be
required to destroy the objective target set (which included Target X)
than Hughes or MD missiles. However, the Air Force concluded that the
data indicated that the outputs from the JMEM model with respect to
the Pd and number of weapons required to destroy Target X were
unreliable due to Target X's vast size. Specifically, the agency
maintains that where the Pd value--that is, the probable single shot
percentage of destruction of the target--is sufficiently small,
inherent uncertainty in the JMEM model renders comparison among
warheads unreliable. Tr. at 47-65.[4] Given the perceived
unreliability of the data with respect to Target X, and the fact that
the solicitation provided for the agency to evaluate the realism of
the primary parameters used to calculate missile effectiveness, the
agency essentially decided to normalize the results to prevent Target
X from becoming a discriminating factor in the source selection; the
agency adjusted the Target X effectiveness of all warheads to that
achieved by the most effective warhead, which was [DELETED].
Hughes argues that it was unreasonable for the Air Force to abandon
the prescribed method (the JMEM model) for calculating missile
effectiveness against Target X, and to instead equalize the warheads'
effectiveness against Target X. Hughes maintains that this had the
effect of improperly eliminating from consideration the likely
superior performance of Hughes's warhead against Target X relative to
the performance of LM's warhead.
It is clear that the Air Force has determined that using the
prescribed evaluation approach--evaluating the effectiveness of
warheads against Target X using the JMEM model--would produce a result
which does not represent the agency's actual minimum needs. According
to the agency, given the relative ineffectiveness of all of the
warheads against Target X, warfighters would be unlikely to mount a
broad, area-based attack with JASSM missiles; they instead likely
would either launch a precision attack against critical elements of
Target X--about which insufficient information was provided to
offerors to permit them to calculate such an attack--or attack Target
X with a different weapon system.
Upon determining that the prescribed evaluation approach would lead to
an overemphasis on Target X and thereby produce a result which did not
represent the agency's actual minimum needs, the agency was faced with
a choice between effectively eliminating Target X from the evaluation
without reopening negotiations or reopening negotiations on the basis
of a revised evaluation approach. In this regard, as a general
matter, source selection officials do not have the discretion to
announce in the solicitation that they will use one evaluation plan
and then follow another without informing all offerors of any
significant changes in the evaluation scheme and affording them an
opportunity to respond to the modified evaluation approach. See
DynCorp, 71 Comp. Gen. 129 (1991), 91-2 CPD para. 575; Colonial Storage
Co.; Paxton Van Lines, Inc., B-253501.5 et al., Oct. 19, 1993, 93-2
CPD para. 234, aff'd, Colonial Storage Co.--Recon., B-253501.8, May 31,
1994, 94-1 CPD para. 335. Here, the Air Force decided to eliminate Target
X from the evaluation but not to reopen negotiations.
Hughes argues that the Air Force made a significant change in the
stated evaluation approach such that it was required to reopen
negotiations so as to afford offerors an opportunity to respond by
modifying their proposals. Specifically, Hughes claims that had it
been advised of the agency's decision, essentially, to remove Target X
from the target list, it would have altered its warhead approach to
account for this, perhaps by choosing a nondevelopmental warhead (and
thereby reducing its evaluated warhead development risk and saving on
the cost of developing the [DELETED]), or perhaps by redesigning the
[DELETED] to optimize its performance against the remaining targets.
(Hughes's chief engineer for the JASSM proposal estimated that
choosing a nondevelopmental warhead could save approximately
$[DELETED] in development costs and reduce Hughes's AUPP by
approximately $[DELETED], while redesigning the [DELETED] could reduce
Hughes's total contract price by approximately $[DELETED] and its AUPP
by approximately $[DELETED]). Tr. 507-518, 540-548, 571-572. (We
note, however, that in its prior submission dated July 25, 1996,
Hughes stated that "Hughes likely would have proposed to use a
significantly less expensive off-the-shelf warhead were it not for
Target X.")
Where the record establishes a procurement deficiency, we will sustain
a protest on that basis only where it resulted in competitive
prejudice; where an agency improperly changes its stated evaluation
approach after receipt of offers, such prejudice exists where the
record shows a reasonable possibility that the protester would have
altered its proposal to its competitive advantage had it been given
the opportunity to respond following the change. Federal Computer
Corp., B-239432, Aug. 29, 1990, 90-2 CPD para. 175; see Global Assocs.
Ltd., B-271693; B-271693.2, Aug. 2, 1996, 96-2 CPD para. 100; Akal Sec.,
Inc., B-261996, Nov. 16, 1995, 95-2 CPD para. 216.
The record in this case is extensive, including contemporaneous
internal documentation of Hughes's design and warhead selection
process and testimony by the Hughes chief engineer for the JASSM
project. Based on a close review of this record, we conclude that, at
most, had the Air Force sought revised proposals, Hughes could have
reduced its development costs by a relatively small amount. While it
is possible that a change in warhead could have resulted in a less
expensive Hughes missile, i.e., a lower AUPP, additional Hughes
missiles would be required to destroy the JASSM target set, so that
overall mission/cost effectiveness would not have materially improved
and might actually have suffered. Given the advantages both LM and MD
had in past performance and technical performance, together worth 75
percent of the evaluation, we share the Air Force's opinion that
Hughes could not have altered its proposal to its competitive
advantage. We reach this view even assuming that given an opportunity
to revise their proposals to reflect the elimination of Target X, LM,
and MD would not have improved their own competitive standing.
As an initial matter, Hughes has not specified which warhead it would
have substituted for the [DELETED]; according to the protester, a
reexamination of its warhead approach "would take thousands of hours
because that's how many hours it took for our engineers to do the
original to determine the best warhead." Tr. at 517, 542, 567. This
inability to specify the particular replacement warhead Hughes would
have selected left the Air Force and the awardees unable to address
specifically Hughes's claim of prejudice, but it also left Hughes with
only speculation about a possible warhead redesign to contradict the
existing record, which leads us to conclude that Hughes would not have
offered a different warhead. In this regard, the JASSM target list
included other blast-intensive targets (besides Target X) and internal
Hughes documentation produced in response to an agency document
request shows that Hughes's proposed [DELETED] warhead clearly would
be the most effective warhead--i.e., it would require fewer total
missiles--against the JASSM target set even after the elimination of
Target X. Tr. at 525-528, 533-539, 549-551, 574-576, 927-928.
For example, Hughes proposed the [DELETED] as a backup in case
developmental problems were encountered on the [DELETED] warhead. Tr.
at 499-500. Approximately 23 percent more JASSM missiles with
[DELETED] warheads than missiles with [DELETED] warheads would be
required to destroy the JASSM target set even after the elimination of
Target X.[5] Further, the Hughes documentation indicates that the
[DELETED] was viewed by Hughes as having a higher risk ("high" risk)
and unit cost than Hughes's [DELETED] warhead design (which was viewed
as "low" risk) from which the [DELETED] design was derived.[6] In
this regard, the Hughes documentation indicates that the [DELETED] was
viewed as having the lowest risk and unit price of any of the warhead
options discussed in the documentation and Hughes's chief engineer
testified that the [DELETED] warhead had the same unit cost and risk
ratings as the [DELETED]. Tr. at 550-551.[7] (According to the
agency, however, since the warhead on Hughes's [DELETED] design was
smaller than the one on the [DELETED] design Hughes's missile
effectiveness ratings would have suffered had it substituted the
[DELETED] for the [DELETED], while the evaluated development risk
would have remained the same.)[8] As for the possibility that Hughes
could have begun developing still another new warhead had it been
advised of the removal of Target X from the target set, we note that
this would not have eliminated the evaluated risk associated with
Hughes's proposal to develop the [DELETED] warhead, but instead could
very well have increased that risk given Hughes's belated start on
designing such a warhead. Since the record--including Hughes's
contemporaneous documentation--indicates that Hughes's [DELETED]
warhead remained its most effective warhead even after removal of
Target X from the JASSM target set, and given the substantial effort
required to change warheads, we simply do not find credible Hughes's
claim that elimination of Target X would have led it to alter its
proposal to its significant competitive advantage.
Furthermore, even if Hughes had selected a different warhead--for
example, the [DELETED]--and, as a result, had been able to reduce its
total contract price for the development phases of the JASSM program
by up to Hughes's estimate of $[DELETED] for the [DELETED] (leaving
its price at $[DELETED], considerably higher than the $[DELETED] and
$[DELETED] prices offered by LM and MD), and also lowered its
estimated AUPP for the [DELETED] production quantities, there is no
basis for concluding that Hughes would have been in line for an award.
Hughes offered no contradiction to agency assertions that the
[DELETED] would have markedly reduced the Hughes mission effectiveness
numbers. Similarly, a $[DELETED] reduction in development cost that
Hughes asserts would result from redesign of the [DELETED], and a
reduction in AUPP, would not have placed Hughes in line for award.
Here, too, agency assertions that Hughes's overall missile
effectiveness numbers would have suffered--because a reduction in
warhead size (which Hughes suggested) would have increased the number
of missiles needed to destroy the JASSM target set--were effectively
uncontradicted by the protester. Hughes's proposal was reasonably
evaluated as having a significant weakness with respect to past
cost/schedule performance, and being less advantageous than LM's or
MD's proposals with respect to overall past performance, which was
worth 50 percent of the evaluation. In addition, Hughes's proposal
was also evaluated as being of higher risk and less advantageous with
respect to technical performance, which was worth another 25 percent
of the evaluation.[9] Further, its total contract price for the
development phases of the JASSM program under any scenario would have
remained substantially higher than either LM's or MD's prices.[10] In
these circumstances, even assuming no improvement in LM's and MD's
proposals had the Air Force reopened negotiations, we see no
reasonable possibility that any improvement in Hughes's position with
respect to the affordability criterion, which accounted for only 25
percent of the evaluation, would be sufficient to offset its evaluated
disadvantages under evaluation criteria worth 75 percent of the
evaluation.
The protest is denied.
Comptroller General
of the United States
1. Information in CPARs (including yellow performance ratings) and
from DCMC indicated that Hughes had experienced schedule delays in its
performance on the AMRAAM program. In response to the PFN issued in
this regard, Hughes conceded that a 9-month schedule delay had been
encountered on an AMRAAM contract, but pointed out that this had not
resulted in a delay on a subsequent production contract and that its
current CPAR performance ratings on the AMRAAM program were either
blue or green. The agency nevertheless considered the delay to be
relevant since it appeared that a delay in production had been avoided
only because the production schedule was able to accommodate the delay
under the preceding contract.
2. Although, as pointed out by Hughes, the TBIP program manager at one
point advised the Air Force that "[i]n general, [he] was pleased with
the performance of both McDonnell Douglas and Hughes" on the overall
Tomahawk program, the record indicates that the program manager
concurred in the evaluation that Hughes had encountered cost and
schedule problems on TBIP and, as noted below, he specifically
testified that Hughes "was largely responsible for the cost overruns"
on the TBIP contract. Tr. at 1012-1024, 1112-1113.
3. Since some of the information concerning the JASSM target set is
classified, our discussion of the Target X issue is somewhat limited.
4. We note, however, that the record indicates that the warheads in
fact possessed different blast capabilities. In this regard, an
agency engineer who participated in the JASSM lethality assessment
testified that the inherent uncertainties in the JMEM model when used
to evaluate the effectiveness of the proposed JASSM warheads against
large targets such as Target X were such that it was "[n]ot
necessarily" the case that the LM warhead would be "considerably less
effective" against Target X than the Hughes warhead. Tr. at 70, 73,
75, 608. However, he also conceded that there was indeed a difference
between Hughes's and LM's warheads; according to the engineer, the
fact that Hughes's warhead [DELETED] will result in Hughes's warhead
producing a bigger blast than LM's, Tr. at 213-214, 237, 605, and a
warhead with a greater blast effectiveness was likely to be more
effective against Target X. Tr. at 607-608. The agency engineer
further conceded that using the effectiveness data for the [DELETED]
warhead would not furnish a more reliable predictor of warhead
effectiveness against Target X than the effectiveness data supplied by
LM for its own warhead. Tr. at 260-261.
5. The [DELETED] warhead's approximate 23 percent advantage in this
regard is based on a comparison between Hughes's effectiveness numbers
for the [DELETED] APW and the evaluated effectiveness of the [DELETED]
at a low circular error probable (CEP). (The CEP denotes the radius
around the target in which 50 percent of the missiles can be expected
to land.) The JASSM program manager testified that the agency
likewise had concluded during its evaluation that the effectiveness
numbers for Hughes's missile would have "declined markedly" and the
unit price increased had Hughes substituted the [DELETED] warhead for
the [DELETED] warhead. Tr. at 915, 925-928, 979-982.
6. The [DELETED] warhead was [DELETED] inches longer, included
approximately [DELETED] pounds more high explosive, and was viewed as
a better penetrator than the [DELETED] warhead. Tr. at 506-507.
7. In this regard, Hughes's chief engineer testified as follows:
"Question: And does this chart show that of these
[warhead] options, the lowest unit cost was for the
[DELETED]?
Answer: Yes.
Question: And do you know if the unit cost of the
[DELETED] also would be the lowest unit cost in
comparison with these other warhead options?
Answer: Yes.
Question: If Target [X] were removed from the target
set, would that change the unit cost on any of these
warhead options?
Answer: These were estimates, but no. . . .
Question: I see the risk assessments for the
[DELETED] warhead candidate, and the risk assessments
across the row are all low. Do you see that?
Answer: Correct.
Question: Would these risk assessments be the same
for the [DELETED] warhead?
Answer: Probably pretty close, yes.
Question: Would any of these risk assessments change
as a result of the removal of Target [X] from the
target set?
Answer: No."
Tr. at 550-551.
8. The Air Force notes that when during discussions it questioned
Hughes's proposed development schedule, Hughes responded that it had
"initiated work on analysis and design of the JASSM [DELETED] series
warhead configurations in September of 1995," that is, prior to the
addition of Target X to the JASSM target set. Hughes's chief
engineer, however, testified that this reference was only "an
after-the-fact term we gave to the entire development series for the
[DELETED] type warheads that we were looking at, like the [DELETED]";
according to the engineer, Hughes only started looking at the
[DELETED] design "when . . . the target set changed, and that was in
October," and "actually arrived at the design in January." Tr. at
508-510.
9. Although Hughes's selection of a nondevelopmental warhead could
have reduced its risk under the technical criterion, its evaluated
higher risk in this area was based on additional considerations beyond
the development risk associated with its new [DELETED] warhead and, in
any case, its substitution of a nondevelopmental warhead such as the
[DELETED] would have resulted in a marked decline in the effectiveness
numbers for its missile.
10. Hughes also challenges the cost/technical tradeoff, primarily on
the ground that the SSA failed adequately to take into account
Hughes's affordability advantage. The record shows that the SSA did
not consider Hughes's "roughly [DELETED] percent" lower AUPP for
production lots 1-5 significant in light of the fact that the AUPP for
LM and MD were not viewed as excessive and the speculative nature of
the AUPP at this stage in the program (in that the program is only at
the beginning of the PDRR phase and production is years away, while
the offerors besides Hughes are not committed to their current
production pricing and Hughes is only committed to its current pricing
for the first 165 missiles of the planned 2,400 missile purchase).
Nothing in the solicitation precluded the SSA from considering the
significance of Hughes's AUPP advantage in light of these factors.