BNUMBER: B-274204.6; B-274204.7
DATE: November 27, 1996
TITLE: Northrop Grumman Corporation; ITT Gilfillan
**********************************************************************
DOCUMENT FOR PUBLIC RELEASE
A protected decision was issued on the date below and was subject to a
GAO Protective Order. This version has been redacted or approved by
the parties involved for public release.
Matter of:Northrop Grumman Corporation; ITT Gilfillan
File: B-274204; B-274204.2; B-274204.3; B-274204.4; B-274204.5;
B-274204.6; B-274204.7
Date:November 27, 1996
Richard P. Rector, Esq., Kevin P. Mullen, Esq., and Chandra Emery,
Esq., Piper & Marbury, for Northrop Grumman Corporation; and Richard
L. Moorhouse, Esq., Dorn C. McGrath III, Esq., Steven A. Diaz, Esq.,
Mary F. Withum, Esq., and Stacey E. Young, Esq., Holland & Knight, for
ITT Gilfillan, the protesters.
Thomas J. Madden, Esq., John J. Pavlick, Jr., Esq., Carla D. Craft,
Esq., Jerome S. Gabig, Jr., and Paul A. Debolt, Esq., Venable,
Baetjer, Howard & Civiletti, for Raytheon Company, an intervenor.
Gregory H. Petkoff, Esq., Marian E. Sullivan, Esq., Roger J. McAvoy,
Esq., Edward L. Fitzmaurice, Jr., Esq., and Stephen T. Davis, Esq.,
Department of the Air Force, for the agency.
David A. Ashen, Esq., and John M. Melody, Esq., Office of the General
Counsel, GAO, participated in the preparation of the decision.
DIGEST
1. Protest that agency improperly determined most likely mean time
between corrective maintenance action (MTBCMA) for each offeror's
proposed airport surveillance radar system, rather than accepting the
proposed MTBCMAs, is denied where solicitation indicated agency's
intent to conduct its own assessment and, moreover, accepting
offerors' proposed figures would render this area of the evaluation
meaningless, since offerors would be expected to employ the most
favorable assumptions in their calculations, leaving the agency with
no basis for comparing the proposed MTBCMAs.
2. Protest that agency improperly failed to disclose during
discussions the mathematical formula used to calculate mean time
between corrective maintenance action (MTBCMA) is denied; such
methodologies for evaluating proposals under the stated evaluation
criteria need not be disclosed by agencies and, in any case, since
nondevelopmental system was required and system operational testing
already had taken place, offerors would not have been in a position to
redesign their proposed systems so as to significantly improve their
MTBCMAs and their competitive standing.
DECISION
Northrop Grumman Corporation (NG) and ITT Gilfillan protest the
Department of the Air Force's award of a contract to Raytheon Company,
under request for proposals (RFP) No. F19628-95-R-0007, for digital
airport surveillance radar (DASR) systems. NG and ITT challenge the
agency's conduct of discussions and evaluation of proposals.
We deny the protests.
BACKGROUND
The solicitation contemplated the award of a contract for site survey,
site design, site preparation, production, installation and logistics
support for up to 213 DASR radar systems. The DASR radar system
includes a primary surveillance radar (PSR) to perform two-dimensional
detection of aircraft and weather intensity and an integrated
monopulse secondary surveillance radar (MSSR) to perform
three-dimensional aircraft position location and identification via
communication with an aircraft transponder. The DASR radar system is
intended to replace Department of Defense (DOD) and Federal Aviation
Administration (FAA) analog radar systems (at low-to-medium density
air traffic control facilities) first fielded 25 years ago, which are
nearing the end of their life cycle and are now subject to sporadic
loss of airport surveillance radar coverage. The DASR radar system
will "improve aircraft detection in clutter, provide National Weather
Service calibrated six level weather, interface to new digital
automation systems, improve reliability, and reduce support costs."
(System Requirements Document (SRD), paragraph 1.0.)
Of particular importance for this protest, and as noted in the
solicitation's executive summary, the solicitation sought the proposal
of "Non-developmental Item (NDI) radars." Toward this end, it
provided that proposed radars would undergo "[a]n Operational
Capability Demonstration (OCD) conducted to verify that the offeror
has a functional, NDI system," and that "the Offeror's NDI product is
representative of the DASR system proposed" and "exists as a 'system'
consisting of all integrated subsystems and other items needed to
operate and maintain the system per the SRD (except for development
required by the contract)." In this regard, the solicitation's
statement of evaluation criteria specifically cautioned offerors that
"[v]ariations between the design proposed in written proposals and the
design/equipment actually demonstrated at the OCD may be considered a
weakness and may add to the evaluated proposal risk rating."
Award was to be made to the responsible offeror whose offer was most
advantageous to the government under three broad criteria: (1)
technical, which was equal in importance to (2) the offeror's
capabilities (as shown by the results of a past and present
performance evaluation, expanded pre-award survey, and possible
in-plant executive review of the offeror's facilities), both of which
were more important than (3) cost/price (for a best estimated quantity
of 89 DASR systems). The technical area included three factors of
equal weight: (1) system performance and system specification, (2)
reliability, maintainability, availability and supportability, and (3)
the offeror's proposed statement of work (SOW), contract data
requirements list (CDRL) and integrated master plan (IMP). Under both
the technical factors for system performance/system specification and
for reliability/maintainability/availability, the solicitation
generally provided for proposal strengths to be given to the extent an
offeror's proposed system specification "captures the desired (i.e.,
'should') SRD requirements in a manner beneficial to the Government";
"[s]pecial emphasis" was to be given to the extent that the offeror's
proposed system offered certain specified "desired characteristics the
Government believes add significant value to the Offeror's product."
(Among these latter characteristics was a desired mean time between
corrective maintenance action (MTBCMA) of 1,070 hours.) Under the
cost/price area, the solicitation provided for calculation of a total
cost/price and for evaluation "of the realism of the Offeror's
proposed cost/price(s)," including an evaluation of "the extent to
which proposed prices and supporting cost data . . . are consistent,
indicate a clear understanding of the solicitation requirements, and
reflect a sound approach to satisfying those requirements."
Four initial technical proposals from three offerors--Raytheon, ITT
and NG (which submitted two independent proposals, only one of which
is relevant here)--were received by the closing time on December 19,
1995. OCDs for each offeror were conducted from February 6 to
February 23, 1996. The Air Force then opened written discussions with
all offerors on April 5, with written responses required by April 22,
and conducted oral discussions during May. Complete proposals,
including cost/price proposals and revised technical proposals, were
received by June 7. Following additional written discussions with
offerors, the Air Force requested best and final offers (BAFO).
Based upon the evaluation of BAFOs and the offerors' capabilities
evaluation, the source selection evaluation board (SSEB) and the
source selection advisory council reported to the source selection
authority (SSA) that, while the evaluation of offerors' past
performance and the results of the expanded pre-award survey indicated
a low performance risk for all three offerors, Raytheon's technical
proposal was superior to that of the other offerors. The specific
results of the evaluation were as follows:
Raytheon NG ITT
TECHNICAL
System
Performance/
Specification Exceptional (Blue)/
Low Risk Acceptable/
Moderate Risk Acceptable/
High Risk
Reliability,
Maintainability,
Availability
and
SupportabilityExceptional/
Low Risk Acceptable/
Low Risk Acceptable/
Moderate Risk
SOW/CDRL
/IMP Acceptable
(Green)/
Low Risk Acceptable/
Low Risk Acceptable/
High Risk
MTBCMA 1,134 Hours 380 Hours 481 Hours
OVERALL
TECHNICAL Exceptional/
Low Risk Acceptable/
Moderate Risk Acceptable/
High Risk
CAPABILITY Low Risk Low Risk Low Risk
COST/PRICE $[DELETED] Million$[DELETED]
Million$[DELETED]
Million
Based upon the evaluation record, the SSA concluded that Raytheon's
offer was most advantageous. The SSA noted that Raytheon's technical
proposal was evaluated as exceptional/low risk and superior to those
of the other two offerors under the system performance/system
specification factor. In this regard, the SSA specifically noted that
Raytheon's proposal, which had no significant weaknesses in this area,
offered several beneficial strengths, including superior MSSR
operating characteristics, easy upgrade to a Mode S radar
capability,[1] a Surveillance Data Translator design which allowed for
easy growth and installation,[2] and a third DASR system to support
the in-plant test program. (In addition, Raytheon's low risk rating
was partly based on the fact that the system demonstrated at OCD was a
current production model pulled from the production line and destined
for a [DELETED] customer.) In contrast, the SSA noted that, although
ITT's proposed Surveillance Data Translator likewise allowed for easy
growth and installation and its technical proposal was considered
acceptable under this factor, the proposal was evaluated as high risk
under the system performance/system specification factor on the basis
that ITT had significantly underestimated the manhours required to
accomplish the proposed software modifications and had failed to
demonstrate a thorough understanding of the throughput requirements
with respect to interfacing with the existing automation system.
Likewise, although NG's proposed system offered several beneficial
operating characteristics, a weather channel design based upon
operationally proven algorithms, and a third in-plant DASR test
system, its proposal was evaluated as moderate risk under this factor
on the basis that NG had significantly underestimated the manhours
required to accomplish the proposed software modifications.
The SSA further noted that Raytheon's technical proposal was evaluated
as exceptional/low risk and superior to those of the other two
offerors under the reliability/maintainability/availability factor.
In this regard, the SSA specifically noted that Raytheon's proposal,
which had no significant weaknesses in this area, offered several
beneficial strengths, including a system with a MTBCMA of 1,134 hours
(that is, in excess of the desired 1,070-hour MTBCMA), a [DELETED]
transmitter and a mature, easy-to-use human interface, and likely
significant software maintenance cost savings from the use of
[DELETED] software development tools. While NG also offered a
[DELETED] transmitter, its proposed system was evaluated as having an
MTBCMA of only 380 hours, significantly below the desired level (and
its proposed MTBCMA of 734 hours). ITT likewise offered a [DELETED]
transmitter, but its system was evaluated as having an MTBCMA of only
481 hours, again, significantly below the desired level of 1,070 hours
(and its proposed MTBCMA of 1,436 hours). In addition, ITT's proposal
was rated as moderate risk under the reliability/maintainability/avail
ability factor on the basis that the system architecture included a
significant amount of hard-to-support [DELETED] software and was
characterized in significant measure by a custom design--rather than
an open architecture design--such that, should the need arise to
replace the [DELETED] processors (which the agency feared would become
unavailable) or associated older hardware, significant redesign would
be required. Further, while both Raytheon's and NG's proposals were
evaluated as acceptable/low risk under the SOW/CDRL/IMP factor, the
SSA noted that ITT's proposal, although considered acceptable, was
rated high risk on the basis of its unrealistically aggressive test
schedule, insufficient test labor hours, and limited understanding of
the site activation process.
The SSA concluded that the overall technical superiority and lower
risk of Raytheon's proposal, which was rated exceptional/low risk
while NG's and ITT's proposals were rated only acceptable/low risk and
acceptable/high risk, respectively, was worth its somewhat higher
cost/price. Upon learning of the resulting August 9 award to
Raytheon, NG and ITT filed these protests with our Office.
NG and ITT raise numerous arguments concerning the agency's conduct of
the procurement generally and the evaluation specifically. We discuss
a number of their most significant arguments below.
MTBCMA
As used by the Air Force, MTBCMA generally measured the mean time
between required immediate, unscheduled corrective maintenance visits
to DASR sites, that is, requirements for maintenance visits other than
the regularly scheduled preventative maintenance visits. For purposes
of calculating MTBCMA, the SRD stated that:
"Corrective maintenance actions are those required to maintain
user confidence in the integrity of the DASR System. Those would
include repair of failed redundant units considered critical to
meeting the DASR System mission of providing PSR, weather and
MSSR data to the automation system."
Although the agency accepted the reliability block diagrams and
component failure rates furnished by each offeror in support of their
estimated MTBCMAs--finding that the failure rates were based on
appropriate reliability analyses and reflected the best estimates of
the actual likely rates of component failures--the agency noted that
each offeror's MTBCMA was based on different assumptions as to what
failed components in their proposed system architecture would
compromise user confidence in the integrity of the DASR system and
when they should be replaced. Accordingly, when discussions failed to
eliminate the differences with respect to the assumed government
maintenance approach, and in order to assure a common basis on which
to measure and compare the MTBCMAs of the proposed systems, the agency
"rationalized" each proposed MTBCMA using its intended maintenance
approach. Thus, in computing the evaluated MTBCMA of each proposed
DASR system, the Air Force essentially assumed that failures of units
in series and in one-of-two redundant configurations--that is, where
only one of the two units is necessary for performance of the function
and the second unit is redundant--would be repaired immediately; it
assumed that repair of a configuration with greater redundancy would
be delayed until the failure of the last redundant unit, such that
failures for a subsystem consisting of three units in which only one
was necessary for operation would not be repaired until two units
failed. This rationalization resulted in evaluated MTBCMAs for NG and
ITT significantly below those estimated by the offerors--that is, an
evaluated 380 hours for NG (734 hours proposed) and 481 hours for ITT
(1,436 hours proposed).
As noted above, in making his source selection decision, the SSA
expressly took into account that NG's and ITT's evaluated MTBCMAs were
significantly below the desired level and Raytheon's MTBCMA. In
addition, the SSA was briefed on the likely cost of the greater or
lesser number of unscheduled corrective maintenance visits to DASR
sites to be expected as a result of MTBCMAs below or above the target
MTBCMA (which was likely to result in eight unscheduled corrective
maintenance visits per year per system). Specifically, the SSA was
advised that NG's MTBCMA of 380 hours was likely to result in an
additional 24,276 unscheduled corrective maintenance visits over the
expected 20-year life of the DASR systems, for an additional total
labor cost of approximately $20 million, and that ITT's MTBCMA of 481
hours was likely to result in an additional 16,372 unscheduled visits,
for an additional labor cost of approximately $13.5 million. In
contrast, the fact that Raytheon's evaluated MTBCMA of 1,134 hours was
likely to result in 755 fewer unscheduled corrective maintenance
visits relative to those likely at the target MTBCMA of 1,070 hours
was expected to result in an approximate $700,000 savings relative to
the target MTBCMA (and thus approximately $20.7 and $14.2 million
savings relative to NG's and ITT's MTBCMAs, respectively). (The
numbers of additional or fewer visits to be expected at the various
MTBCMAs which were briefed to the SSA were mistakenly based on 82 DASR
systems and not the correct total of 89 systems. When corrected to a
total of 89 DASR systems, the resulting number of additional visits
would have been 26,461 visits for NG and 17,845 for ITT, while
Raytheon's evaluated MTBCMA was likely to result in 822 fewer
visits.)[3]
Notice of Evaluation Approach
NG and ITT challenge several aspects of the Air Force's calculation of
the MTBCMAs for the proposed DASR systems and its determination of the
related impact on future maintenance costs. As an initial matter, NG
challenges the Air Force's rationalization of the offerors' proposed
MTBCMAs. NG argues that the agency failed to adequately advise
offerors of its approach to evaluating MTBCMA and that the agency's
action in rationalizing proposed MTBCMAs amounted to the imposition of
an unstated evaluation criterion; the protester essentially maintains
that the agency instead was required to accept the proposed MTBCMAs.
We find that the RFP adequately advised offerors of the agency's
intended approach in evaluating MTBCMA. The solicitation provided for
evaluation of "[t]he Offeror's approach to meeting and substantiating
the DASR SRD RMA [reliability/maintainability/availability] and
supportability requirements," with "[p]roposal strength [to] be given
for the extent to which the Offeror's proposed system specification
captures the desired (i.e., 'should') SRD requirements in a manner
beneficial to the Government" and for "[s]pecial emphasis" to be given
for certain listed desired SRD requirements. One of the "special"
requirements included in the SRD as a reliability/maintainability/avai
lability requirement was a desired DASR system MTBCMA "greater than or
equal to 1070 hours." The evaluation was consistent with this
scheme--the agency considered the offerors' ability to meet the
desired MTBCMA of 1,070 hours. The solicitation also generally
provided for the agency to "assess the risk associated with the
Offeror's proposed approach as it relates to accomplishing the
requirements of this solicitation" and specifically required that
"[a]ll [reliability/maintainability/availability] claims . . . be
substantiated through the use of the Reliability Model for all
elements of the proposed DASR system." We think this was a clear
indication that the agency intended to assess the likelihood that the
offeror's proposed MTBCMA would be achieved in actual operation.
NG's position that the agency was required to accept the offerors'
estimated MTBCMAs--even where such numbers differed from those most
likely to be achieved in actual operation of the proposed systems--is
simply untenable. First, there is no language in the RFP suggesting
that the agency intended such an approach. More importantly, such an
approach would render this area of the evaluation meaningless, since
offerors would be expected to employ the most favorable assumptions in
their calculations, leaving the agency with no basis for comparing the
proposed MTBCMAs.
Inadequate Discussions
NG and ITT argue that the Air Force should have furnished more
detailed information concerning its approach to rationalizing
offerors' proposed MTBCMAs.
The solicitation did not furnish a precise definition of when
corrective maintenance would be required and how to calculate the
MTBCMA of the proposed DASR systems. The definition of MTBCMA in the
SRD did not specify what maintenance actions were required "to
maintain user confidence in the integrity of the DASR System" other
than to indicate that they "would include repair of failed redundant
units considered critical to meeting the DASR System mission of
providing PSR, weather and MSSR data to the automation system."
Further, as noted by the agency, MTBCMA as defined in the SRD "is a
non-standard parameter in the sense that it is not specifically
addressed in the usual handbooks and standards that are commonly found
in nearly every reliability engineer's tool box." Indeed, the SSEB's
technical adviser testified at the hearing conducted on this protest
that he had never seen the term MTBCMA used before. Hearing
Transcript (Tr.) at 646, 647.
However, the Air Force did advise offerors during discussions, where
necessary, of the need for immediate corrective maintenance actions
with respect to failed critical units and of the agency's position
concerning the number of components to be characterized as critical.
For example, the agency issued written clarification requests to
Raytheon and NG advising that their assumption that all failed
redundant units could be replaced at fixed intervals (rather than
immediately) conflicted with their proposed system specifications,
modeled on the SRD MTBCMA provision, which stated that:
"Corrective maintenance actions are those required to maintain
user confidence in the integrity of the [DASR] System. These
[include] repair of failed redundant units considered critical to
meeting the [DASR] system mission."
In addition, NG and Raytheon were furnished with two scenarios to
illustrate the agency's position in this regard. Scenario 1 concerned
a multiple redundancy configuration in which the first failure did not
require immediate corrective maintenance, while Scenario 2 illustrated
a one-of-two-redundancy configuration in which immediate corrective
maintenance was required, as set forth below:
"Scenario 2
--One of two redundant antenna drive motors fail.
--Impact: Only one drive motor operating; integrity of DASR
compromised.
--Immediate corrective maintenance action IS required."
NG's consulting engineer for the DASR proposal (and development
manager in the air traffic control area) acknowledged in testimony
that, as a result of this clarification request and subsequent May
20-22 oral discussions with the agency, NG was aware of "what's
critical, what's not," and of the need for immediate, not deferred,
maintenance of critical units. Tr. at 137. This understanding of the
agency's position as to when corrective maintenance action would be
required was evidenced by NG's subsequent reduction in its proposed
MTBCMA from the 1,130 hours previously proposed to 734 hours in its
BAFO.
NG complains that the agency did not disclose that the mathematical
formula used to calculate MTBCMA included the failure rates of both
the on-line and the redundant unit in a one-of-two redundancy
configuration (and not just the on-line unit), resulting in a
significant reduction to the MTBCMA for NG's proposed system; on the
contrary, the agency allegedly indicated during discussions that only
the failure rate for one of the units would be counted. Tr. at
146-147.
Contracting agencies are not required to disclose their methodology
for evaluating proposals under the stated evaluation criteria.
Lexis-Nexis, B-260023, May 22, 1995, 95-2 CPD para. 14; see DynCorp et
al., B-257037.2 et al., Dec. 15, 1994, 95-1 CPD para. 34. The Air Force
thus was not required to disclose to offerors the formula to be used
in its MTBCMA calculations. Moreover, it does not appear that
offerors were in a position after the OCD to redesign their proposed
systems so as to significantly improve both their MTBCMA and their
competitive position. Again, the DASR solicitation sought the
proposal of NDI radars and required the proposed radars to undergo an
OCD "to verify that the offeror has a functional, NDI system" that
"exists as a 'system' consisting of all integrated subsystems and
other items needed to operate and maintain the system per the SRD
(except for development required by the contract)." Although offerors
were not prohibited from changing their proposed systems after the
OCD, which occurred before discussions commenced, as indicated above,
the solicitation specifically cautioned that variations between the
proposed design and the design/equipment actually demonstrated at the
OCD could be considered a weakness and add to the evaluated proposal
risk.[4] NG argues that it could have retained its proposed design
and improved its evaluated MTBCMA by substituting more optimistic
assumed component failure rates. This argument is unpersuasive--if
more optimistic rates legitimately could have been used, it is unclear
why NG did not do so when it realized that its MTBCMA could be no
higher than 734 hours, that is, well below the desired 1,070 hours.
Moreover, NG's sudden adoption of significantly more optimistic
component failure rates--after having submitted what presumably was
its best, good faith estimate of likely failure rates--reasonably
would have been considered suspect.
ITT argues that its discussions were inadequate because, unlike NG's
and Raytheon's clarification requests, ITT's clarification request did
not include Scenario 2, which set forth a requirement for immediate
corrective maintenance in a hypothetical system where one of two
antenna drive motors fail. However, discussions with different
offerors need not be identical; on the contrary, discussions should be
tailored for different offerors, since the need for revision
ordinarily will vary with each proposal. Delta Data Sys. Corp.,
B-260791, July 21, 1995, 95-2 CPD 42; The Pragma Corp., B-255236 et
al., Feb. 18, 1994, 94-1 CPD 124. The Air Force explains that it was
not necessary to furnish Scenario 2 to ITT because its proposal
indicated that, unlike NG and Raytheon, it understood the requirement
for immediate corrective maintenance when critical components fail;
the only question about ITT's proposal in this area concerned the
firm's understanding as to which components were considered critical,
and the discussions with ITT addressed this concern.[5] The
discussions therefore were adequate.
Reliability of Redundant Configurations
ITT argues that the Air Force improperly used a model for determining
the reliability of configurations with multiple redundancies (that is,
with more than one redundant or backup unit) which had the effect of
understating ITT's MTBCMA; according to the protester, had the agency
used the correct model, ITT's evaluated MTBCMA would have increased
from 481 hours to 651 hours (still 785 fewer hours than ITT proposed).
The Air Force (and Raytheon) maintains that ITT's calculation of the
effect of applying a different model, although purportedly based on
the agency's expected maintenance approach, nevertheless departs from
that maintenance approach by failing to account for the impact of
requiring immediate corrective maintenance when the last redundant
element in a multiple redundant configuration fails and that ITT's
calculation instead assumes deferring maintenance until the next
scheduled preventative maintenance visit.
Our review of the record confirms the Air Force's position that ITT's
calculations fail to take certain aspects of the required maintenance
approach into consideration. For example, although the configuration
of ITT's [DELETED] system is shown to have a two-of-four
redundancy--two of the four units are redundant--in the worksheets
supporting ITT's claimed 651 hour MTBCMA, the worksheets do not appear
to include any allowance for the required immediate repair when the
second unit fails. Taking these discrepancies into account, the
record suggests that, at best, use of ITT's suggested reliability
model would have increased its evaluated MTBCMA to no more than 497
hours (as calculated by Raytheon). (Furthermore, it is not clear why
use of ITT's suggested model would not also have resulted in some
offsetting increase in the MTBCMAs for the other offerors, thereby
eliminating or reducing any improvement in ITT's relative competitive
position in this regard.) Thus there is no basis to conclude that the
reliability model used by the agency resulted in an understated MTBCMA
for ITT.
Raytheon Reliability Data
NG challenges the Air Force's acceptance of Raytheon's estimated
component failure rates, which were based on historical factory and
field test data for the radars proposed. According to the protester,
the assumed component failure rates instead should have been based on
the predictions in Military Handbook No. 217 (MIL-HDBK-217),
"Reliability Prediction of Electronic Equipment," as were NG's
proposed component failure rates. DOD's instructions for use of the
handbook, however, state that "[i]t should be emphasized that if
similar equipment field historical data are available in sufficient
quantity and quality it should always be used in lieu of a prediction.
If the operating or environmental conditions are different, then
modifications can be performed using MIL-HDBK-217 adjustments."
MIL-HDBK-217 Use and Application (Technical Brief, April 1990). These
instructions are consistent with testimony by the SSEB's technical
adviser, who possesses extensive experience in reliability prediction,
that data obtained from the operation of similar equipment in a
similar environment is the most reliable predictor of the reliability
of a radar system and is "a much more reliable predictor and gives a
higher confidence in the prediction" than handbook reliability
predictions. Tr. at 610-612.
Raytheon's estimated component failure rates were based on [DELETED]
hours of experience--more than [DELETED] years of continuous
operation--with its proposed MSSR, and [DELETED] hours--nearly
[DELETED] years of continuous operation--with its proposed PSR.
Testimony by government engineers confirms that there was sufficient
similarity between the factory and field environments in which
Raytheon's radars were tested to justify combining the total test
data; there was a sufficient quantity of test data to form the basis
for a reasonable reliability prediction; and there was sufficient
similarity between the radars tested and those proposed by Raytheon to
render the historical data a reliable predictor of reliability. Tr.
at 612-618, 648-649, 716-734. (If anything, the Raytheon historical
data may understate the likely reliability of its radars; normally,
the reliability of electronic systems improves over time from when it
is first introduced until mid-life. Tr. at 649-650.) Thus, we find
no basis to question the agency's acceptance of Raytheon's estimated
component failure rates.
Additional Corrective Maintenance Visits
In calculating the average labor cost of an unscheduled corrective
maintenance visit to a DASR site, the Air Force made a number of
assumptions, some of which appear to have overstated to some extent
the likely cost of a visit.[6] For example, one of the most
significant determinants of the cost of a visit is the number of
technician man-hours required. It is undisputed that when working on
radar systems accepted safety practice requires the presence of at
least two workers. A working group comprised of persons familiar with
radar system maintenance from the FAA and DOD considered in detail
each activity to be performed in connection with a corrective
maintenance visit, including closing off the current activity of the
technicians after notification of a fault, identifying the faulty unit
(usually to within one of three possible units), obtaining replacement
units from logistics control, travelling to the DASR site from the
maintenance center, accessing the site, coordinating the repair with
operations/maintenance control, repairing the fault, securing the DASR
site, returning to the maintenance center, returning units to
logistics control, and closing out the maintenance action. The
working group concluded that an average corrective maintenance visit
by two technicians to a DASR site will take 6 hours from notification
of fault to close out of the maintenance action, for a total of 12
technician man-hours.
NG challenges the Air Force's estimate of the cost of the additional
unscheduled corrective maintenance visits to DASR sites--$20 million
for NG and $13.5 million for ITT--expected as a result of the lower
evaluated MTBCMAs. NG argues that the required work actually can be
performed in less than 6 hours and with fewer than 12 technician
man-hours.[7]
Our review of the record, including testimony by an experienced
military maintenance technician, suggests that the agency's estimate
of 12 technician
man-hours may in fact be somewhat overstated. This testimony
indicated that (1) after allowing for the fact that not all of the
tasks to be undertaken before departing for the DASR site and after
returning to the maintenance center require two technicians, and (2)
given that the potential for task sharing and concurrent task
performance exists, it may be possible to accomplish the required work
with no more than approximately 7-1/2 technician man-hours (and
perhaps with only a little more than 7 hours). Tr. at 702-710.
However, the military maintenance technician also testified that when
two technicians visit a radar site, which as indicated above is a
safety requirement when working on radar systems, "[t]ypically one
person does it all and the other person watches." Tr. at 711. The
record furnishes no basis for questioning the technician's assessment
in this regard, which was based on approximately 16 years of
experience maintaining air traffic control radar systems. Thus, the
historical record suggests that the potential for task sharing and
concurrent task performance may not in fact be realized in practice.
While we cannot precisely quantify the hours required under this
latter scenario, it appears from our review of the record, including
the numbers suggested by the agency, NG and Raytheon, that after
allowing for the fact that some of the tasks to be undertaken before
departing for the DASR site and after returning to the maintenance
center only require one technician, an average corrective maintenance
visit may require as much as approximately 9-3/8 technician hours.
In addition, although the agency assumed that the technicians
repairing the DASR system will be higher cost civilian employees
rather than military personnel, the agency reports that the FAA will
man no more than 67 of the 89 sites--its own as well as some of the
DOD sites--and it appears reasonable to assume that the remaining 22
sites will be serviced by lower cost military technicians. Further,
it appears unreasonable for the agency to have calculated a work-year
of 1,800 hours by reducing the available time of 2,080 hours--40 hours
times 52 weeks--by expected holidays and vacations, and then
allocating the fully burdened cost of a technician, including a factor
for leave, over 1,800 hours. In so doing, the agency appears to have
double-counted the effect of leave in calculating a burdened hourly
rate for the technicians.
After allowing for military manning of some sites and correcting the
productive man-year to 1,800 hours, and assuming a continuation of
historic working patterns among air traffic control radar maintenance
technicians such that the average corrective maintenance visit
requires approximately 9-3/8 technician hours, it appears that the
additional labor cost to the government of NG's and ITT's lower
MTBCMAs relative to the target MTBCMA (and after giving ITT credit for
an MTBCMA of 497 hours) could total as much as approximately $12.7
million for NG and $8.1 million for ITT.[8] Furthermore, whatever the
precise labor cost of the additional maintenance visits, the very fact
that NG's and ITT's lower evaluated MTBCMAs can reasonably be expected
to result in a requirement for thousands more unscheduled corrective
maintenance visits to DASR sites--26,461 additional visits for NG's
system and 17,845 additional visits for ITT's--than desired by the
agency or required by Raytheon's system could reasonably be evaluated
as a major competitive disadvantage.[9]
IN PRODUCTION STATUS
ITT and NG argue that the Air Force improperly failed to recognize
that their proposed DASR systems, like Raytheon's, were "in
production." As noted by the agency, however, ITT's proposed DASR
system was a combination of modified production subsystems and
engineering model subsystems--e.g., its [DELETED]--integrated as a
complete system for the first time at OCD, while NG's proposed DASR
system, although it included many components currently in production,
required a newly developed [DELETED] and was comprised of subsystems
that must be integrated for the first time to meet the DASR
requirements. In contrast, Raytheon demonstrated at OCD a fully
integrated production system taken directly off the production line
(and scheduled for delivery to [DELETED]), which required only minor
modification to meet the DASR system requirements. The agency thus
reasonably assigned Raytheon's system a relative advantage with
respect to production status (and also reasonably concluded that the
mature character of the system would result in lower risk).
MODE-S UPGRADE/SURVEILLANCE DATA TRANSLATOR GROWTH CAPABILITY
NG and ITT challenge the Air Force's determination that the ease with
which Raytheon's proposed DASR system could be upgraded to Mode-S
capability was a discriminator offering a significant benefit to the
government. As noted by the agency, ITT's approach to upgrading to
Mode-S was to [DELETED], while NG's approach required [DELETED]. In
contrast, Raytheon's Mode-S upgrade approach essentially consisted of
[DELETED]. We find nothing unreasonable in the agency's determination
that the greater ease with which Raytheon's proposed DASR system could
be upgraded to Mode-S capability represented a relative advantage.
Although NG suggests that its Mode-S approach offers other unique,
offsetting advantages based on a more advanced Mode-S configuration,
its position essentially amounts to mere disagreement with the agency
as to the relative desirability of various features and in no way
demonstrates that the agency's focus on the greater simplicity of
Raytheon's approach was unreasonable.
NG also challenges the Air Force's assignment of a strength to
Raytheon's proposal on the basis of the ability of its Surveillance
Data Translator to support more [DELETED] display indicators, an
ability the agency viewed as desirable since it will eliminate the
necessity for the purchase of additional translators at some sites.
NG contends that its proposal should have received similar credit
because its proposed translator likewise can support more [DELETED]
display indicators. However, the contracting officer reports, and the
agency's Proposal Analysis Report confirms, that Raytheon's strength
in this area also was based on the greater ease with which its
translator can support additional display indicators. Specifically,
the agency states that Raytheon's translator [DELETED]. There is no
basis to object to the agency's determination that Raytheon's proposal
was superior on this basis.
[DELETED]
NG and ITT challenge the Air Force's determination that Raytheon's use
of [DELETED] was a discriminator offering a significant benefit to the
government (because the use of [DELETED] increases efficiency during
software development and produces more maintainable code). The
protesters argue that they likewise deserved credit in this regard
since they also proposed to use [DELETED] during software development.
As noted by the agency, however, while [DELETED]. The agency
reasonably rated Raytheon's proposal superior based on Raytheon's
greater experience with [DELETED] and the greater proportion of its
existing code that was developed using [DELETED].
ITT SYSTEM ARCHITECTURE
ITT challenges the Air Force's determination that its system
architecture warranted a rating of moderate risk under the
reliability/maintainability/availability factor. The rating is
unobjectionable. ITT's designs for its [DELETED] and its [DELETED]
use a custom architecture--as opposed to an open, modular architecture
which facilitates substitution of functionally similar parts that
is--designed around [DELETED] processors and uses a total of [DELETED]
lines of [DELETED] software code. (In contrast, ITT's remaining
processors, and all of Raytheon's and NG's processors, are of an open
architecture design.) [DELETED] generally is relatively more
difficult to maintain and support, since it typically is more
difficult to understand and requires as much as three times more lines
of code than higher order languages to develop the same functionality,
with more lines of code resulting in higher maintenance costs. In
addition, the agency determined that should the need arise to replace
the processors (which at least one prior manufacturer had ceased to
produce) or associated older hardware (such as [DELETED]), or to
otherwise upgrade the system during its expected 20-year life using
new processors, ITT's custom design would require significant, costly
and time-consuming hardware and software redesign. Tr. at 505-514,
553-563.[10]
ITT asserts that, had the agency raised this matter during
discussions, it could have advised it of the existence of several
manufacturers still producing [DELETED] processors (or their
equivalent), at least one of which reportedly possesses a significant
inventory of the processors. However, the current availability of the
processors is not a substitute--and does not establish as unreasonable
the agency's preference--for an open system architecture that
facilitates any processor upgrades that might be required during the
20-year life of the system.[11] (In practice, the service life of
radar systems of this type usually extends to 25 to 30 years. Tr. at
506.) There thus is no basis to question ITT's moderate risk rating.
COST/TECHNICAL TRADEOFF
In a negotiated procurement, the government is not required to make
award to the lowest-cost, technically acceptable offeror unless the
RFP specifies that cost will be determinative. General Servs. Eng'g,
Inc., B-245458, Jan. 9, 1992, 92-1 CPD para. 44. Cost/technical tradeoffs
may be made in selecting an awardee, subject only to the tests of
rationality and consistency with the established evaluation factors.
Varian Assocs., Inc., B-238452.4, Dec. 11, 1990, 90-2 CPD para. 478.
Raytheon's technical proposal was evaluated as exceptional/low risk
and superior to those of the other two offerors. First, Raytheon's
technical proposal was evaluated as exceptional/low risk and superior
to NG's and ITT's under the system performance and system
specification factor. Although NG and ITT challenge their moderate
(NG) and high (ITT) risk ratings under this factor, we note that their
proposals were considered as only acceptable, while Raytheon's was
evaluated as exceptional under this factor on the basis of a number of
significant, beneficial strengths, including superior Mode-S upgrade
and Surveillance Data Translator growth capabilities. In addition,
the agency viewed the fact that the Raytheon system demonstrated at
OCD was a current production model to be a significant advantage
indicative of the lower risk associated with its proposal. These
advantages would remain notwithstanding the elimination of NG's and
ITT's higher risk ratings. Second, Raytheon's technical proposal was
evaluated as exceptional/low risk and superior to those of the other
two offerors under the reliability/maintainability/availability
factor. Third, while both Raytheon's and NG's proposals were
evaluated as acceptable/low risk under the SOW/CDRL/IMP factor, ITT's
proposal was rated high risk. Furthermore, Raytheon's proposed DASR
system was evaluated as having a MTBCMA of 1,134 hours, significantly
higher than NG's and ITT's, leading the agency reasonably to conclude
that award to Raytheon would result in fewer unscheduled corrective
maintenance visits and significant labor cost savings. Our review of
the arguments raised by NG and ITT furnishes no basis to question the
agency's determination of the technical superiority of Raytheon's
proposal. Since cost/price was less important than the technical
criterion and, in any case, the evaluated costs of Raytheon's proposal
were not substantially higher than NG's and ITT's, we conclude that
the Air Force reasonably found Raytheon's proposal to be most
advantageous.
The protests are denied.
Comptroller General
of the United States
1. The Mode S radar is designed to (1) identify, locate, and track
aircraft by using radar signals to obtain information from up to 700
individual aircraft at a time and (2) provide users with a
communications channel between aircraft and ground facilities.
2. The surveillance data translator converts the digital target and
weather information from the DASR system and reformats the information
into a form acceptable for use in current air traffic control
automation computers and analog displays.
3. The agency did not calculate the parts costs associated with the
various MTBCMAs.
4. To the extent that NG believes that the agency was required to
furnish a more detailed statement of its rationalization approach in
the solicitation, such argument had to be raised prior to the closing
time for receipt of initial proposals in order to be timely. EG&G
Management Sys., Inc., B-270509.2, Apr. 1, 1996, 96-1 CPD para. 195.
5. Specifically, the Air Force issued ITT a clarification request
stating that:
"You have identified only the [DELETED]and the [DELETED]
as critical elements which need immediate fixing. This
seems to be in conflict with the SRD definition stating
the MTBCMA critical elements include more than just those
two. Please clarify and update availability calculations
as required."
6. Although the SSA was briefed on the likely labor cost of the
additional or fewer unscheduled corrective maintenance visits to DASR
sites expected as a result of the evaluated MTBCMAs and was presented
with a briefing chart which showed the labor cost of the additional
visits added to the cost of the NG and ITT systems, the record read as
a whole (including the source selection decision) indicates that the
additional labor cost essentially was viewed by the SSA as a
quantification of the impact of the MTBCMAs for purposes of the
technical evaluation. According to the SSA, "the quantification of
future potential cost savings was not used by me as a
dollar-for-dollar offset against the evaluated cost/prices in making
my source selection decision." See also Tr. at 83-84, 502-503.
7. In addition, NG argues that the elapsed time of an average visit
actually must be less than 3 hours if the operational availability
requirements for the DASR system are to be met. However, in view of
the agency's detailed review of each activity to be performed in
connection with a corrective maintenance visit, undertaken by a
working group comprised of persons familiar with radar system
maintenance, we do not find such a general allegation to be
persuasive.
8. The above numbers for the additional labor costs resulting from
NG's and ITT's lower MTBCMAs--$12.7 million for NG and $8.1 million
for ITT--and the lower labor costs resulting from Raytheon's higher
MTBCMA ($400,000) were derived from calculations performed by the
parties in response to various scenarios suggested by our Office based
on the record here.
9. NG also argues that the Air Force improperly took into account an
inflation factor (2 percent per annum) in calculating future labor
costs. The agency defends its use of an inflation factor and, in
fact, asserts, with some justification, that an inflation factor of 3
percent per annum would have been more accurate. Even if NG were
correct in its position, the consequent approximate 19-percent
decrease in estimated labor cost it calculates would not affect the
reasonableness of the award decision.
10. Although agency evaluators also noted that there were [DELETED]
Raytheon items for which there was only a single source, the agency
did not view this as a matter of significant concern since the open
architecture of Raytheon's system facilitated the substitution of
functionally equivalent items if that should prove necessary.
11. Further, since redesigning this aspect of its system to move from
a custom to an open architecture and rewriting the extensive body of
[DELETED] code in a higher order software language would have required
a significant effort and redesign, especially in the context of an NDI
procurement, Tr. at 563, the agency was not required to raise the
matter during discussions; agencies are not required to point out
weaknesses derived from an inherent aspect of an offeror's approach
which would require substantial revision to resolve the agency's
concern. Northrop Worldwide Aircraft Servs., Inc., B-262181, Oct. 27,
1995, 95-2 CPD para. 196; Tracor Flight Sys., Inc., B-245132, Dec. 17,
1991, 91-2 CPD para. 549.