TITLE: Raytheon Company, B-291449, January 7, 2003
BNUMBER: B-291449
DATE: January 7, 2003
**********************************************************************
Raytheon Company, B-291449, January 7, 2003
DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective
Order. This redacted version has been approved for public release.
Decision
Matter of: Raytheon Company
File: B-291449
Date: January 7, 2003
William H. Butterfield, Esq., Lawrence M. Prosen, Esq., and Brian Cohen,
Esq., Bell, Boyd & Lloyd, for the protester.
Paul F. Khoury, Esq., Scott M. McCaleb, Esq., William J. Colwell, Esq.,
and Daniel Patrick Graham, Esq., Wiley Rein & Fielding, for The Boeing
Company, an intervenor.
John D. Inazu, Esq., Gregory H. Petkoff, Esq., William H. Kirschiner III,
Esq., and Capt. Donna M. Sikora Snyder, Department of the Air Force, for
the agency.
Henry J. Gorczycki, Esq., and James A. Spangenberg, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision.
DIGEST
In making source selection on procurement for development of advanced
beyond-line-of-sight terminals, source selection authority (SSA) acted
within scope of authority and reasonably in (1) lowering awardee's
proposal's risk rating from high risk, assessed by lower‑level
evaluators because of concerns about its antenna design on two of seven
proposed platforms, to low risk, based on SSA's determination that this
did not represent significant weakness because of non‑technical
programmatic considerations and awardee's mitigation plan addressing this
weakness, and (2) increasing a weakness, assessed by lower-level
evaluators to protester's proposal associated with protester's proposed
reuse of legacy code written using Ada computer language, to a significant
weakness, because this weakness had systemic effect that could inhibit
long-term benefits of entire system.
DECISION
Raytheon Company protests an award to The Boeing Company under request for
proposals (RFP) No. F19628-01-R-0033, issued by the Department of the Air
Force, Air Force Materiel Command, Hanscom Air Force Base, Massachusetts,
for the development of the Family of Advanced Beyond-line-of-sight
Terminals (FAB-T) and related tasks. Raytheon protests the agency's
technical evaluation and source selection decision.
We deny the protest.
BACKGROUND
Currently, the Air Force has over 100 different types of satellite
communication terminals, each of which requires specially trained
personnel to operate and maintain it; the Air Force states that it cannot
afford to continue that communications approach and seeks to replace it
with the FAB-T program. Agency Report, Tab 5E, Technical Requirements
Document (TRD), at 1. A FAB-T is a satellite communications vehicle
consisting of an antenna that transmits and receives radio signals to and
from communications satellites, a modem/processor that provides primary
computer processing resources, operator interface devices that provide the
operator with the capabilities to command and monitor the terminal
operation, and supporting equipment. Hearing Transcript (Tr.) at 7-10;
Agency Report, Tab 5E, TRD, at 4. Terminals can be located on the ground
or on aircraft. The FAB-T will have: a common design approach with
common interfaces that work across multiple platforms, multiple
applications, and for all branches of the U.S. military; multiple
configurations available to meet various user needs; and an architecture
that can be modified (or extended) to incorporate technology/capability
upgrades. Agency Report, Tab 5E, TRD, at 1.
The FAB-T program has an evolutionary acquisition approach. The present
RFP addresses the first increment, which is to provide the FAB-T
architecture and satisfy the TRD. The RFP requires prototype platforms to
be developed for five airborne platforms, i.e., B-2, B-52, E-4B, E-6B, and
RC-135, as well as for two ground platforms, i.e., ground‑fixed and
ground-transportable command posts. This acquisition does not include the
production and delivery of FAB-T equipment.
The RFP, issued March 27, 2002, contemplated the award of a
cost-plus-award-fee contract (with cost-plus-fixed-fee and fixed-price
components and options) for a period of approximately 6 years. The RFP
established a split proposal submission schedule under which cost
proposals were submitted last, after the agency had evaluated technical
proposals. Award would be on a *best value* basis considering three
evaluation factors: (1) mission capability/proposal risk, (2) past
performance and (3) cost. The first two factors were of equal importance
and each was more important than cost. Mission capability/proposal risk
had two subfactors with subfactor 1--architecture and system
performance--more important than subfactor 2‑‑integrated
processes.[1]
The RFP evaluation plan under the mission capability/proposal risk factor
provides for evaluating proposals based on the extent that they address,
meet or exceed the statement of objectives (SOO) for the FAB-T program,[2]
the threshold requirements stated in the TRD, and additional non-mandatory
*objective requirements* (enhancements) in support of future
communications capability identified in the TRD. The RFP included a case
study to evaluate proposals under the architecture and system performance
subfactor to assess such matters as the proposal's flexibility to
accommodate future changes and upgrades, and the ability to provide
engineering development models for the B-2 and the ground command post
terminals by February and October 2006, respectively.
Proposals were to be rated under the mission capability/proposal risk
factor with a color rating and a proposal risk rating. A color
rating--blue (exceptional), green (acceptable), yellow (marginal) or red
(unacceptable)--was to be assigned under each subfactor to reflect the
assessed strengths and inadequacies of each proposal. A proposal risk
rating--low, moderate or high--was to be assigned under each subfactor to
reflect the risks and weaknesses associated with a proposed approach. The
evaluation of proposal risk assessed the potential for disruption of
schedule, increased cost, degradation of performance, the need for
increased government oversight, the likelihood of unsuccessful contract
performance, and any proposed approach to mitigate risk and whether the
mitigation approach is or is not manageable.
The agency received proposals from Raytheon and Boeing, and conducted
multiple rounds of evaluations and discussions. The source selection
authority (SSA) ultimately determined that the two proposals were
essentially equivalent under the past performance and cost factors. In
this regard, both Raytheon and Boeing received a *confidence* past
performance rating, and the difference in evaluated costs was much less
than 1 percent on a $300 million contract. The source selection decision
determined that consideration of the Raytheon and Boeing proposals under
these two factors did not provide a basis for selecting one proposal over
another. Accordingly, the selection decision rested entirely on the
evaluation of proposals under the mission capability/proposal risk
factor. Agency Report, Tab 5B, Source Selection Decision, at 3-5.
The initial and final ratings for the subfactors of the mission
capability/proposal risk factor are stated in the following table:
+------------------------------------------------------------------------+
| |Raytheon |Boeing |
| |---------------------------+----------------------|
| |Initial |Final |Initial |Final |
|---------------------+-------------+-------------+---------+------------|
|Architecture & System|Blue |Blue |Blue |Blue |
|Performance |Low Risk |Moderate Risk|High Risk|Low Risk[3] |
|---------------------+-------------+-------------+---------+------------|
|Integrated Processes |Green |Green |Green |Green |
| |Moderate Risk|Moderate Risk|High Risk|Moderate |
| | | | |Risk |
+------------------------------------------------------------------------+
Because its reasonableness is the key issue in this protest, we reproduce
here a long excerpt from the source selection decision:
Architecture & System Performance Subfactor
Both offerors not only met the threshold requirements, but also proposed
meeting all 11 of the high priority objectives. Additionally, both
offerors proposed meeting a significant number of the 64 remaining
objectives and both offered additional capability beyond that.
Significant positive consideration was given to each offeror for this
added capability, and both proposals were rated as blue.
Both Offerors proposed architectures based on the Software Communication
Architecture (SCA) version 2.2. While applying the SCA outside the
low-frequency domain of the Joint Tactical Radio System (JTRS)[[4]] for
which the SCA was developed has inherent risk for both Offerors, this risk
is consistent with the overall program risk . . . . Boeing's software
approach is based on their SCA infrastructure developed during the JTRS
risk reduction phase, and includes [extremely high frequency (EHF)] and
terminal control software components developed during their internal FAB-T
risk reduction effort. These software components were developed
specifically for operation with the SCA environment and promise
standardized implementation and simplified upgrades. While Raytheon also
proposed compliance with SCA and a standard implementation[,] their
proposal does not assure the simplified upgrades that the Boeing approach
offers. I believe Boeing's architecture is more consistent with the FAB-T
[SOO], as set out in the solicitation, which call for the development of a
Family of Terminals that can be easily upgraded or expanded to incorporate
additional communications capabilities . . . and the provision of a
layered, open system architecture that will provide the foundation to
accommodate future increments in a cost effective and timely manner.
Although both offerors have experience with SCA through the JTRS risk
reduction effort, I believe that the FAB-T program would likely benefit
from Boeing's SCA approach which simplifies future upgrades and from the
lessons learned gained from Boeing's role as the prime contractor for the
Army's Cluster I JTRS program, which is the first [Department of Defense
(DOD)] operational application of SCA.
Boeing's proposal contained one weakness in the Architecture and System
Performance subfactor characterized by the SSET as *significant.* It was
their antenna group design for two of the seven EHF FAB-T
platforms/installations, the B‑52 and RC-135. The SSET was unable
to substantiate the performance (gain and receiver sensitivity) of the
baseline antenna design based on the information in the proposal, and felt
there was risk in achieving the higher range of data rates needed for
these two platforms. Boeing has recognized this risk, and as a risk
mitigator has included [DELETED] in their proposal, with [DELETED] planned
for [DELETED]. Based on consultation with the SSAC, I am convinced that
there exists sufficient antenna expertise within the Boeing team (Boeing
[and other team members]) that this issue could be resolved within the
current schedule and with a minimal cost impact. The potential program
risk is further mitigated in that neither the B[-]2 nor the ground Command
Post Terminal Replacement, the two schedule drivers for this program, make
use of this antenna. Based on this analysis, I believe this issue does
not constitute a Significant Weakness, and that a Low Risk assessment is
more appropriate from my perspective than the Moderate Risk as assigned by
the SSET. . . .
Raytheon's proposal also contained a significant weakness in this
subfactor. Raytheon's reuse of Ada code, designed for the different
software architectures of legacy programs, will adversely influence the
SCA-based FAB-T architecture and seems at odds with the expandability and
upgradeability objectives of the FAB-T program. I believe this is indeed
a significant weakness and will limit the ability to achieve the Family of
Terminals objectives outlined in both the SOO and evaluation criteria.
Further, I am convinced that the long-term viability of Ada and its
industrial base is uncertain, and will likely adversely impact the
upgradeability and supportability of Raytheon's FAB-T software.
Raytheon's proposal includes more objective requirements and enhancements
to EHF functionality. While these additional features and objective
requirements offered in Raytheon's proposal are attractive when compared
to Boeing's approach, Raytheon's proposed adaptation of reuse code and
continued reliance on Ada create long‑term risks which outweigh the
near term advantage of their EHF expertise. I concluded that, in the
aggregate, the Boeing proposal strengths were superior and the weaknesses
less disconcerting in this subfactor than those found in the Raytheon
proposal.
Integrated Processes Subfactor
Both Boeing's and Raytheon's proposals were rated green in this
subfactor. The Boeing proposal had two strengths: providing [DELETED]
which enhances future development and maintenance; and use of [DELETED].
The first strength promises additional flexibility in future upgrades and
the second more efficient program management and thus the potential for
greater cost control. The Boeing proposal had one weakness relating to
test simulation. The weakness is readily correctable and a minor
adjustment was made to the government[']s most probable cost estimate.
The Raytheon proposal had no strengths in this subfactor and one weakness
which was an assessed overestimation of software productivity.
I have concluded that the strengths of the Boeing proposal in this
subfactor offer greater benefit to the government. In addition, because
FAB-T is a software-defined radio based on a new Software Communications
Architecture, success of the program will depend on disciplined and
repeatable processes for the software development and integration effort.
Boeing's CMM level 5[[5]] processes and software build plan increase my
confidence in Boeing's ability to complete software development and
integration.
Agency Report, Tab 5B, Source Selection Decision, at 3-5.
The SSA concluded that Boeing's proposal represented the best value and,
on September 20, the agency awarded the contract to Boeing. Following a
debriefing, Raytheon filed this protest.
PROTEST
Raytheon challenges the agency's evaluation of Boeing's proposed antenna
design and of Raytheon's proposed reuse of Ada code, and challenges the
source selection decision concerning the weight accorded Raytheon's
proposal to provide more enhancements than Boeing's.
The evaluation of technical proposals is primarily the responsibility of
the contracting agency; the agency is responsible for defining its needs
and the best method of accommodating them and must bear the burden of any
difficulties arising out of a defective evaluation. Microcosm, Inc.,
B-277326 et al., Sept. 30, 1997, 97-2 CPD P: 133 at 4. In reviewing an
agency's evaluation and source selection decision, we will not reevaluate
the proposals; we will only review the evaluation to determine whether the
evaluation was reasonable and consistent with the stated evaluation
criteria, and with applicable procurement laws and regulations. Id.;
Gemmo Impianti SpA, B‑290427, Aug. 9, 2002, 2002 CPD P: 146 at 3. A
protester's disagreement with the agency's judgment is not sufficient to
establish that the agency acted unreasonably. Microcosm, Inc., supra.
EVALUATION OF BOEING'S ANTENNA DESIGN
With regard to the evaluation of Boeing's antenna design, the protester
alleges that Boeing's proposal did not meet the stated minimum
requirements for transmit power and receive performance, and thus must be
rated either red (unacceptable) or yellow (marginal) under the
architecture and system performance subfactor. This allegation arises
primarily from the SSET's assessment of risk associated with Boeing's
proposed antenna design for the B-52 and RC-135 aircraft, where the SSET
identified the following significant weakness:
Antenna group design approach may not meet TRD radiated power ([effective
isotropic radiated power (EIRP)]) and sensitivity ([gain to noise
temperature (G/T)]) requirements and will likely result in an inability to
support all required data rates . . .
Agency Report, Tab 6A(5)c, SSET Briefing to SSAC, June 12, 2002, at 33.
The precise EIRP and G/T requirements, which are stated in a classified
attachment to the RFP, are mandatory. Contracting Officer's Statement at
6; Tr. at 33, 359. Boeing's proposal specifically states that it will
comply with these requirements. Boeing Proposal, Vol. IIA, S: 2.6, at
33. Indeed, the SSET did not find Boeing's proposal noncompliant with the
TRD requirements. Tr. at 28, 35, 328, 364‑65, 369. In this regard,
preceding the above-quoted statement of the significant weakness, the SSET
stated that no proposal inadequacies or deficiencies existed and that
Boeing's proposal warranted a *blue* rating, which indicates that the
proposal met and exceeded minimum performance or capability requirements,
for this subfactor. Agency Report, Tab 6A(5)c, SSET Briefing to SSAC,
June 12, 2002, at 30, 32. Our Office also performed an in camera review
of the classified requirements, and of the corresponding classified
portions of Boeing's proposal and the agency's evaluation. Based on our
review, we find nothing in any part of Boeing's proposal that indicated an
intent to take exception to the EIRP and G/T requirements or stated an
inability to support all required data rates. Thus, considering the
developmental nature of this procurement, the record does not support the
protester's allegation of noncompliance with TRD requirements.
The protester alternatively alleges that the reduction of the risk rating
under the architecture and system performance subfactor because of
Boeing's proposed antenna design from high, as initially assessed by the
SSET, to low, as assessed by the SSA, was unreasonable.
The SSET initially assessed the risk as high under this subfactor because
of the identified significant weakness that Boeing's analysis of the
design risks for the antennas for the B-52 and RC-135 platforms did not
support the effects of the particular design features of this antenna on
EIRP and G/T requirements, such that the SSET did not have confidence that
Boeing's antenna design would achieve the higher range of data rates
needed by the agency (in other words, there was a risk it would not). Tr.
at 31-49. The significant weakness and high risk rating under this
subfactor remained even after Boeing provided additional information
addressing the weakness in response to discussions. Agency Report, SSET
Briefings to SSAC, Tab 6A(5)b, Aug. 6, 2002, at 30, 33; Tab 6A(5)A,
Aug. 26, 2002, at 30, 33.
In response to the agency's concerns expressed during discussions, Boeing
also proposed a mitigation plan on July 18 and again in its final proposal
revision on August 19. Protester's Hearing Exhibits, Tab 5, Boeing's Risk
Mitigation Plan, at 1. The plan was to identify and develop [DELETED]
with the development of the [DELETED], during the [DELETED] of the
contract and [DELETED]. If, at the conclusion of this period, the
[DELETED] does not satisfy the agency's concerns, the agency can proceed
with [DELETED]. Essentially, Boeing would assign a [DELETED] to identify
and develop [DELETED]; Boeing [DELETED] for this effort [DELETED]. Id. at
8-9; Tr. at 66-67.
Notwithstanding this mitigation plan, Boeing's antenna design for these
two platforms was still considered to be a significant weakness, which was
reported by the SSAC to the SSA, although the risk rating for this
subfactor was lowered to moderate at the end of the evaluation and
negotiation process in part because of the
mitigation plan. Agency Report, Briefings to SSA, Tab 6A(6)d, June 25,
2002, at 30, 33; Tab 6A(6)c, July 10, 2002, at 33; Tab 6A(6)b, Aug. 7,
2002, at 30, 33; Tab 6A(6)a, Sept. 3, 2002, at 32, 37; Tab 5C, Proposal
Analysis Report, Sept. 13, 2002, at 13; Tr. at 251-52.
The SSA lowered the risk rating under the architecture and system
performance subfactor from moderate to low, and found that Boeing's
antenna design problem did not represent a significant weakness but only a
weakness. Agency Report, Tab 5B, Source Selection Decision, at 4. The
protester notes that this rating change came late in the evaluation
process, and not in close proximity to proposal revisions by Boeing, which
indicates (in the protester's view) that the revised ratings are not
reasonable. However, this type of late change in evaluation ratings was
consistent with the SSA's responsibility and authority. Source selection
officials have broad discretion in determining the manner and extent to
which they will make use of technical and cost evaluation results, subject
only to the tests of rationality and consistency with the evaluation
criteria. KPMG Consulting LLP, B-290716, B‑290716.2, Sep. 23, 2002,
2002 CPD P: 196 at 13; A & W Maint. Servs., Inc.--Recon.,
B‑255711.2, Jan. 17, 1995, 95-1 CPD P: 24 at 4. Here, as explained
below, the SSA, in changing the risk rating for Boeing's proposal,
reasonably exercised and explained his judgment consistent with the
evaluation criteria.
Upon receiving the final evaluation results, the SSA considered the
proposals to be very close, the closest competition he has ever presided
over. Tr. at 196. Thus, the SSA provided feedback to the SSAC and the
SSET chairperson after they presented him briefings on the evaluation of
proposals. Prior to the SSA's feedback, the risk evaluation focused
primarily on Boeing's technical approach with regard to this antenna
design issue; however, the SSA considered other aspects, including
Boeing's proposed mitigation plan that offered and [DELETED] antenna
design, the potential impact on schedule and cost if there was a problem
in Boeing's baseline antenna design, and the relative importance of the
antenna to the overall procurement. Tr. at 172‑76, 202-03, 217-18,
259-60; Agency Report, Tab 5B, Source Selection Decision, at 4.
The SSA considered that the antenna design from which the risk arose
applied only to two of seven platforms. The due date scheduled for
completion of these two platforms was late in 2006, and their development
was not on the critical path of the procurement schedule. Rather, the RFP
identified other platforms as critical and with shorter schedules, i.e.,
the B-2 and ground command post terminals. See RFP S: M.2.2.1. The SSA
thus found that any performance problems associated with the design of the
B-52 and RC‑135 antennas would not adversely affect performance for
the critical path. Tr. at 173, 202-03, 217-19, 259-60; Agency Report, Tab
5B, Source Selection Decision, at 4.
Moreover, the costs associated with those platforms was found to be a
small fraction of the overall program cost, and there also were adequate
costs allocated to support the [DELETED] contemplated by the mitigation
plan. Tr. at 173, 202-03. Furthermore, the SSA found that Boeing had the
capability to overcome the design risk and successfully complete the
design of the proposed baseline antenna, notwithstanding its proposal
weakness in this area, and the SSA expected that outcome.[6] Tr. at
217-19, 241-46.
Thus, when viewed in the overall scope of the procurement, the SSA found
the technical risk associated with the baseline antenna approach to be
narrow, isolated from the critical path, and not of great concern to the
overall cost and success of the contract. Tr. at 172‑76, 202-03,
217-19, 241-46, 259-60; Agency Report, Tab 5B, Source Selection Decision,
at 4. Considering that antenna design was but one aspect of the
architecture and system performance subfactor, the developmental nature of
this procurement, and the fact that the SSA's rationale for changing the
risk evaluation is apparent in the source selection decision, we find this
rating change to be within the discretion of the SSA and reasonable. See
KPMG Consulting LLP, supra, at 13-14. As indicated above, this type of
analysis by the SSA, giving due consideration to the evaluation
conclusions of the lower‑level evaluators, was entirely appropriate
and reasonable. See GTE Hawaiian Tel. Co., Inc., B-276487.2, June 30,
1997, 97-2 CPD P: 21 at 18-19.
The protester alleges, however, that Boeing's mitigation plan did not
propose anything of substance that could reasonably reduce the risk
associated with the antenna design. The record shows that, although the
SSET technical evaluators did not believe that the plan addressed the
risk, the higher-level evaluators did. Protester's Hearing Exhibits, Tab
6, Draft Evaluation Comments, at 2-3; Tr. at 156-58, 162-63, 251-53,
258-65. That is, as indicated above, the SSET evaluators focused on
technical concerns that the baseline approach might not succeed, and since
the mitigation plan did nothing to make the baseline plan more likely to
succeed, the evaluators did not change the high risk rating. Tr. at 68,
156-58. However, as discussed above, the SSA and other high‑level
selection officials looked at other considerations, such as the potential
of the mitigation plan to address cost, schedule and performance risks in
the event the baseline approach proved unworkable. Agency Report, Tab 5C,
Proposal Analysis Report, at 14. While Boeing's plan to [DELETED] did not
identify any technical aspects [DELETED], the basis of this risk
mitigation plan was not to [DELETED] for proposal evaluation purposes, but
to have [DELETED] from which the agency could [DELETED]. As such, the
risk mitigation plan addressed reducing risks associated with contract
cost, schedule and performance in the event of failure of the baseline
antenna approach. These considerations are consistent with the RFP
evaluation plan, which stated that the evaluation of proposal risk would
assess the potential for disruption of schedule, increased cost,
degradation of performance, increased oversight, and unsuccessful contract
performance.[7] See RFP S: M.2.2.
Raytheon also alleges that an assessment by a non-governmental technical
advisor from Massachusetts Institute of Technology (MIT)/Lincoln Labs of
Boeing's antenna design was not reasonably considered by the SSA in making
his source selection.[8] In August, after the SSA had asked questions
about the characterization of the antenna design as a significant
weakness, the SSAC and SSET chairpersons retained the services of this
advisor to assess Boeing's antenna design. Tr. at 64; Protester's Hearing
Exhibits, Tab 3, MIT/Lincoln Labs' Notes. The advisor submitted
handwritten notes to the SSET chairperson that identified concerns not at
issue here, and also confirmed the design risk identified by the SSET.
His notes also stated the following:
Boeing corporate experience certainly includes antenna designers capable
of developing workable design. Unclear as to why that expertise has not
been brought to bear in FAB-T.
Protester's Hearing Exhibits, Tab 3, MIT/Lincoln Labs' Notes at 1. The
advisor went on to identify two likely methods for addressing the risk,
one of which was *extensive rework of feed (almost invention class
work).* Id. at 2. He concluded that either method he had identified
*could and should be able to use same [basic] design* proposed by Boeing.
Id. This analysis does not appear in the written record of the subsequent
briefings to the SSA, nor did the SSA otherwise see the advisor's notes
prior to this protest. Tr. at 174. However, the SSA had been told, prior
to his selection decision, that the advisor had reviewed Boeing's antenna
design, had confirmed the nature of the risk identified by the agency's
evaluation, and had concluded that Boeing had the expertise to address the
risk and make the design workable. Tr. at 216-19.
Raytheon essentially alleges that the advisor's notes express doubt about
Boeing's ability to make its proposed design work, and that the SSA was
not given an accurate summary of the advisor's analysis. We disagree.
Although the SSA was not informed of the advisor's comment about Boeing's
experience not having been brought to bear in its proposal, nor that the
advisor characterized one of the methods of resolution as *invention
class* work, he was told that the advisor corroborated the agency's
evaluation of the risk and that the risk could be resolved. Tr. at
217-18, 331-32. The advisor's opinion was sought to determine whether the
SSET's evaluation of the risk was reasonable, which it did; it was not
sought to replace the agency's detailed evaluation of the risk, which was
presented to the SSA. Tr. at 64-66, 160-61. Thus, contrary to the
protester's arguments, we think that SSA was presented with an accurate
assessment of the evaluated significant weakness and associated high risk
rating with regard to Boeing's antenna design.
In sum, we find the SSA had a reasonable basis to change Boeing's risk
rating under the architecture and system performance subfactor to low, and
to regard Boeing's antenna design as something less than a significant
weakness.
EVALUATION OF RAYTHEON'S PROPOSED REUSE OF ADA CODE
The protester alleges that the agency unreasonably evaluated Raytheon's
proposed reuse of Ada computer software code as a significant weakness,
which unreasonably caused its proposal to be rated a moderate risk under
the architecture and system performance subfactor. The protester also
challenges the agency's reclassification of this risk/weakness from the
evaluation under the integrated processes subfactor to the architecture
and system performance subfactor.
Raytheon proposed to reuse a large amount of code previously written for
other applications (i.e., legacy code), instead of writing new code. Much
of this reused legacy code was written using Ada computer language.
Initially, the SSET evaluated a significant risk in Raytheon's proposal
under the integrated processes subfactor. This significant weakness was
attributed to: (1) an unrealistic productivity rate for writing software;
(2) reuse of software from another procurement that the agency believed
was being developed concurrently with the FAB-T procurement, so that
Raytheon's proposal was dependent on the success of the concurrent
procurement; and (3) reuse of multiple programs with multiple
languages/programming environments that presented supportability and
upgrade risks, which could affect life cycle costs. Agency Report, Tab
6A(5)c, SSET Briefing to SSAC (June 12, 2002), at 68. Taken together, the
agency associated these concerns with Raytheon's management processes and
scheduling, and thus evaluated the risk under the integrated processes
subfactor. Tr. at 79-83. However, as discussions were conducted, the
first concern decreased, and the second was eliminated based on schedule
clarifications. Tr. at 84-85.
The third concern remained and, as a result of discussions, the SSET began
to better understand the risk associated with reusing multiple programs
with multiple languages for this particular application. This risk was
closely tied to Raytheon's proposed use of the Software Communication
Architecture (SCA) version 2.2 for the FAB-T system architecture.
DOD has developed the SCA over recent years, although the SCA was not a
requirement of this RFP. The SCA is designed to permit a complex software
system to evolve over time. It is a layered and evolvable approach in
which components of the software framework can be developed independently
and be incorporated easily into the framework. This approach requires
clear definitions of interfaces between components to facilitate replacing
or adding components that are developed over time to take advantage of
evolving technologies. The initial defining of the interfaces is
difficult and requires significant work up-front to develop.
Tr. at 93-97.
Although both offerors stated that they planned to use the SCA, neither
offeror's initial proposal addressed it in detail. Tr. at 91-93. Thus,
as discussions progressed, the agency's subsequent rounds of evaluations
reflected the agency's better understanding of the proposed SCA
approaches. The evaluated risk of Raytheon's proposal to reuse multiple
programs with multiple languages became associated with system
performance, and thus with the architecture and system performance
subfactor, rather than the integrated processes subfactor. Tr. at 87-93,
96-97. The agency's later evaluation of Raytheon's proposal reflects a
concern regarding the performance risk associated with using large amounts
of legacy software designed for applications other than the SCA. The
concern was not that integration of legacy software could not be done, but
rather that risks associated with cost and schedule increases arose, as
well as a risk that the primary long-term advantage of using the SCA--the
ease of replacing or adding software components in the future--would be
compromised. Tr. at 90-91.
More specifically, the legacy software proposed for reuse by Raytheon was
not developed for the SCA and could not be incorporated into the SCA
without writing new portions of code to make the interface possible.[9]
This process of modifying software code to facilitate the interfacing of
legacy software with the SCA is known as *wrapping.* Tr. at 123-24, 191,
478-80. The risks associated with *wrapping* legacy software to function
within the SCA include *legacy drag,* which means that evolution of the
system is slowed by the adaptations and compromises that have been made to
make the legacy software work within the system. Every new development to
the system may require modifications to the legacy applications or
adaptations to the SCA environment. The agency believed that such
modifications would likely be a step away from the SCA framework, and thus
a loss of some of the long-term benefits of using the open architecture
that is the SCA. Tr. at 109-24, 190‑92, 479-80. This is a risk to
system performance.
In addition, a significant portion of Raytheon's reused code was in the
Ada computer language. The agency, and particularly the SSA, consider Ada
to be a language in decline. Tr. at 187, 225-27. In light of this, the
agency found risk that obtaining qualified personnel to maintain and
revise this legacy code may become increasingly difficult during the
25-year life cycle of the FAB-T program. Tr. at 82-83, 91, 101,
225-27.
The risk of Raytheon's use of the reused Ada code was first evaluated as a
weakness and moderate risk under the architecture and system subfactor by
the SSAC in July. Agency Report, Tab 6A(6)c, SSAC Briefing to SSA (July
10, 2002), at 37, 41. The SSET thereafter affirmed that rating. Agency
Report, Tab 6A(5)b, SSET Briefing to the SSAC (Aug. 6, 2002), at 37, 41.
This was the evaluation presented to the SSA.
As indicated above, the SSA considered this to be a very close competition
and sought additional information after reviewing the evaluation results.
In this case, he further considered the software code reuse issue in
Raytheon's proposal and sought a top-level review by the Carnegie Mellon
Software Engineering Institute (SEI), a non‑governmental advisor
identified in the RFP. SEI confirmed that Raytheon's proposal presented a
risk arising from substantial reuse of legacy code with the SCA and from
use of non-mainstream programming languages, such as Ada, which result in
significant risk to program cost, schedule and performance. Agency
Report, Tab 6A(4)a, SEI Outbrief to SSA (Aug. 9, 2002).
Based on his review, with the concurrence of representatives of the SSET
and SSAC, the SSA determined that Raytheon's reuse of Ada code represented
a significant weakness under the architecture and system performance
subfactor.[10] Agency Report, Tab 5B, Source Selection Document, at 4;
Tr. at 425-26.
We think that this evaluation was reasonable. The SSA's personal
experience indicated to him that for the reasons detailed above the reuse
of legacy code for this application was risky, even though not
impossible. The SSA's experience also led him to the conclusion that the
future supportability of the Ada computer language was a serious concern.
Tr. at 177-83, 187‑92. Although there is evidence that individual
evaluators may have disagreed that Raytheon's proposal presented a
*significant weakness* on either of these points, Tr. at 222-24, 469;
Protester's Hearing Exhibits, Tabs 21-27, Point Paper and E-mails from
MITRE Representatives Regarding the Raytheon's Ada Reuse Approach,[11]
these are matters that are predominantly professional judgments on which
different authorities may reasonably have different opinions. Here, the
SSA's judgment was based on his own experience and supported by the
testimony of knowledgeable experts. Tr. at 235-36, 484-85. The protester
has not shown that substantial reuse of legacy code prepared for
applications other than the SCA does not carry with it the risk identified
by the agency when integrated into an SCA system. Also, although the
protester has presented evidence to show that some experts consider the
future of Ada sound, the record before us shows that, at best, the future
of Ada is hotly debated within the computer science industry; the
existence of such a debate supports, rather than weakens, the SSA's
determination, which we find reasonable.
Raytheon alleges that this evaluation is contrary to one of the stated
objectives of the FAB-T program: to leverage commercial-off-the-shelf,
government-off-the-shelf, non‑developmental items, and best
commercial practices, where possible. RFP, attach. 3, SOO, at 2. While
we agree that this objective encourages appropriate use of legacy
software, this objective must not be considered in isolation from the
other SOO objectives. Other objectives in the SOO include developing a
family of terminals that can easily be upgraded or expanded to incorporate
additional communication capabilities, and providing a layered, open
system architecture that will provide the foundation to accommodate future
increments in a cost effective and timely manner. Id. at 1. The agency
reasonably evaluated that Raytheon's selection of legacy software created
a risk to open architecture system performance, ease of upgrading and
expanding, and accommodating future increments in a timely manner.
Tr. at 126-27. This evaluation is not inconsistent with the SOO.
The protester also alleges that the evaluation concerning the
supportability of Ada legacy software did not reasonably consider
Raytheon's risk mitigation proposal, which was a proposed contract clause
under which Raytheon [DELETED] for Ada software for [DELETED] years at
[DELETED] in order to ensure that the government will have [DELETED] *at a
rate not to exceed what the Government would expect to pay to a third
party supplier for the same or similar service.* Raytheon Proposal, vol.
IV (Aug. 19, 2002), ESC-H-RAY3, at 18; Protester's Hearing Exhibits, Tab
8, Raytheon Letter & Risk Mitigation Plan (Aug. 19, 2002) with attach., at
1, 6. While this clause guarantees that the agency will have at least one
source available for [DELETED] regardless of the state of Ada within the
industry, it does not specifically control the cost risk associated with
those services; specifically, *what the Government would expect to pay to
a third party supplier* could be high to reflect the scarcity of such
services in the marketplace. This plan also does nothing to address the
risk of legacy drag on the system or other aspects of the risk evaluated
by the agency. Tr. at 106-08, 129-32.
The protester further alleges that, since (in contrast to Raytheon's
specific plan) the risk mitigation offered by Boeing had little substance
and did not address the evaluated risk in the proposed antenna approach,
the agency treated the offerors unequally by reducing Boeing's evaluated
risk and not similarly reducing Raytheon's evaluated risk. While we
believe that both offerors' proposed mitigation plans were limited in
scope and did not fully address the respective risks, we think that the
agency reasonably found that the nature of the two risks was markedly
different.
As explained above, Boeing's risk was found to be limited to a portion of
the requirements that were not part of the critical path and comprised
only a fraction of the overall contract cost; Raytheon's risk was systemic
in that the performance risk could result in adverse effects on the entire
architecture. Also, the agency reasonably found that Boeing's risk would
either be resolved in the first 6 months of the contract or the agency
could choose from among [DELETED] other options in comparable stages of
development so as not to adversely affect the schedule of performance;
Raytheon's risk would be recurring for as long as wrapped code and
corresponding architectural compromises remained in the system.
Furthermore, the potential costs associated with Boeing's risk were found
to be smaller and more quantifiable than were Raytheon's. Also, Boeing
[DELETED] associated with its mitigation plan [DELETED]; Raytheon did
not.[12] In sum, the agency's reduction in the risk rating for Boeing's
proposal was based on more than the proposed mitigation plan; it was based
on the limited effect of the risk on overall contract performance, cost
and schedule concerns. Raytheon's proposal risk was not so limited. We
think the agency's actions were reasonable and did not constitute unequal
treatment.
Finally, Raytheon alleges that the agency unreasonably shifted the
evaluation of this risk from the less important subfactor, integrated
processes, to the more important subfactor, architecture and system
performance, and did so to prejudice Raytheon. We disagree. Although the
agency initially viewed this risk as part of a management process risk and
evaluated it under the applicable integrated processes subfactor, as
discussed above, the nature of the risk shifted to one associated with the
offeror's technical approach. This shift in the nature of the evaluated
risk occurred as the agency better understood the proposal's impact on
system performance and ability to upgrade the system after Raytheon
presented details about its SCA framework and approach. The architecture
and system performance subfactor is the appropriate subfactor for
assessing this type of risk. RFP S: M.2.2.1; Tr. at 96-99, 269‑70,
291. As for the allegation that the agency actions in this respect were
with the intent to prejudice Raytheon, there is no evidence to support the
allegation.[13]
THE SOURCE SELECTION DECISION
The protester alleges that the agency did not reasonably account for the
*objective* or *additional requirements* (i.e., features attractive to the
agency as identified in the RFP, but not mandatory) that Raytheon's
proposal offered to provide over and above those offered in Boeing's
proposal under the architecture and system performance subfactor.
Boeing proposed [DELETED] *objective requirements,* of which the agency
deemed [DELETED] significant, and [DELETED] *additional requirements,* of
which the agency deemed [DELETED] significant. Agency Report, Tab 5C,
Proposal Analysis Report (Sept. 13, 2002), at 12‑13. Raytheon
proposed [DELETED] objective requirements, of which the agency deemed
[DELETED] significant, and [DELETED] additional requirements, of which the
agency deemed [DELETED] significant. Id. at 14-15.
The protester essentially contends that the sheer numerical advantage of
the objective and additional requirements in Raytheon's proposal should
require that it be rated higher than Boeing's proposal under the
architecture and system performance subfactor, and thus Raytheon's
proposal should be selected for award over Boeing's proposal.
However, the SSA clearly knew of Raytheon's numerical advantage in offered
additional features and expressly addressed it in the source selection
document, stating that although Raytheon's additional features were
attractive when compared to Boeing's, Raytheon's proposed adaptation of
reuse code and reliance on Ada create long-term risks that outweighed any
advantage in Raytheon's more numerous additional features. The SSA
favored Boeing's proposal under the subfactor, concluding that Boeing's
strengths were superior and its weaknesses less disconcerting than
Raytheon's. Tr. at 195-96, 206-13. Raytheon does not identify any
specific additional features or group of features that would refute the
SSA's judgment under this subfactor.
Moreover, under the integrated processes subfactor, the agency evaluated
Boeing's proposal as having two strengths and one readily correctible
weakness; Raytheon's had no strengths and one weakness. Boeing's
strengths provided additional flexibility in future upgrades, and more
efficient program management that had the potential for greater cost
control. Raytheon's weakness was that the proposal overestimated software
productivity. The SSA concluded that Boeing's strengths offered greater
benefit to the government under this subfactor. Furthermore, since the
success of this software-based satellite terminal would depend on software
development and integration, Boeing's higher software certification (CMM
level 5, the highest level possible, versus Raytheon's CMM level 3)
provided the SSA with higher confidence in Boeing's ability to complete
software development and integration. Tr. at 207-11. Raytheon has not
challenged this determination.
In sum, the SSA determined that Boeing's proposal was superior under both
subfactors under the mission capability/proposal risk factor, the only
evaluation criterion under which the proposals differed. The record
supports the SSA's selection decision, and the protester has not shown it
to be unreasonable or inconsistent with the RFP or applicable procurement
law and regulation.
The protest is denied.
Anthony H. Gamboa
General Counsel
------------------------
[1] The architecture and system performance subfactor generally concerns
the technical aspects of the proposal and the integrated processes
subfactor generally concerns management and quality control. Tr. at
77-79.
[2] The SOO defines the top-level objectives for the FAB-T program. These
objectives include developing a family of terminals that can easily be
upgraded or expanded to incorporate additional communications
capabilities; providing a layered, open system architecture that will
provide the foundation to accommodate future increments in a cost
effective and timely manner; minimizing the number of unique
configurations; and leveraging commercial-off-the-shelf,
government-off-the-shelf, non-developmental items and best commercial
practices, where possible. RFP attach. 3, SOO, at 1-2.
[3] Boeing's risk rating under this subfactor was first reduced from high
to moderate by the source selection evaluation team (SSET). Agency
Report, Tab 6A(5)a, SSET Briefing to Source Selection Advisory Council
(SSAC) (Aug. 26, 2002), at 30. As discussed below, the SSA reduced this
rating to low in the source selection decision. Agency Report, Tab 5B,
Source Selection Decision, at 4.
[4] JTRS is another acquisition program under which both Boeing and
Raytheon participated. Tr. at 184-85, 288, 452-53.
[5] The Capability Maturity Model (CMM) rating assesses a firm's ability
to develop quality software. Tr. at 209. The RFP required a minimum
rating of CMM level 3. RFP S: L‑III-4.2.2.
[6] Thus, contrary to the protester's assertion, the source selection
decision considered Boeing's baseline antenna design.
[7] The protester also alleges that the evaluation revisions are suspect
because they followed a letter from Boeing questioning whether the agency
had considered its proposed mitigation plan. See Protester's Hearing
Exhibits, Tab 7, Letter from Boeing to the Air Force (Aug. 23, 2002).
However, since this letter was not submitted by Boeing as part of the
normal procurement correspondence, the agency did not consider it; it was
not part of the materials evaluated and upon which the source selection
decision was made. Tr. at 423.
[8] The RFP stated that the agency might use this and other
non-governmental advisors to review and analyze proposals. RFP at K-15.
[9] Boeing also proposed to reuse code; however, Boeing's proposal did not
contemplate reuse to the same extent as Raytheon's, and the code to be
reused was developed for the SCA. Thus, Boeing's proposal did not incur
the same risk as Raytheon's. Tr. at 127-28, 186-87, 199-200, 207, 213.
[10] The SSA did not change the moderate risk rating under the
architecture and system performance subfactor. Tr. at 430.
[11] MITRE was also used by the agency to assist in the evaluation.
[12] The agency adjusted Raytheon's proposed costs upwards to account for
the agency's estimate of increased costs associated with the evaluated
risk associated with the reuse of Ada code. The protester alleges that it
was unreasonable to both increase the offeror's evaluated cost and assess
a moderate risk to the proposal. We disagree. An agency is not
prohibited from making cost realism adjustments and also downgrading a
technical proposal, where, as here, the cost adjustments are necessary to
reflect the offeror's probable costs of performance based on its proposal,
and the weaknesses assessed in the offeror's technical evaluation reflect
the performance risk stemming from the inadequacy of the technical
proposal. See Basic Contracting Servs., Inc., B‑284649, May 18,
2000, 2000 CPD P: 120 at 12. Here, regardless of the cost adjustment to
Raytheon's proposed costs, the risk evaluated to exist in Raytheon's
proposed reuse of legacy code written in the Ada language includes
performance and schedule risks, which were not accounted for in the upward
cost adjustment.
[13] The protester quotes vague statements of evaluators/technical
advisors expressing disagreement with the evaluation of risk associated
with reuse of legacy software and the Ada language. As discussed above,
the evaluation of the risk was reasonable, even if individual evaluators
did not agree with it. None of these statements indicated that the agency
was making the evaluation decision it made with the express intent of
either harming or favoring either offeror. While the wording of some of
the comments from the lower-level evaluators leaves room for creative
inference, it is the protester's burden to show more than inference or
suspicion in alleging that source selection officials acted with intent to
harm the protester, a burden the protester did not satisfy here. See E.F.
Felt Co., Inc., B‑289295, Feb. 6, 2002, 2002 CPD P: 37 at 3-4.