TITLE:  Systems Management, Inc.; Qualimetrics, Inc., B-287032.5; B-287032.6, November 19, 2001
BNUMBER:  B-287032.5; B-287032.6
DATE:  November 19, 2001
**********************************************************************
Decision

Matter of: Systems Management, Inc.; Qualimetrics, Inc.

File: B-287032.5; B-287032.6

Date: November 19, 2001

Donald J. Walsh, Esq., Scaldara & Potler, for the protesters.

William A. Shook, Esq., and Kelley P. Doran, Esq., Preston Gates Ellis &
Rouvelas, Meeds, for Coastal Environmental Systems, the intervenor.

John D. Inazu, Esq., Gregory Petkoff, Esq., Richard C. Bean, Esq., and Maj.
Marc Fox, Department of the Air Force, for the agency.

Katherine I. Riback, Esq., and James A. Spangenberg, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision

DIGEST

1. Agency was not required to consider in the past performance evaluation
the performance of all proposed subcontractors of the offerors or all
references submitted by an offeror.

2. Agency reasonably determined, consistent with the stated evaluation
factors, that there were strengths in the awardee's higher-priced technical
proposal that warranted the award selection based on a cost/technical
tradeoff.

DECISION

Systems Management, Inc. (SMI) and Qualimetrics, Inc. protest an award of a
fixed-price indefinite-quantity contract to Coastal Environmental Systems
under request for proposals (RFP) No. F19628-00-R-0032, issued as a small
business set-aside by the Department of the Air Force for a fixed-base
weather observation system for the 21st century (OS-21 FBS). [1] The
protesters challenge the Air Force's evaluation of their and the awardee's
proposals.

We deny the protests.

BACKGROUND

The OS-21 FBS represents an integrated system of multiple weather sensors
and data automation components that is designed to continually measure the
conditions of the environment at military installations in support of flight
safety and asset protection. The FBS weather sensors and data automation
components will be integrated by the contractor into a fully automated
weather observation system that will interface with a number of local
communications and weather forecast systems at each site.

The evaluation factors listed in the RFP were mission capability
(technical), proposal risk, past performance and price. The RFP further
provided that, when combined, the technical and past performance factors
were more important than price. The agency warned that the contract might be
awarded based on a higher-priced offer which offered technical superiority,
if the price differential was determined to be worth the technical merit
offered over a lower-priced offer. RFP at 41-42.

The agency received initial proposals in response to the RFP from Coastal,
Qualimetrics, SMI, and Northern NEF, Inc. (NNEF). After written discussions,
the agency awarded the contract to Coastal.

Both Qualimetrics and SMI protested the award to Coastal to our Office. In
Systems Mgmt., Inc.; Qualimetrics, Inc., B-287032.3, B-287032.4, Apr. 16,
2001, 2001 CPD para. 85, our Office sustained these protests, finding that the
agency had overstated its minimum needs in requiring that the proposed
systems be FAA-certified, and either waived or relaxed this requirement when
it made award to Coastal, whose proposed system was not certified. We also
noted that the evaluation scheme for past performance apparently inhibited a
qualitative evaluation of past performance because the agency had an
internal $60 million threshold before the agency would determine whether a
contract was considered "very relevant," no matter how otherwise relevant
the past performance. Id. at 11 n.18. We recommended that the Air Force
amend the RFP to represent its actual needs concerning certification and
resolicit.

In response to our decision, the agency amended the solicitation to remove
the FAA certification requirement and to amend the past performance
evaluation criteria. The amendment and a request for proposal revisions were
transmitted to the four offerors who had originally responded to the
solicitation, all of which then submitted revised proposals.

As was the case in the previous evaluation, the proposals were evaluated
using a color/adjectival system with extra credit to be given to proposals
that exceeded the RFP's threshold requirements. [2] Proposal risk was
evaluated to assess the risks associated with the offeror's proposed
approach, as well as the likelihood of unsuccessful contract performance.
[3] RFP at 42. Past performance was reviewed and each proposal was assigned
a past performance rating. [4]

The findings of the source evaluation board (SEB) are summarized as follows,
showing the color rating and proposal risk assessment for the two subfactors
under the mission capability factor, as well as the past performance
assessments and evaluated prices:

                 SMI            Qualimetrics    Coastal        NNEF

 Subfactor 1.1   Green/         Green/          Blue/          [DELETED]

 Technical       Low Risk       Low Risk        Low Risk

 Compliance

 Subfactor 1.2   Green/         Green/          Blue/          [DELETED]

 Interim         Low Risk       Low Risk        Low Risk
 Contractor

 Support

 Past            Significant    Significant     Significant    [DELETED]
                                                Confidence
 Performance     Confidence     Confidence

 Evaluated       $60.8 Million  $53.2 Million   $59.2 Million  [DELETED]
 Price

Agency Report, Tab 7.7, Proposal Analysis Report (June 22, 2001), at 19.

Based on the evaluation, the agency determined to make award to NNEF. A
small-business size challenge was filed with the Small Business
Administration, which resulted in NNEF being disqualified when it was
determined to be other than a small business concern. The agency then
determined, in another detailed source selection document, that, of the
remaining offerors, Coastal's highest-rated proposal offered strengths that
offset the price premium, and it made award to that firm. Qualimetrics and
SMI then filed these protests, contesting the evaluation of past performance
and the technical merit of the proposals.

PAST PERFORMANCE

With regard to past performance, the protesters primarily contest the
revised evaluation criteria in the solicitation used for determining whether
a contractual effort was "very relevant," "relevant," or "somewhat
relevant," and contend that this rendered the past performance evaluation
irrational. This argument is untimely made under our Bid Protest Regulations
because this alleged solicitation impropriety, which was apparent from the
revised solicitation, was required to be protested prior to the May 21, 2001
closing date for receipt of revised proposals, but was not protested to our
Office until August 15. [5] 4 C.F.R. sect. 21.2(a)(1)(2001).

The protesters also argue that the methodology used by the agency to
evaluate past performance was improper. For example, the protesters claim
that certain technical proposal evaluation results should have been, but
were not, considered in order to properly assess performance risk, and,
thus, no effort was made to evaluate past performance in light of the
contemplated effort here.

However, under the RFP evaluation scheme, an offeror's past performance was
evaluated as performance risk. RFP at 42-43. This assessment was different
from the consideration of proposal risk, which considered the risk
associated with the offeror's proposed approach as laid out in its proposal.
See Modern Techs. Corp. et al., B-278695 et al., Mar. 4, 1998, 98-1 CPD para. 81
at 7. Thus, in evaluating performance risk, the agency appropriately did not
consider matters relating to proposal risk or the technical evaluation.

The protesters also contend that the offerors' past performance was
incompletely, and thus improperly, evaluated. This is so, according to the
protesters, because the past performance of Coastal's subcontractor, who
developed software that is incorporated in Coastal's system was not
considered. Also, SMI's performance under a subcontract with NNEF was not
considered.

Where a solicitation contemplates the evaluation of offerors' past
performance, the agency has the discretion to determine the scope of the
offeror's performance history to be considered, provided all proposals are
evaluated on the same basis and the evaluation is consistent with the terms
of the RFP. USATREX Int'l, Inc., B-275592, B-275592.2, Mar. 6, 1997, 98-l
CPD para. 99 at 3. While agencies may consider the prior relevant experience of
subcontractors, in the absence of a solicitation provision to the contrary,
there is no broad requirement that they do so. North State Res., Inc.,
B-282140, June 7, 1999, 99-2 CPD para. 60 at 7. Nor is there a requirement in
all circumstances that an agency obtain or consider all of an offeror's
references in the past performance evaluation. Kellie W. Tipton Constr. Co.,
B-281331.3, Mar. 22, 1999, 99-1 CPD para. 73 at 6-7.

Here, there was no solicitation provision that obligated the agency to
consider the past performance of all proposed subcontractors. While the
record reflects that the agency did evaluate the past performance of one of
Coastal's subcontractors (as well as some of other offerors'
subcontractors), we find no basis to conclude that the agency abused its
discretion in failing to evaluate the past performance of one particular
subcontractor of Coastal.

The agency declined to consider a contract reference in SMI's proposal where
NNEF was the prime contractor and SMI was a subcontractor because NNEF was
an offeror in the present competition. The agency believed that requesting
past performance information obtained from NNEF regarding this contract
would have forced the agency to disclose the identity of SMI, a competing
offeror, and that the reference may have a possible conflict of interest or
bias towards the OS-21 FBS competitors (even after NNEF was out of the
competition). Agency Report at 10. There is also no solicitation provision
that obligated the agency to consider all of an offeror's submitted past
performance references. In any event, the agency asserts that SMI's
confidence rating would not have increased to high confidence, even if this
contract had been evaluated, because the Air Force did not consider this
contract "very relevant," given its dollar value. Agency Report at 11. We
have no basis to question the reasonableness of the agency's evaluation in
this regard, particularly since the agency has the discretion to determine
the scope of an offeror's past performance history.

Finally, the protesters assert that the past performance evaluation was
unreasonable because SMI has performed contracts totaling more than
Coastal's, yet both received the same "significant confidence" performance
risk rating. However, under the evaluation scheme in the RFP, the total
number or value of contracts should not drive the performance risk rating,
which assessed various aspects of the relevant individual past performance
references, such as relevance, technical compliance, logistics and
management support, currency, and type of contractual arrangement. From our
review of the record, the agency reasonably rated the offerors' past
performance in accordance with the RFP evaluation criteria.

TECHNICAL EVALUATION

The protesters challenge each of the strengths identified in Coastal's
mission capability proposal that led to its blue/low proposal risk rating,
most of which were referenced in the source selection document justifying
the selection of Coastal's proposal. The evaluation of technical proposals
is primarily the responsibility of the contracting agency. Our Office will
not make an independent determination of the merits of technical proposals;
rather we will examine the record to ensure that the agency's evaluation was
reasonable and consistent with the stated evaluation criteria. Litton Sys.,
Inc., B-237596.3, Aug. 8, 1990, 90-2 CPD para. 115 at 8. A protester's mere
disagreement with the agency's evaluation does not render the evaluation
unreasonable. SWR Inc., B-286044.2, B-286044.3, Nov. 1, 2000, 2000 CPD para. 174
at 3. As discussed below, we find the agency evaluation of the strengths in
Coastal's mission capability proposal to be reasonable.

Graphical User Interface (GUI)

The agency noted as a strength in Coastal's proposal that its proposed GUI
(called Airport Weather Advisory (AWA)) on the OS-21 FBS is tailorable.
Agency Report, Tab 7.7, Proposal Analysis Report, at 8; Tab 7.10, Source
Selection Document, at 3. The protesters contend that the agency had no
basis for its conclusion that Coastal's GUI system is more tailorable than
their proposed GUI software system (called StormFront).

However, in contrast to the protesters' proposals, Coastal's proposal was
replete with details concerning the tailorability of its user interface. For
example, Coastal describes its AWA as follows:

[DELETED]

Agency Report, Tab 9.1, Coastal's Proposal, vol. 2, at 3.4.2.2. In addition,
Coastal's proposal notes that the key to AWA's easy-to-use design is
[DELETED]:

[DELETED]

[DELETED] Another significant tailorability feature of Coastal's system is
the ability to add sensors through the GUI, rather than, as required by the
Qualimetrics system, through software source code reconfiguration. Id. at
3.4.2.2.

In contrast, the proposal of Qualimetrics addresses GUI tailorability in the
following section:

Qualimetrics, working with SMI, has developed a software system called
StormFront that enables the easy configuration of custom GUI screens for any
and all applications. Using StormFront, the user can define and build his
own display screens.

Agency Report, Tab 8.3, Qualimetrics' Revised Proposal, at 38. SMI's
proposal simply asserts the tailorability of the GUI with no significant
details. See Agency Report, SMI Proposal, vol. 2, sect. 2.8.

The tailorability of the GUI was clearly a feature that the agency greatly
valued. The agency notes that the ability of the operators to reconfigure
the data displays to best meet their needs is of significant importance
since multiple configurations of the display may be necessary at different
Air Force and Army sites due to varying complements of sensors and varying
missions. Agency Report, Tab 7.7, Proposal Analysis Report, at 8. It is
apparent that Coastal's proposal emphasized and described the tailorability
of its system in much greater detail than the proposals of Qualimetrics and
SMI. In negotiated procurements, since the agency's technical evaluation of
proposal quality generally is based upon information submitted with the
proposal, the burden is on the offeror to submit an adequately written
proposal. See Will-Burt Co., B-250626.2, Jan. 25, 1993, 93-1 CPD para. 61 at 3.
Under the circumstances, while the protesters disagree with the evaluators'
conclusions that Coastal's GUI was more tailorable than their StormFront
GUI, the protesters have not provided any basis to show that the agency
evaluation was unreasonable or inconsistent with the solicitation's
evaluation criteria.

Use of C++

The protesters contest the agency's determination that the fact that
Coastal's GUI was written in the native language of Windows, C++, which
allows all of the display tab windows to update simultaneously, thereby
preventing any delay or latency of data when the operator selects a
different display window, represented a strength in Coastal's proposal.
Agency Report, Tab 7.7, Proposal Analysis Report, at 8; Tab 7.10, Source
Selection Document, at 3. The protesters argue that their proposed systems,
which both used C++ in the programming, possess this same capability.

Coastal's proposal states [DELETED]. Agency Report, Tab 9.1, Coastal's
Revised Proposal, at 12. The agency found, and our review confirms, that the
protesters' proposals did not articulate their use of C++ in the
implementation of the StormFront software system, but, more importantly,
their proposals did not state that their systems provided the same
capability of [DELETED]. Contracting Officer's Statement at 10. Given that
Qualimetrics' and SMI's proposals do not provide the same level of detail
that was provided in Coastal's proposal in this respect, we cannot find this
evaluation to be unreasonable or inconsistent with the evaluation criteria.

Use of Windows 2000

The protesters disagree with the agency's conclusion that Coastal's use of
Windows 2000, rather than Windows NT, in its proposed system provided
enhanced security features.

While Qualimetrics' and SMI's proposed systems, which contain the protection
features of Windows NT, met the agency's minimum requirements, Coastal's
system using Windows 2000 was found by the agency to provide significant
evidence of greater security capabilities in the areas of data storage,
password protection, multiserver security, database security, and group
assignments. Contracting Officer's Statement at 11; Agency Report, Tab 9.1,
Coastal's Revised Proposal, at 12-17. The source selection decision stated
that the security features provided by Coastal's system will greatly
facilitate security accreditation and net-worthiness certification, which
will lead to timely major command coordination and site-level system
acceptance. Agency Report, Tab 7.10, Source Selection Decision, at 3.

The protesters contend that the security advantages of Windows 2000 are
disputed and list several sources on the Internet to prove their point.
Protest at 19. While these sources indicate that the security advantages of
Windows 2000 versus Windows NT are a matter of dispute, these sources also
provide information that supports the agency position in this regard. Under
the circumstances, we will not question the agency's technical judgment on
this point. Notwithstanding the protesters' assertion that the agency did
not adequately quantify or discuss the relevant security advantages in
Coastal's proposal, they have not shown that the agency's evaluation was
unreasonable in this respect.

Unique Interfaces

The protesters also contest the strength assigned Coastal's proposal with
regard to its system's unique interfaces. The agency found that Coastal's
proposal highlighted the extent of the firm's open architecture from a
hardware and software perspective, which allowed the system to be adaptable,
flexible and expandable. This was primarily because of Coastal's unique
interfaces, which permit the easy addition of new sensors. Specifically,
Coastal's Universal Sensor Interfaces, the Universal Digital Interface, and
the Universal Analog Interface, and its hardware/software interface design,
coupled with its GUI, were found to provide a more user-friendly, flexible
and scalable system for sensor reconfiguration, modification and addition.
According to the agency, because of these features, future system upgrades
and evolving interface requirements will be more easily implemented with
respect to cost and schedule. Contracting Officer's Statement at 11; Agency
Report, Tab 7.10, Source Selection Document, at 3.

In contrast, the protesters' open architecture description in this respect
addresses only the hardware perspective. Contracting Officer's Statement at
11. Regarding the addition of new sensors, both Qualimetrics and SMI
indicated that updates to the software that runs their systems may be
required to meet the data archiving and data dissemination requirements of
the new sensors. Agency Report, Tab 6.2, SMI's Evaluation Notices, at 11;
Tab 6.3, Qualimetrics' Evaluation Notices, at 5.

The protesters assert that the agency is mistaken if it believes that
software changes are not necessary to add new sensors under Coastal's
system. However, even the protesters acknowledge that if such software
changes are necessary under Coastal's system, "the means under which they
are done may be minimized." Protesters' Comments at 19. For example,
Coastal's system [DELETED]. Based on our review, the agency reasonably
determined that Coastal's unique interfaces represented a strength in its
proposal.

Interim Contractor Support (ICS)

Another strength found in Coastal's proposal was its inclusion of a sample
integrated support plan, which illustrated a superior, "proactive" approach
to the ICS requirements. Agency Report, Tab 7.7, Proposal Analysis Report,
at 14. The agency states that "the sample ICS plan demonstrated an
outstanding approach in providing ICS support that went well beyond the
expectations of the ICS requirements." Contracting Officer's Statement at
12. Although the protesters' proposals did not include a sample plan, the
protesters contend that this strength is a "complete mystery," because the
protesters both have more experience in meeting ICS requirements than
Coastal can demonstrate. Protest at 25. Based upon a review of the contents
of the proposals, however, we see no mystery in Coastal's being awarded a
strength in this area.

Warranty

The protesters dispute that Coastal deserves extra credit for providing
additional warranties. The agency gave Coastal's proposal a strength in this
area because it offered a 3-year warranty on workmanship (parts and labor)
on certain system components, as compared to the 1-year warranty offered by
the protesters. Contracting Officer's Statement at 12; Agency Report, Tab
6.5, Coastal's Responses to Evaluation Notices, at 2; Tab 7.7, Proposal
Analysis Report, at 14. We see no reasonable basis to dispute that this is a
relative strength in Coastal's proposal. The protesters nevertheless argue
that the agency failed to exactly quantify the advantages that this
additional warranty provides. However, the agency was not required to
quantify the value of each strength.

ISO 9002

The protesters assert that Coastal's proposal should not have been given a
strength for being ISO 9002 certified, given that the protesters are both
ISO 9001 certified and ISO 9001 is a more stringent qualification because it
also includes system design, and because one of Coastal's subcontractors is
not ISO certified. The protesters' ISO 9001 certifications were reported as
a strength by the agency. The agency reports that "[b]oth [9001 and 9002]
certifications demonstrated to the technical evaluation team that each of
these companies had progressed to the point that they were organizationally
equipped to assume the responsibilities of the OS-21 program." The agency
asserts that the added dimension of design in the ISO 9001 certification,
which is not part of the ISO 9002 certification, is not of significant
importance in this procurement since the OS-21 FSB program is a FAR Part 12
commercial acquisition, and is not a developmental effort. Contracting
Officer's Statement at 12. The protesters' assertion that their ISO 9001
certifications should have been more highly regarded, because they believe
some developmental effort is involved in performing this contract,
constitutes mere disagreement with the agency's evaluation and does not
demonstrate that awarding both the awardee's and the protesters' proposals a
strength in this area was unreasonable. Moreover, the fact that one of
Coastal's subcontractors is not ISO certified does not negate Coastal's
certification.

Other Strengths

The protesters also contest the strengths evaluated in Coastal's proposal
for "plug and play" capability and remote real-time display. They assert
that these capabilities are merely features of the Windows operating system
and that their Windows-based proposals should also have received similar
strengths. The agency responds that these strengths are sub-elements of the
overall strength in Coastal's proposal of its secure, flexible open
architecture, and these strengths were not simply based on Coastal's
Windows-based operating system, but also considered its specific and unique
sensor interfaces and hardware/software design, coupled with its
user-friendly secure user-interface. The protesters have not substantively
responded to the agency's reasonable explanation.

The protesters also challenge the strength in Coastal's proposal for having
multiple simultaneous language outputs, asserting that this is inconsistent
with the solicitation. The agency denies this allegation, and responds that
as a result of the debriefings that the protesters received in association
with the previous award selection of Coastal, they were aware that the
agency regarded this aspect of Coastal's proposal as a strength, yet they
did not include this attribute in their revised proposals. Here too, the
protesters have not responded to the agency's explanation.

COST/TECHNICAL TRADEOFF & SOURCE SELECTION

The protesters contend that the agency failed to conduct a rational
cost/technical tradeoff. To the extent that this argument depends on the
challenges to the Air Force's technical and past performance evaluation, it
fails, since, as discussed above, we find no basis to question the
reasonableness of the agency's evaluation. To the extent that the protester
believes that the agency was required to quantify the dollar value of
Coastal's exceptional rating, the protest is without merit. There is no
requirement that an agency quantify the value of technical superiority
vis-a-vis low cost/price to determine the best value to the government. See
DDD Co., B-276708, July 16, 1997, 97-2 CPD para. 44 at 8. On the record here,
the agency has adequately justified its cost/technical tradeoff and award
decision in finding the particular strengths in Coastal's proposal were
worth the associated price premium. [6]

The protests are denied.

Anthony H. Gamboa

General Counsel

Notes

1. SMI and Qualimetrics are affiliated companies.

2. The agency rated proposals with color/adjectival ratings of
blue/exceptional, green/acceptable, yellow/marginal, and red/unacceptable.

3. The proposal risk ratings assigned were high, moderate, or low.

4. The past performance ratings assigned were high confidence, significant
confidence, confidence, unknown confidence, little confidence, and no
confidence.

5. The protester references a May 3, 2001 e-mail addressed to agency counsel
raising concerns about the past performance evaluation criteria, Agency
Comments (Oct. 2, 2001), attach. 1, and suggests that this correspondence
constituted an agency-level protest. Protester's Comments at 3. The agency
responded to the protesters' e-mail on May 4, advising that this inquiry
should be directed to the contracting office. Agency's Comments (Oct. 2,
2001), attach. 1. Since the protesters did not pursue this matter further at
that time, we do not consider this communication to be an agency-level
protest.

6. Qualimetrics and SMI also protest the agency's affirmative determination
of Coastal's responsibility, asserting that Coastal is ineligible to receive
the award due to its alleged "meager financial status." Protest at 31. Our
Office does not review affirmative determinations of responsibility absent a
showing of possible bad faith on the part of government officials or that
definitive responsibility criteria in the solicitation were not met. 4
C.F.R. sect. 21.5(c). Since the protester does not allege bad faith by the
agency or that definitive responsibility requirements were not met by
Coastal, the matter is not for our review. M-Cubed Info. Sys., Inc.,
B-284445, B-284445.2, Apr. 19, 2000, 2000 CPD para. 74 at 9-10.