TITLE: B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5, IMLCORP LLC; Wattre Corporation, January 9, 2008
BNUMBER: B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5
DATE: January 9, 2008
**********************************************************************************************************
B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5, IMLCORP LLC; Wattre Corporation, January 9, 2008
Decision
Matter of: IMLCORP LLC; Wattre Corporation
File: B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5
Date: January 9, 2008
John H. Horne, Esq., Carl A. Gebo, Esq., and Vickie S. Carlton, Esq.,
Powell Goldstein LLP, for IMLCORP LLC; and A.J. Ballard for Wattre
Corporation, the protesters.
Angela J. Cosentino, Esq., and Marvin D. Rampey, Esq., Department of the
Navy, for the agency.
Guy R. Pietrovito, Esq., and James A. Spangenberg, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision.
DIGEST
Protests challenging the testing methodology employed by an agency in
product sample testing is denied, where the solicitation did not specify a
particular testing methodology and the protesters have failed to show that
the agency's test was an unreasonable method of assessing how well the
firms' products would meet the agency's needs.
DECISION
IMLCORP LLC and Wattre Corporation protest the award of a contract to
American Technology Corporation (ATC) under request for proposals (RFP)
No. N00164--07-R-8555, issued by the Department of the Navy, for acoustic
hailing devices.
We deny the protests.
The RFP was issued as a commercial item acquisition in accordance with
Federal Acquisition Regulation part 12.6, and provided for the award of a
fixed-price contract for acoustic hailing device systems. This device is a
"rugged and lightweight loudspeaker system with very high directivity that
is intended for long-range hailing and warning" and will be used on Navy
vessels to warn other vessels that they are entering the Navy's 500-yard
exclusion zone. RFP, Performance Specification, at 3. Operational and
reliability requirements were identified in the performance specification,
including that the acoustic hailing device must have "100% voice
intelligibility at 500 yards with 88 dB [decibels] of ambient background
noise.[1] Id. at 7-8.
The RFP provided for award on the basis of a cost/technical tradeoff and
identified the following evaluation factors and subfactors, each listed in
descending order of importance:
+------------------------------------------------------------------------+
|Capability |
|------------------------------------------------------------------------|
| |Product sample evaluation[2] |
| |-------------------------------------------------------------|
| |Written performance specification compliance |
| |-------------------------------------------------------------|
| |Delivery/production schedule/warranty repair turn around time|
| |-------------------------------------------------------------|
| |Warranty |
|------------------------------------------------------------------------|
|Past Performance |
|------------------------------------------------------------------------|
| |Quality |
| |-------------------------------------------------------------|
| |Customer satisfaction |
| |-------------------------------------------------------------|
| |Subcontracting |
|------------------------------------------------------------------------|
|Price |
+------------------------------------------------------------------------+
RFP amend. 1, at 2-3.
The RFP required the submission with proposals of a product sample and
"any optional accessories or upgrades that provide claimed improvements in
functionality, performance, or additional capabilities." RFP at 38. In
this regard, offerors were informed:
The product sample shall be tested in a non-destructive manner by
Government personnel. Product sample testing is planned to occur in a
laboratory and field environment at [the Naval Surface Weapons Center in
Crane, Indiana]. The Government may test the product sample against the
technical requirements set forth in the Performance Specification
contained in this solicitation.
RFP at 38.
With respect to the past performance factor, the RFP required offerors to
identify up to five contracts, which the offeror had performed, or was
performing, within the last 3 years for the same or similar products, and
complete a "contractor performance data sheet" for each. RFP at 39, 41.
Offerors were informed that the past performance quality subfactor would
be assessed using the information provided on the contractor performance
data sheet.
The proposals received from ATC, IMLCORP, and Wattre were evaluated as
follows:
+------------------------------------------------------------------------+
| | ATC | IMLCORP | Wattre |
|--------------------+------------------+------------------+-------------|
|Capability | Highly | Satisfactory/Low |Marginal/High|
| | Satisfactory/Low | Risk | Risk |
| | Risk | | |
|--------------------+------------------+------------------+-------------|
|Product Sample | Highly | Satisfactory |Unacceptable |
| | Satisfactory | | |
|--------------------+------------------+------------------+-------------|
|Specification | Highly | Satisfactory |Satisfactory |
|Compliance | Satisfactory | | |
|--------------------+------------------+------------------+-------------|
|Delivery/Prod. | Satisfactory | Satisfactory |Satisfactory |
|Schedule/Warranty | | | |
|Turn Around | | | |
|--------------------+------------------+------------------+-------------|
|Warranty | Satisfactory | Satisfactory |Satisfactory |
|--------------------+------------------+------------------+-------------|
|Past Performance |Favorable/Low Risk|Favorable/Low Risk|Neutral /High|
| | | | Risk |
|--------------------+------------------+------------------+-------------|
|Quality | Favorable | Favorable | Neutral |
|--------------------+------------------+------------------+-------------|
|Customer | Favorable | Favorable | Neutral |
|Satisfaction | | | |
|--------------------+------------------+------------------+-------------|
|Subcontracting | Favorable | Favorable | Neutral |
|--------------------+------------------+------------------+-------------|
|OVERALL TECHNICAL | Highly | Satisfactory/Low |Marginal/High|
|RATING | Satisfactory/ | Risk | Risk |
| | | | |
| | Low Risk | | |
|--------------------+------------------+------------------+-------------|
|PRICE | $4,988,989 | $6,313,955 | $5,561,489 |
+------------------------------------------------------------------------+
Agency Report (AR), Tab 16, Source Selection Evaluation Board (SSEB)
Consensus Report, at 4-5, 7-8; Tab 17, Pre-Negotiation Business Clearance
Memorandum, at 4.[3]
ATC's higher overall technical rating primarily reflected the firm's
highly satisfactory ratings under the two most important
subfactors--product sample and specification compliance--under the most
important evaluation factor--capability. IMLCORP's and Wattre's
satisfactory and unacceptable overall technical ratings, respectively,
primarily reflect the ratings assigned those firms' product samples.
Product sample testing was conducted in accordance with the agency's
"Shout-Off Test Plan."[4] AR, Tab 3, Shout-Off Test Plan. That test
consisted of placing the firms' acoustic hailing devices side-by-side on
the edge of a lake, and projecting pre-recorded voice and sound messages
over the lake to two small outboard-motor powered boats each containing
three evaluators and each located 500 yards from the hailing devices. The
pre-recorded messages used in this testing were the messages provided by
the offerors with their product samples, and in conducting the testing the
Navy used the offeror's own messages to test that particular firm's
device. Each device was the subject of six tests, three with the boats
stationary and three with the boats moving. The evaluators assigned a
numerical score from 1 to 10 for voice intelligibility with 10 being the
most intelligible for each of the six tests. In calculating the firm's
intelligibility percentage, the Navy took the highest ratings assigned by
the evaluators for one of the six tests, and averaged those ratings.
IMLCORP AR at 9-10. Based upon this methodology, ATC's product sample
received a 100-percent voice intelligibility rating over the 500-yard
distance, IMLCORP's product sample received a 91.66-percent rating, and
Wattre's received a 66.6-percent rating. AR, Tab 16, SSEB Consensus
Evaluation Report, apps. A, B, and C (Product Sample Ratings).
Award was made to ATC, based upon that firm's higher technical rating and
lowest evaluated price. These protests followed, which raise numerous
challenges of the Navy's evaluation of the firms' product samples and the
evaluation under the other evaluation factors.[5]
The protesters first complain that the Navy's evaluators were not trained,
experienced, "operational military personnel" and that the hearing of
these evaluators was not tested before the product sample tests were
conducted. IMLCORP Supp. Protest at 2; Wattre Protest at 8. The Navy
responds that the solicitation did not require the use of operational
military personnel or identify any other experience requirements for these
evaluators and that, in any event, the SSEB chair is an electrical
engineer with significant experience with acoustic hailing devices. The
Navy also asserts that there was no requirement that the evaluators'
hearing be tested prior to the product sample evaluation and that the
evaluators reflected the average hearing that would be expected of actual
users of the shipboard devices. See, e.g., Wattre AR at 10.
We find that the protesters' speculative challenges to the qualifications
of the Navy's evaluators provide us with no basis to question the agency's
product sample evaluation. See Philadelphia Produce Mkt. Wholesalers, LLC,
B-298751.5, May 1, 2007, 2007 CPD para. 87 at 5 n.2. Moreover, we have
long found that the selection of individuals to serve as evaluators is a
matter within the discretion of the agency, and, accordingly, we do not
review allegations, such as these, concerning the evaluators'
qualifications or the composition of evaluation panels absent a showing of
possible fraud, conflict of interest, or actual bias on the part of
evaluation officials, none of which have been alleged or shown here. See
Eggs & Bacon, Inc., B-310066, Nov. 20, 2007, 2007 CPD para. 209 at 4;
Glatz Aeronautical Corp., B-293968.2, Aug. 10, 2004, 2004 CPD para. 160 at
3 n.1.
IMLCORP and Wattre also object that the Navy's shout-off test was not an
objective, scientific test of the firms' acoustic hailing devices.
Specifically, the protesters argue that numerous entities, such as
American National Standards Institute, the Department of Defense's Joint
Non-Lethal Weapons Program, and the Applied Research Laboratory of
Pennsylvania State University, have established standardized tests for the
evaluation of acoustic hailing devices, which the Navy should have, but
did not, use here. In this regard, IMLCORP has provided the statement of a
professor of mechanical engineering, who contends that the Navy's use of a
subjective scale for assessing voice intelligibility was not an objective
test of voice intelligibility and therefore was unreasonable. IMCORP Supp.
Comments, attach. E, Affidavit of Professor of Mechanical Engineering, at
3.
The Navy disagrees that there exists an industry standard test for
assessing these devices and notes, in this regard, that the RFP did not
identify or require any particular testing methodology. See IMLCORP Supp.
AR at 4; Wattre AR at 3-4. The Navy recognized that there are a variety of
other tests that could have been employed,[6] and states that the agency's
shout-off test used elements similar to those employed by the Applied
Research Laboratory of Pennsylvania State University as modified to more
closely apply to the agency's requirements (for example, testing over
water instead of in a building) and to account for the testing resources
that the agency had available.[7] See IMLCORP Supp. AR at 3-4. The Navy
also contends that neither protester has shown that the use of a different
testing methodology would have resulted in a different evaluation
conclusion; in other words, the agency contends that neither protester has
provided any data or test results that show that its acoustic hailing
device would have been assessed as being superior to that offered by ATC,
even if a different testing methodology were used.
It is well established that it is a procuring agency's responsibility to
determine its needs and how best to meet them. See CardioMetrix, B-270701,
Mar. 13, 1996, 96-1 CPD para. 149 at 3. This responsibility includes
determining the amount and type of testing necessary to evaluate a
product's suitability. Terex Corp.; Caterpillar Tractor Co., B-217053;
B-218535, July 24, 1985, 85-2 CPD para. 76 at 6. Our review of an agency's
evaluation tests, as with other agency evaluation judgments, does not
include conducting our own testing or reevaluating proposals; rather, our
review is limited to examining the record to determine whether the
agency's judgment was reasonable and in accord with the stated evaluation
criteria and applicable procurement laws and regulations. See Abt Assocs.,
Inc., B-237060.2, Feb. 26, 1990, 90-1 CPD para. 223 at 4.
We find no basis in the record to disagree with the agency's view that,
although there are a variety of tests available to assess voice
intelligibility of devices such as the acoustic hailing devices, there is
not an standard industry test, which offerors could reasonably expect
would be used in this procurement. Although the protesters' believe that
some other test would be more reasonable, their disagreement alone does
not establish that the shout-off test used by the agency was an
unreasonable method of assessing how well the firms' products would meet
the agency's needs.
The protesters nevertheless argue that the way the Navy conducted the
shout-off test resulted in an unequal evaluation and that the test was
also not conducted in accordance with the solicitation's specifications.
Specifically, the protesters object that the Navy's use of the firms' own
pre-recorded messages was unreasonable because the firms' product samples
were not all assessed using a standard message and that the Navy used
pre-recorded voices rather than live voices in the test.
The Navy responds that the firms were permitted to submit, and in fact did
submit, their own messages, which the agency presumed were chosen to
optimize the performance of that firm's acoustic hailing device. Moreover,
the Navy states that using
multiple sound files reduces the likelihood that a listener will become
familiar with the phrases and begin anticipating words in the phrase in
later test runs. Utilizing the same phrase repeatedly can cause the
listeners to anticipate the words in the phrase and distort the results
for later test runs.
IMLCORP Supp. Report at 5. The agency also contends that the use of a
pre-recorded message, rather than broadcasting a live message, more
accurately reflected the agency's needs, given that the "primary use of
the [acoustic hailing devices] will be in danger zones outside U.S.
territorial waters" and therefore these "messages are particularly useful
due to the language and dialect variations where U.S. Naval vessels may
anchor." Id. In other words, the agency reasonably anticipates that its
actual use of the acoustic hailing devices would include the transmission
of pre-recorded messages.
We do not agree with the protesters that they were treated unreasonably in
the shout-off test, even though the agency did not use the same message,
or set of messages, to assess the voice intelligibility of the firms'
devices in the shout-off tests and only used recorded messages. The
protesters have not shown the agency's explanation for doing so to be
unreasonable.[8] We also find that the Navy otherwise equally tested the
firms' devices, given that these products were evaluated at the same time
and place, operated by the same individuals, and evaluated by the same
group of evaluators.
The protesters also complain that Navy did not conduct the shout-off test
in accordance with the RFP's specification requirements, given that the
test was conducted with a background noise between 58 to 82 decibels, but
the solicitation's performance specification provided for 100-percent
voice intelligibility at 500 yards with a background noise of 88
decibels.[9] While the Navy did not assess the voice intelligibility of
the product samples against the background noise level identified by the
RFP, the protesters have not shown that they were prejudiced in this
regard, given that they have not explained how their voice intelligibility
rating would improve and surpass that of ATC, if a higher background noise
were employed.[10] The record also shows that the Navy assessed the
offerors' compliance with this specification requirement under its
evaluation of the written technical proposals under the specification
compliance subfactor, and ATC's proposal demonstrated compliance with the
requirement for 100-percent voice intelligibility at 500 yards with a
background noise of 88 decibels. See AR, Tab 10, ATC Technical Proposal,
at 22-23.
IMLCORP also protested the Navy's evaluation of the firm's proposed
acoustic hailing under a number of the solicitation's specification
requirements. We have carefully reviewed each of IMLCORP's complaints in
this regard and find them to be without merit. For example, IMLCORP
complains that the Navy assessed a weakness with respect to the weight of
IMLCORP's proposed product, but the record shows that no weakness was
assigned to IMLCORP's proposal for the weight of its proposed acoustic
hailing device. See AR, Tab 16, SSEB Consensus Report, at 8. As another
example, IMLCORP complains that the Navy assessed a weakness under the
"audio main lobe" specification because its proposed product exceeded the
30-degree maximum dispersion pattern required by the RFP; IMLCORP admits
however that its proposed device does exceed this requirement by 1 degree.
Although IMLCORP contends that this was a minor deviation, this does not
establish that the Navy could not reasonably assess a weakness for failing
to meet this specification requirement.
IMLCORP also challenges the Navy's evaluation of ATC's and IMLCORP's past
performance, arguing generally that the two firms should not have received
the same favorable, low risk rating under the past performance factor
because ATC's acoustic hailing device had received a lower performance
rating than IMLCORP's in "limited military utility assessment" (LMUA)
testing performed by Alion Science and Technology and the Applied Research
Laboratory of The Pennsylvania State University. See IMLCORP Comments,
attach. A. However, the RFP provided for an assessment of offerors' past
performance quality based upon an evaluation of the offerors' performance
of prior contracts for the same or similar products, whereas the LMUA
testing, to which IMLCORP refers concerns other entities' assessments of
the quality of the products themselves, and not the offerors' past
performance quality. We agree with the Navy that the RFP provided for the
quality assessment of the offerors' proposed acoustic hailing devices
under the capability evaluation factor, and that there was no requirement
that the Navy consider other entities' testing of the acoustic hailing
devices under this evaluation factor.
The protests are denied.[11]
Gary L. Kepplinger
General Counsel
------------------------
[1] The RFP does not define "voice intelligibility."
[2] The product sample evaluation subfactor was said to be significantly
more important that the other subfactors of the capability factor. RFP
amend. 1, at 2.
[3] The protests filed by IMLCORP and Wattre were not consolidated, and
the Navy submitted separate reports for these protests. The core
documents, such as the evaluation and source selection documents, were
identical in the reports but were identified by different tab numbers. In
this decision we refer to the agency report submitted in response to the
IMLCORP protests, except where it is necessary to cite to differing
documents in the agency report submitted in response to Wattre's protests.
[4] The Shout-Off Test Plan was not included in the solicitation or
otherwise provided to the offerors.
[5] IMLCORP and Wattre raised a number of other protest allegations in its
protests and supplemental protests that we dismissed as untimely during
our development of this case. For example, Wattre untimely complained that
the Navy had refused the firm's offer to provide training to the Navy's
evaluators for Wattre's acoustic hailing device.
[6] The Navy states that these other entities identified by the protesters
all used different testing methodologies to assess voice intelligibility
in prior studies. Wattre AR at 5.
[7] The Navy provided a statement from an associate professor of acoustics
at the Applied Research Laboratory of Pennsylvania State University, who
states that the Applied Research Laboratory had not established a standard
in its testing of the voice intelligibility of acoustic hailing devices.
IMLCORP Supp. AR, attach. 1, Statement of Associate Professor of
Acoustics. IMLCORP had earlier indicated that it would provide a statement
from this individual but ultimately did not. IMLCORP Comments at 4.
[8] While IMLCORP also argues that the use of recorded messages instead of
live voices was inconsistent with the solicitation's requirements, it has
not identified any solicitation provision requiring the use of live voices
in testing the offerors' acoustic hailing devices, nor have we found any
such requirement.
[9] Wattre also complains that the Navy's shout-off test did not assess
the acoustic hailing devices against every one of the RFP's specification
requirements. The RFP, however, informed offerors that the agency "may
test the product sample against the technical requirements set forth in
the Performance Specification" and did not require the agency to do so.
RFP at 38.
[10] Wattre also protested that the Navy may not have properly calibrated
its test equipment prior to the shout-off testing. The Navy responded that
the test equipment used in the shout-off test were two sound level meters,
which the Navy states it did not need to calibrate. Wattre Report at 23.
Because Wattre did not address the Navy's arguments in its comments, we
consider this protest allegation to be abandoned. See Planning Sys., Inc.,
B-292312, July 29, 2003, 2004 CPD para. 83 at 6.
[11] In their comments and supplemental comments, IMLCORP and Wattre
untimely raised a number of other challenges to the Navy's selection of
ATC's proposal for award. Our Bid Protest Regulations require that
protests based on other than alleged improprieties in a solicitation be
filed no later than 10 calendar days after the protester knew, or should
have known, of the basis for protest, whichever is earlier. 4 C.F.R.
sect. 21.2(a)(2) (2007). Because these numerous other protest allegations
were not timely filed, we have not considered them.