TITLE: B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5, IMLCORP LLC; Wattre Corporation, January 9, 2008
BNUMBER: B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5
DATE: January 9, 2008
**********************************************************************************************************
B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5, IMLCORP LLC; Wattre Corporation, January 9, 2008

   Decision

   Matter of: IMLCORP LLC; Wattre Corporation

   File: B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5

   Date: January 9, 2008

   John H. Horne, Esq., Carl A. Gebo, Esq., and Vickie S. Carlton, Esq.,
   Powell Goldstein LLP, for IMLCORP LLC; and A.J. Ballard for Wattre
   Corporation, the protesters.

   Angela J. Cosentino, Esq., and Marvin D. Rampey, Esq., Department of the
   Navy, for the agency.

   Guy R. Pietrovito, Esq., and James A. Spangenberg, Esq., Office of the
   General Counsel, GAO, participated in the preparation of the decision.

   DIGEST

   Protests challenging the testing methodology employed by an agency in
   product sample testing is denied, where the solicitation did not specify a
   particular testing methodology and the protesters have failed to show that
   the agency's test was an unreasonable method of assessing how well the
   firms' products would meet the agency's needs.

   DECISION

   IMLCORP LLC and Wattre Corporation protest the award of a contract to
   American Technology Corporation (ATC) under request for proposals (RFP)
   No. N00164--07-R-8555, issued by the Department of the Navy, for acoustic
   hailing devices.

   We deny the protests.

   The RFP was issued as a commercial item acquisition in accordance with
   Federal Acquisition Regulation part 12.6, and provided for the award of a
   fixed-price contract for acoustic hailing device systems. This device is a
   "rugged and lightweight loudspeaker system with very high directivity that
   is intended for long-range hailing and warning" and will be used on Navy
   vessels to warn other vessels that they are entering the Navy's 500-yard
   exclusion zone. RFP, Performance Specification, at 3. Operational and
   reliability requirements were identified in the performance specification,
   including that the acoustic hailing device must have "100% voice
   intelligibility at 500 yards with 88 dB [decibels] of ambient background
   noise.[1] Id. at 7-8.

   The RFP provided for award on the basis of a cost/technical tradeoff and
   identified the following evaluation factors and subfactors, each listed in
   descending order of importance:

   +------------------------------------------------------------------------+
   |Capability                                                              |
   |------------------------------------------------------------------------|
   |          |Product sample evaluation[2]                                 |
   |          |-------------------------------------------------------------|
   |          |Written performance specification compliance                 |
   |          |-------------------------------------------------------------|
   |          |Delivery/production schedule/warranty repair turn around time|
   |          |-------------------------------------------------------------|
   |          |Warranty                                                     |
   |------------------------------------------------------------------------|
   |Past Performance                                                        |
   |------------------------------------------------------------------------|
   |          |Quality                                                      |
   |          |-------------------------------------------------------------|
   |          |Customer satisfaction                                        |
   |          |-------------------------------------------------------------|
   |          |Subcontracting                                               |
   |------------------------------------------------------------------------|
   |Price                                                                   |
   +------------------------------------------------------------------------+

   RFP amend. 1, at 2-3.

   The RFP required the submission with proposals of a product sample and
   "any optional accessories or upgrades that provide claimed improvements in
   functionality, performance, or additional capabilities." RFP at 38. In
   this regard, offerors were informed:

     The product sample shall be tested in a non-destructive manner by
     Government personnel. Product sample testing is planned to occur in a
     laboratory and field environment at [the Naval Surface Weapons Center in
     Crane, Indiana]. The Government may test the product sample against the
     technical requirements set forth in the Performance Specification
     contained in this solicitation.

   RFP at 38.

   With respect to the past performance factor, the RFP required offerors to
   identify up to five contracts, which the offeror had performed, or was
   performing, within the last 3 years for the same or similar products, and
   complete a "contractor performance data sheet" for each. RFP at 39, 41.
   Offerors were informed that the past performance quality subfactor would
   be assessed using the information provided on the contractor performance
   data sheet.

   The proposals received from ATC, IMLCORP, and Wattre were evaluated as
   follows:

   +------------------------------------------------------------------------+
   |                    |       ATC        |     IMLCORP      |   Wattre    |
   |--------------------+------------------+------------------+-------------|
   |Capability          |      Highly      | Satisfactory/Low |Marginal/High|
   |                    | Satisfactory/Low |       Risk       |    Risk     |
   |                    |       Risk       |                  |             |
   |--------------------+------------------+------------------+-------------|
   |Product Sample      |      Highly      |   Satisfactory   |Unacceptable |
   |                    |   Satisfactory   |                  |             |
   |--------------------+------------------+------------------+-------------|
   |Specification       |      Highly      |   Satisfactory   |Satisfactory |
   |Compliance          |   Satisfactory   |                  |             |
   |--------------------+------------------+------------------+-------------|
   |Delivery/Prod.      |   Satisfactory   |   Satisfactory   |Satisfactory |
   |Schedule/Warranty   |                  |                  |             |
   |Turn Around         |                  |                  |             |
   |--------------------+------------------+------------------+-------------|
   |Warranty            |   Satisfactory   |   Satisfactory   |Satisfactory |
   |--------------------+------------------+------------------+-------------|
   |Past Performance    |Favorable/Low Risk|Favorable/Low Risk|Neutral /High|
   |                    |                  |                  |    Risk     |
   |--------------------+------------------+------------------+-------------|
   |Quality             |    Favorable     |    Favorable     |   Neutral   |
   |--------------------+------------------+------------------+-------------|
   |Customer            |    Favorable     |    Favorable     |   Neutral   |
   |Satisfaction        |                  |                  |             |
   |--------------------+------------------+------------------+-------------|
   |Subcontracting      |    Favorable     |    Favorable     |   Neutral   |
   |--------------------+------------------+------------------+-------------|
   |OVERALL TECHNICAL   |      Highly      | Satisfactory/Low |Marginal/High|
   |RATING              |  Satisfactory/   |       Risk       |    Risk     |
   |                    |                  |                  |             |
   |                    |     Low Risk     |                  |             |
   |--------------------+------------------+------------------+-------------|
   |PRICE               |    $4,988,989    |    $6,313,955    | $5,561,489  |
   +------------------------------------------------------------------------+

   Agency Report (AR), Tab 16, Source Selection Evaluation Board (SSEB)
   Consensus Report, at 4-5, 7-8; Tab 17, Pre-Negotiation Business Clearance
   Memorandum, at 4.[3]

   ATC's higher overall technical rating primarily reflected the firm's
   highly satisfactory ratings under the two most important
   subfactors--product sample and specification compliance--under the most
   important evaluation factor--capability. IMLCORP's and Wattre's
   satisfactory and unacceptable overall technical ratings, respectively,
   primarily reflect the ratings assigned those firms' product samples.

   Product sample testing was conducted in accordance with the agency's
   "Shout-Off Test Plan."[4] AR, Tab 3, Shout-Off Test Plan. That test
   consisted of placing the firms' acoustic hailing devices side-by-side on
   the edge of a lake, and projecting pre-recorded voice and sound messages
   over the lake to two small outboard-motor powered boats each containing
   three evaluators and each located 500 yards from the hailing devices. The
   pre-recorded messages used in this testing were the messages provided by
   the offerors with their product samples, and in conducting the testing the
   Navy used the offeror's own messages to test that particular firm's
   device. Each device was the subject of six tests, three with the boats
   stationary and three with the boats moving. The evaluators assigned a
   numerical score from 1 to 10 for voice intelligibility with 10 being the
   most intelligible for each of the six tests. In calculating the firm's
   intelligibility percentage, the Navy took the highest ratings assigned by
   the evaluators for one of the six tests, and averaged those ratings.
   IMLCORP AR at 9-10. Based upon this methodology, ATC's product sample
   received a 100-percent voice intelligibility rating over the 500-yard
   distance, IMLCORP's product sample received a 91.66-percent rating, and
   Wattre's received a 66.6-percent rating. AR, Tab 16, SSEB Consensus
   Evaluation Report, apps. A, B, and C (Product Sample Ratings).

   Award was made to ATC, based upon that firm's higher technical rating and
   lowest evaluated price. These protests followed, which raise numerous
   challenges of the Navy's evaluation of the firms' product samples and the
   evaluation under the other evaluation factors.[5]

   The protesters first complain that the Navy's evaluators were not trained,
   experienced, "operational military personnel" and that the hearing of
   these evaluators was not tested before the product sample tests were
   conducted. IMLCORP Supp. Protest at 2; Wattre Protest at 8. The Navy
   responds that the solicitation did not require the use of operational
   military personnel or identify any other experience requirements for these
   evaluators and that, in any event, the SSEB chair is an electrical
   engineer with significant experience with acoustic hailing devices. The
   Navy also asserts that there was no requirement that the evaluators'
   hearing be tested prior to the product sample evaluation and that the
   evaluators reflected the average hearing that would be expected of actual
   users of the shipboard devices. See, e.g., Wattre AR at 10.

   We find that the protesters' speculative challenges to the qualifications
   of the Navy's evaluators provide us with no basis to question the agency's
   product sample evaluation. See Philadelphia Produce Mkt. Wholesalers, LLC,
   B-298751.5, May 1, 2007, 2007 CPD para. 87 at 5 n.2. Moreover, we have
   long found that the selection of individuals to serve as evaluators is a
   matter within the discretion of the agency, and, accordingly, we do not
   review allegations, such as these, concerning the evaluators'
   qualifications or the composition of evaluation panels absent a showing of
   possible fraud, conflict of interest, or actual bias on the part of
   evaluation officials, none of which have been alleged or shown here. See
   Eggs & Bacon, Inc., B-310066, Nov. 20, 2007, 2007 CPD para. 209 at 4;
   Glatz Aeronautical Corp., B-293968.2, Aug. 10, 2004, 2004 CPD para. 160 at
   3 n.1.

   IMLCORP and Wattre also object that the Navy's shout-off test was not an
   objective, scientific test of the firms' acoustic hailing devices.
   Specifically, the protesters argue that numerous entities, such as
   American National Standards Institute, the Department of Defense's Joint
   Non-Lethal Weapons Program, and the Applied Research Laboratory of
   Pennsylvania State University, have established standardized tests for the
   evaluation of acoustic hailing devices, which the Navy should have, but
   did not, use here. In this regard, IMLCORP has provided the statement of a
   professor of mechanical engineering, who contends that the Navy's use of a
   subjective scale for assessing voice intelligibility was not an objective
   test of voice intelligibility and therefore was unreasonable. IMCORP Supp.
   Comments, attach. E, Affidavit of Professor of Mechanical Engineering, at
   3.

   The Navy disagrees that there exists an industry standard test for
   assessing these devices and notes, in this regard, that the RFP did not
   identify or require any particular testing methodology. See IMLCORP Supp.
   AR at 4; Wattre AR at 3-4. The Navy recognized that there are a variety of
   other tests that could have been employed,[6] and states that the agency's
   shout-off test used elements similar to those employed by the Applied
   Research Laboratory of Pennsylvania State University as modified to more
   closely apply to the agency's requirements (for example, testing over
   water instead of in a building) and to account for the testing resources
   that the agency had available.[7] See IMLCORP Supp. AR at 3-4. The Navy
   also contends that neither protester has shown that the use of a different
   testing methodology would have resulted in a different evaluation
   conclusion; in other words, the agency contends that neither protester has
   provided any data or test results that show that its acoustic hailing
   device would have been assessed as being superior to that offered by ATC,
   even if a different testing methodology were used.

   It is well established that it is a procuring agency's responsibility to
   determine its needs and how best to meet them. See CardioMetrix, B-270701,
   Mar. 13, 1996, 96-1 CPD para. 149 at 3. This responsibility includes
   determining the amount and type of testing necessary to evaluate a
   product's suitability. Terex Corp.; Caterpillar Tractor Co., B-217053;
   B-218535, July 24, 1985, 85-2 CPD para. 76 at 6. Our review of an agency's
   evaluation tests, as with other agency evaluation judgments, does not
   include conducting our own testing or reevaluating proposals; rather, our
   review is limited to examining the record to determine whether the
   agency's judgment was reasonable and in accord with the stated evaluation
   criteria and applicable procurement laws and regulations. See Abt Assocs.,
   Inc., B-237060.2, Feb. 26, 1990, 90-1 CPD para. 223 at 4.

   We find no basis in the record to disagree with the agency's view that,
   although there are a variety of tests available to assess voice
   intelligibility of devices such as the acoustic hailing devices, there is
   not an standard industry test, which offerors could reasonably expect
   would be used in this procurement. Although the protesters' believe that
   some other test would be more reasonable, their disagreement alone does
   not establish that the shout-off test used by the agency was an
   unreasonable method of assessing how well the firms' products would meet
   the agency's needs.

   The protesters nevertheless argue that the way the Navy conducted the
   shout-off test resulted in an unequal evaluation and that the test was
   also not conducted in accordance with the solicitation's specifications.
   Specifically, the protesters object that the Navy's use of the firms' own
   pre-recorded messages was unreasonable because the firms' product samples
   were not all assessed using a standard message and that the Navy used
   pre-recorded voices rather than live voices in the test.

   The Navy responds that the firms were permitted to submit, and in fact did
   submit, their own messages, which the agency presumed were chosen to
   optimize the performance of that firm's acoustic hailing device. Moreover,
   the Navy states that using

     multiple sound files reduces the likelihood that a listener will become
     familiar with the phrases and begin anticipating words in the phrase in
     later test runs. Utilizing the same phrase repeatedly can cause the
     listeners to anticipate the words in the phrase and distort the results
     for later test runs.

   IMLCORP Supp. Report at 5. The agency also contends that the use of a
   pre-recorded message, rather than broadcasting a live message, more
   accurately reflected the agency's needs, given that the "primary use of
   the [acoustic hailing devices] will be in danger zones outside U.S.
   territorial waters" and therefore these "messages are particularly useful
   due to the language and dialect variations where U.S. Naval vessels may
   anchor." Id. In other words, the agency reasonably anticipates that its
   actual use of the acoustic hailing devices would include the transmission
   of pre-recorded messages.

   We do not agree with the protesters that they were treated unreasonably in
   the shout-off test, even though the agency did not use the same message,
   or set of messages, to assess the voice intelligibility of the firms'
   devices in the shout-off tests and only used recorded messages. The
   protesters have not shown the agency's explanation for doing so to be
   unreasonable.[8] We also find that the Navy otherwise equally tested the
   firms' devices, given that these products were evaluated at the same time
   and place, operated by the same individuals, and evaluated by the same
   group of evaluators.

   The protesters also complain that Navy did not conduct the shout-off test
   in accordance with the RFP's specification requirements, given that the
   test was conducted with a background noise between 58 to 82 decibels, but
   the solicitation's performance specification provided for 100-percent
   voice intelligibility at 500 yards with a background noise of 88
   decibels.[9] While the Navy did not assess the voice intelligibility of
   the product samples against the background noise level identified by the
   RFP, the protesters have not shown that they were prejudiced in this
   regard, given that they have not explained how their voice intelligibility
   rating would improve and surpass that of ATC, if a higher background noise
   were employed.[10] The record also shows that the Navy assessed the
   offerors' compliance with this specification requirement under its
   evaluation of the written technical proposals under the specification
   compliance subfactor, and ATC's proposal demonstrated compliance with the
   requirement for 100-percent voice intelligibility at 500 yards with a
   background noise of 88 decibels. See AR, Tab 10, ATC Technical Proposal,
   at 22-23.

   IMLCORP also protested the Navy's evaluation of the firm's proposed
   acoustic hailing under a number of the solicitation's specification
   requirements. We have carefully reviewed each of IMLCORP's complaints in
   this regard and find them to be without merit. For example, IMLCORP
   complains that the Navy assessed a weakness with respect to the weight of
   IMLCORP's proposed product, but the record shows that no weakness was
   assigned to IMLCORP's proposal for the weight of its proposed acoustic
   hailing device. See AR, Tab 16, SSEB Consensus Report, at 8. As another
   example, IMLCORP complains that the Navy assessed a weakness under the
   "audio main lobe" specification because its proposed product exceeded the
   30-degree maximum dispersion pattern required by the RFP; IMLCORP admits
   however that its proposed device does exceed this requirement by 1 degree.
   Although IMLCORP contends that this was a minor deviation, this does not
   establish that the Navy could not reasonably assess a weakness for failing
   to meet this specification requirement.

   IMLCORP also challenges the Navy's evaluation of ATC's and IMLCORP's past
   performance, arguing generally that the two firms should not have received
   the same favorable, low risk rating under the past performance factor
   because ATC's acoustic hailing device had received a lower performance
   rating than IMLCORP's in "limited military utility assessment" (LMUA)
   testing performed by Alion Science and Technology and the Applied Research
   Laboratory of The Pennsylvania State University. See IMLCORP Comments,
   attach. A. However, the RFP provided for an assessment of offerors' past
   performance quality based upon an evaluation of the offerors' performance
   of prior contracts for the same or similar products, whereas the LMUA
   testing, to which IMLCORP refers concerns other entities' assessments of
   the quality of the products themselves, and not the offerors' past
   performance quality. We agree with the Navy that the RFP provided for the
   quality assessment of the offerors' proposed acoustic hailing devices
   under the capability evaluation factor, and that there was no requirement
   that the Navy consider other entities' testing of the acoustic hailing
   devices under this evaluation factor.

   The protests are denied.[11]

   Gary L. Kepplinger
   General Counsel

   ------------------------

   [1] The RFP does not define "voice intelligibility."

   [2] The product sample evaluation subfactor was said to be significantly
   more important that the other subfactors of the capability factor. RFP
   amend. 1, at 2.

   [3] The protests filed by IMLCORP and Wattre were not consolidated, and
   the Navy submitted separate reports for these protests. The core
   documents, such as the evaluation and source selection documents, were
   identical in the reports but were identified by different tab numbers. In
   this decision we refer to the agency report submitted in response to the
   IMLCORP protests, except where it is necessary to cite to differing
   documents in the agency report submitted in response to Wattre's protests.

   [4] The Shout-Off Test Plan was not included in the solicitation or
   otherwise provided to the offerors.

   [5] IMLCORP and Wattre raised a number of other protest allegations in its
   protests and supplemental protests that we dismissed as untimely during
   our development of this case. For example, Wattre untimely complained that
   the Navy had refused the firm's offer to provide training to the Navy's
   evaluators for Wattre's acoustic hailing device.

   [6] The Navy states that these other entities identified by the protesters
   all used different testing methodologies to assess voice intelligibility
   in prior studies. Wattre AR at 5.

   [7] The Navy provided a statement from an associate professor of acoustics
   at the Applied Research Laboratory of Pennsylvania State University, who
   states that the Applied Research Laboratory had not established a standard
   in its testing of the voice intelligibility of acoustic hailing devices.
   IMLCORP Supp. AR, attach. 1, Statement of Associate Professor of
   Acoustics. IMLCORP had earlier indicated that it would provide a statement
   from this individual but ultimately did not. IMLCORP Comments at 4.

   [8] While IMLCORP also argues that the use of recorded messages instead of
   live voices was inconsistent with the solicitation's requirements, it has
   not identified any solicitation provision requiring the use of live voices
   in testing the offerors' acoustic hailing devices, nor have we found any
   such requirement.

   [9] Wattre also complains that the Navy's shout-off test did not assess
   the acoustic hailing devices against every one of the RFP's specification
   requirements. The RFP, however, informed offerors that the agency "may
   test the product sample against the technical requirements set forth in
   the Performance Specification" and did not require the agency to do so.
   RFP at 38.

   [10] Wattre also protested that the Navy may not have properly calibrated
   its test equipment prior to the shout-off testing. The Navy responded that
   the test equipment used in the shout-off test were two sound level meters,
   which the Navy states it did not need to calibrate. Wattre Report at 23.
   Because Wattre did not address the Navy's arguments in its comments, we
   consider this protest allegation to be abandoned. See Planning Sys., Inc.,
   B-292312, July 29, 2003, 2004 CPD para. 83 at 6.

   [11] In their comments and supplemental comments, IMLCORP and Wattre
   untimely raised a number of other challenges to the Navy's selection of
   ATC's proposal for award. Our Bid Protest Regulations require that
   protests based on other than alleged improprieties in a solicitation be
   filed no later than 10 calendar days after the protester knew, or should
   have known, of the basis for protest, whichever is earlier. 4 C.F.R.
   sect. 21.2(a)(2) (2007). Because these numerous other protest allegations
   were not timely filed, we have not considered them.