TITLE: Recon-Optical, Inc., B-286529, January 18, 2001
BNUMBER: B-286529
DATE: January 18, 2001
**********************************************************************
Recon-Optical, Inc., B-286529, January 18, 2001
Decision
Matter of: Recon-Optical, Inc.
File: B-286529
Date: January 18, 2001
William B. Barton, Jr., Esq., William T. Welch, Esq., and Jerome H. Gress,
Esq., Barton, Baker, McMahon & Tolle, and Thomas Earl Patton, Esq., Tighe,
Patton, Armstrong Teasdale, for the protester.
Andrew E. Squire, Esq., and Charles L. Bidwell, Esq., Department of the
Navy, for the agency.
Guy R. Pietrovito, Esq., and James A. Spangenberg, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision.
DIGEST
1. Protest of the inclusion of an algorithmic formula--the general image
quality equation (GIQE)--as part of the evaluation in a procurement for
aerial imaging sensors is denied, where the GIQE is an accepted method
of assessing imaging sensors performance, where the protester has not
shown that the equation cannot reasonably predict performance of
proposed sensors, and where the solicitation permits offerors to submit
other data and analyses, in addition to the GIQE analysis, to
substantiate performance claims for proposed sensors.
2. Solicitation pricing scheme that provided different quantities of
sensors to be priced, depending upon whether an offeror proposed a
single sensor meeting all the performance requirements or two sensors
working together to meet the requirements, is not defective where the
scheme provided for the pricing of essentially equivalent numbers of
sensors and reflected a reasonable approach to evaluating the pricing
of different technical approaches.
DECISION
Recon-Optical, Inc. protests the terms of request for proposals (RFP) No.
N00019-99-R-1567, issued by the Naval Air Systems Command, for imaging
sensors and associated work to be installed in the Shared Reconnaissance Pod
(SHARP) system.
We deny the protest.
The Navy is seeking a high and medium altitude reconnaissance capability to
replace the Tactical Air Reconnaissance Pod System it currently uses on the
F-14 aircraft for reconnaisance. To this end, the RFP was issued for sensor
and sensor suite fabrication, integration, installation, and testing as a
Program Engineering and Manufacturing Development (EMD) effort. RFP attach.
1, Statement of Work (SOW), at 1.0. Offerors were informed that that they
could provide either separate and interchangeable medium and high altitude
sensors or a single sensor; the RFP stated that a single sensor meeting all
medium and high altitude requirements was desirable. RFP attach. 2,
Performance Specification, at 3.2. The sensors will eventually be installed
in pods to be attached to the F/A-18E/F aircraft, six of which the Navy is
procuring under a separate procurement action. Agency Report at 2.
Sensor imaging requirements were among the performance requirements
identified in the RFP. These requirements are stated in terms of threshold
(minimum) and objective National Imagery Interpretability Rating Scales
(NIIRS) at various altitudes and various angles of view. For example, for
visible resolution at a medium altitude overflight (imaging within 10
degrees or less of nadir) the RFP requested sensors that would provide a
threshold rating of NIIRS 6 and an objective rating of NIIRS 7. The NIIRS is
used by the aerial imaging community, and others, to define the quality of
images and performance of imaging systems. Under NIIRS an image is assigned
a number from 0 to 9 (with a lower number indicating less image recognition)
to reflect a level of image recognition and interpretability. See
Alternative Dispute Resolution (ADR) Conference Videotape (VT) at 10:04-05;
[1] see also L.A. Maver, C.D. Erdman, K. Riehl, Imagery Interpretability
Rating Scales, 20 Society for Information Display (SID) Digest at 117-20
(1995), reprinted at . Simply stated, a NIIRS rating represents a subjective
judgment of what can be humanly perceived. VT at 10:26. For example, "[a]t
NIIRS 1 for visible imagery, the criterion indicates the ability to detect a
medium sized port facility. At NIIRS 9, the requirement is to differentiate
cross-slot from single-slot heads on aircraft skin panel fasteners." General
Image Quality Equation (GIQE) User's Guide, version 4.0 (December 1996) at 2
(italics in original).
The RFP included fixed-price line items for the sensors and
cost-reimbursement line items for engineering/integration and logistics
support services. RFP amend. 1 sect. B at 3-4. With respect to the sensors, the
RFP requested fixed-prices for basic and option quantities as follows:
Offer of Two Separate Sensors
Quantity Price
Base Quantity:
High Altitude Sensor 1 unit
Option Quantity:
Medium Altitude Sensor 1 unit
Sensor Suite [2] 1-2 Suites
2-4 suites
Offer of Single Combined Sensor
Base Quantity:
Combined High/Medium Altitude 1 unit
Sensor
Option Quantity:
Combined High/Medium Altitude 1-2 units
Sensor
3-4 units
5-6 units
7-8 units
RFP amend. 1 sect. B at 3-4. Offerors were informed that in evaluating price the
agency would add the price for the base quantity to the price for the option
quantity. RFP sect. M at 57.
The RFP provided for award on a cost/technical tradeoff basis, and
identified the following evaluation factors in descending order of
importance: technical, price/cost, logistics, management, and past
performance. Subfactors were stated for each of the non-price/cost
evaluation factors. For example, under the technical factor, offerors were
informed that the agency would evaluate the maturity of the offeror's design
with emphasis placed upon the maximum use of commercial-off-the-shelf
(COTS)/nondevelopmental item (NDI) sensors and/or components that have been
demonstrated in a flight environment. Also under the technical factor, the
RFP stated that the proposed design would be evaluated to determine if
performance claims are "substantiated via analysis, lab, and flight data."
RFP sect. M at 61.
Proposal preparation instructions were provided in the RFP. Among other
things, offerors were requested to provide any laboratory or flight test
data to establish the maturity of its proposed system. With respect to
system performance, offerors were instructed to
provide analysis which demonstrates how the proposed design(s) meets or
exceeds the imaging requirements, and address any assumptions used in the
analysis. The analysis shall include an analysis of predicted imaging
performance using the General Imaging Quality Equation [GIQE], as specified
in paragraph 3.4.1.3 of the Performance Specification for SHARP Imaging
Sensors. If requirements cannot be met, the offeror should address
trade-off's and provide a recommended approach. The offeror should support
the analyses with lab[oratory] or flight performance data.
RFP sect. L at 49. Paragraph 3.4.1.3 of the Performance Specification, as
amended, provided that for the purpose of system design analysis, the NIIRS
performance predictions would be made using the GIQE, in accordance with the
GIQE User's Guide, version 4.0, which was available on the agency's website
at
.
The GIQE was developed under the auspices of the government's Imagery
Resolution Assessment and Reporting Standards Committee and is designed to
predict NIIRS ratings for visible and infrared images based on knowledge of
electro-optical and infrared system design and operating parameters. See
GIQE User's Guide at ii, 1, 3; see also J.C. Leachtenauer, W. Malila, J.
Irvine, L. Colburn, and N. Salvaggio, General Image-Quality Equation: GIQE,
36 Applied Optics 8322-28 (Nov. 1997). The equation "is a regression-based
model that accounts for perceptual quality attributes of scale, resolution
or sharpness, contrast, and noise." [3] GIQE User's Guide at 7. Application
of the GIQE algorithmic formula results in a numerical NIIRS rating for the
sensor. VT at 10:18 (testimony of the Navy's expert).
Recon-Optical complains that the requirement that offerors use the GIQE to
establish their proposed sensor's performance is overly restrictive because
the equation tends to favor one imaging technology over another.
Specifically, Recon-Optical states that the GIQE underpredicts the NIIRS
rating actually achievable by framing technology (the technology that
Recon-Optical would offer) because, among other things, the equation
provides insufficient weight to the framing technology's inherent ability to
enhance signal-to-noise ratio (SNR). Recon-Optical asserts that the GIQE was
created around linear or scanning technology, and thus does not adequately
recognize the benefits of framing technology. [4] In this regard,
Recon-Optical states that the GIQE is not a generally accepted method of
predicting performance and argues that the current industry measure for
aerial image resolution is the use of the ground resolvable distance (GRD)
analysis, which is a laboratory analysis that uses a rating system different
from the NIIRS scale. [5] Recon-Optical also complains that the application
of the GIQE to this procurement is ambiguous because the Navy has not
specified uniform parameters for offerors to use in applying the formula.
The Navy responds that the GIQE is an established and reasonable predictor
of sensor performance and in fact is the only recognized method of
predicting performance of sensors in terms of a numerical NIIRS rating.
Supplemental Declaration of Navy's Expert (Dec. 4, 2000) at 2. The agency
also states that GRD, which the protester advocates, will not predict a
sensor's performance in terms of a NIIRS rating. Id. In any event, the Navy
states that the RFP allows an offeror to provide whatever information or
explanation the firm wishes to substantiate performance claims for proposed
sensors, so long as the firm also provides the GIQE analysis establishing
the claimed NIIRS rating for the proposed sensors. With respect to the
argument that the GIQE is ambiguous because the agency had not specified
uniform parameters for offerors to use in the GIQE assessment, the Navy
states that the GIQE User's Guide explains the parameters and their use, and
notes that the inherent flexibility of the GIQE allows offerors to select
their own parameters (with supporting analysis) to maximize the performance
of their proposed sensor(s).
We find no basis from our review of the record to object to the Navy's
proposed use of the GIQE formula as one means of assessing a proposed
sensor's predicted performance. Contrary to Recon-Optical's arguments, the
GIQE is an accepted method of reasonably assessing imaging sensors'
performance in terms of NIIRS. See, e.g., Leachtenauer, Malila, Irvine,
Colburn, and Salvaggio, General Image-Quality Equation: GIQE, 36 Applied
Optics at 8328, which concludes:
It thus appears that, in terms of NIIRS-prediction accuracy, the GIQE is at
least equal to, and probably better than, other available metrics.
See also Maver, Erdman, Riehl, Imagery Interpretability Rating Scales, 20
SID Digest at 120, which states:
The NIIRS has also become a standard by which the performance of new imaging
systems are specified in procurements and evaluated in acceptance testing.
The NIIRS is frequently used in algorithm evaluations [such as the GIQE] to
measure the impact of degradations resulting from data compression or
enhancements such as sharpening algorithms.
Furthermore, the GIQE has been used in other federal procurements concerning
imaging sensors; for example, the GIQE was used by the Department of the Air
Force in the Global Hawk unmanned aerial vehicle program as a means of
predicting NIIRS performance of that vehicle's electro-optical and infrared
sensors. Declaration of Navy's Expert (Oct. 31, 2000) at 2-3.
We are also not persuaded by Recon-Optical that the GIQE unduly favors
scanning technology over framing technology. The protester and agency have
provided competing technical arguments concerning whether the GIQE formula
will fairly capture the performance of Recon-Optical's sensor, specifically,
or the performance of framing technology sensors, generally. Ultimately,
these submissions demonstrate that the parties disagree, but do not
establish that it was unreasonable for the agency to require offerors to
submit the GIQE analysis of offerors' proposed sensors. [6] As demonstrated
by the Global Hawk procurement mentioned above, where the awardee (Teledyne
Ryan Aeronautical) [7] offered sensors based upon framing technology, the
use of GIQE does not necessarily mean that offers based on framing
technology will not receive the award. Id. Moreover, as noted, the GIQE is
currently the only method of assessing imaging sensors' peformance that
provides a numerical NIIRS rating as the result of the application of the
equation. Supplemental Declaration of Navy's Expert (Dec. 4, 2000) at 2.
Finally, as indicated above, other observers have found that, in terms of
predicting NIIRS accuracy, the GIQE is "at least equal to, and probably
better than, other available metrics." [8] Leachtenauer, Malila, Irvine,
Colburn, and Salvaggio, General Image-Quality Equation: GIQE, 36 Applied
Optics at 8328.
Of significance to our decision is that the GIQE analyis, although required,
is not the only data and analyses which offerors can provide and the agency
will consider. The RFP provides that the agency will assess an offeror's
predicted imaging performance and whether the offeror's performance claims
were substantiated. RFP sect. M at 61. To accomplish this, offerors were
required to provide whatever data, analyses, or laboratory results the firm
wished to provide (so long as the firm also provided the requested GIQE
analysis) to establish or explain claimed performance. See RFP sect.sect. L, M at
49, 61. Thus, Recon-Optical is permitted to provide the GRD data that it
states the agency should be considering. It also is permitted to provide
data supporting its claim that the GIQE assessment does not completely
capture its particular sensor's performance. In short, we think the
solicitation request for any manner of data and analyses supporting a
sensor's claimed peformance can reasonably be expected to provide the Navy
with a solid technical foundation upon which to exercise its business
judgment in selecting which offer reflects the best value to the government.
[9]
We also do not agree with Recon-Optical that the use of the GIQE formula is
ambiguous here because the agency did not define uniform parameters or
factors that offerors were to use. As explained by the Navy, the RFP
provided that the offerors would themselves specify the values (along with
analyses supporting the values chosen) of the GIQE factors depending upon
the sensor(s) proposed, which allowed offerors to maximize the claimed
performance capable from a proposed sensor. Definitions and computational
guidance for applying the GIQE formula are provided in the GIQE User's Guide
referenced by the RFP. [10] Based on our review, we find that the RFP's
instructions for the use of GIQE are not ambiguous.
Recon-Optical also protests the RFP pricing scheme, which had different line
items to be completed depending upon whether an offeror proposed a single
sensor or multiple sensors to satisfy the RFP requirements. The protester
complains that this pricing scheme favors single sensor offerors that are
allowed to price a greater base and option quantity of sensors (eight total)
than multiple sensor offerors (four sensor suites). Multiple sensor offerors
are disadvantaged, Recon-Optical states, because there are fewer items
against which the offeror can allocate its developmental and project costs.
The Navy responds that the RFP does not require pricing for a greater
quantity of single sensors, but provides for the acquisition of an
essentially equivalent number of sensors. That is, the Navy states that a
single sensor offeror would be pricing a total of 9 sensors (base and option
quantities) and a multiple sensor (a high and a medium sensor) offeror would
be pricing a total of 10 sensors (base and option quantities). The Navy
contends that this pricing scheme was required to allow vendors to offer
either a single sensor or multiple sensor approach.
We find no merit to the protester's complaint concerning the RFP's pricing
scheme. As stated by the agency, the RFP was structured to allow offerors
flexibility to propose varying approaches (single sensor as compared to
multiple sensors in sensor suites) to satisfying the agency's requirements.
As noted above, the agency has a need for high/medium altitude sensors or
sensor suites to install in the six pods that it is acquiring under a
separate procurement action (with the remaining sensors being purchased as
spares). Thus, it has set up a scheme that provided for the pricing of
essentially equivalent numbers of sensors recognizing the two possible
approaches of satisfying the agency's requirements. Under the circumstances,
we think that the agency has devised a reasonable approach to allow single
sensor and multiple sensor vendors to compete without unfairly evaluating
price/cost.
The protest is denied.
Anthony H. Gamboa
Acting General Counsel
Notes
1. An ADR conference was conducted to provide settlement assistance to the
parties. At the conference, we received legal argument and testimony from
the protester's technical staff and the Navy's expert (a privately employed
research scientist in the field of reconnaissance sensor development). The
parties were unable to reach agreement resolving the protest.
2. According to the RFP, "[a] sensor suite is defined as that combination of
High Altitude and Medium Altitude Sensors that best meets the mission
requirements as contained in the attached Statement of Work and
Specification." RFP amend. 1 sect. B at 3.
3. The GIQE formula consists of five factors (or parameters): ground sampled
distance (a measure of scale and resolution), relative edge response (a
measure of perceived sharpness), edge height overshoot (edge sharpening, if
image processing techniques used), noise gain (increase in noise due to edge
sharpening), and signal-to-noise ratio (a measure of noise). Agency Report,
Declaration of Navy's Expert (Oct. 31, 2000) at 2; see Leachtenauer, Malila,
Irvine, Colburn, and Salvaggio, General Image-Quality Equation: GIQE, 36
Applied Optics at 8324-25. The GIQE User's Manual provides definitions and
computational procedures for each of these factors.
4. There are two basic technologies used for aerial imaging sensors: framing
technology and scanning technology. Framing technology receives light or
infrared energy from each section of a field of view simultaneously.
Scanning technology divides up a field of view into squares arranged into
rows and columns, and each square is integrated into a picture. See Protest
at 6.
5. GRD, simply stated, is a physical test under which a target (such as a
chart of bars) is used to determine at what distance the bars on the chart
can be distinguished by a sensor. From this test, other imaging parameters
of the sensor can be extrapolated. VT at 9:36-37, 10:08-09.
6. Although the record indicates that there are differences between the two
technologies, see VT at 10:06-07 (each technology has its own strengths and
weaknesses), the Navy's expert testified that the GIQE assessment reasonably
captures the sensor's performance characteristics regardless of technology
used. VT at 10:07.
7. Teledyne Ryan Aeronautical is now Northrop Grumman Ryan Aeronautical
Center.
8. Recon-Optical's challenge to the use of the GIQE to predict sensor
performance is, in part, based upon the firm's apparent belief that the RFP
is seeking a COTS/NDI system and that therefore the use of a predictive
model is unwarranted. Although the RFP requested that offerors propose NDI
components to the maximum extent practicable, the procurement is an EMD
effort and not an acquisition of a COTS/NDI system.
9. To the extent that Recon-Optical is complaining that the Navy will not
reasonably consider or weigh the data and analyses provided by the offerors,
in addition to the GIQE analysis, this challenge to possible, future agency
action is speculative and premature, and will not be considered at this
time.
10. Recon-Optical also complains that the GIQE model does not specifically
explain the relationship between relative edge response and modulation
transferability function. The Navy responds that this relationship is
provided in the GIQE User's Guide at pages 10 to 11.