TITLE: ENMAX Corporation, B-281965, May 12, 1999
BNUMBER: B-281965
DATE: May 12, 1999
**********************************************************************
ENMAX Corporation, B-281965, May 12, 1999
Decision
Matter of: ENMAX Corporation
File: B-281965
Date: May 12, 1999
Blake S. Atkin, Esq., Atkin & Lilja, for the protester.
John E. Lariccia, Esq., Michael Mullin, Esq., and Cpt. Rex D. Hockaday,
Department of the Air Force, for the agency.
Ralph O. White, Esq., and Christine S. Melody, Esq., Office of the General
Counsel, GAO, participated in the preparation of the decision.
DIGEST
1. Protester's contention that the awardee's proposal was improperly
considered technically acceptable is sustained where the record shows that
the solicitation required offerors to address all of the identified areas
within each technical evaluation factor, where the awardee clearly failed to
address certain areas, and where there is no evidence in the record that the
agency evaluators either recognized the failure of the proposal to address
these areas, or concluded that the proposal was acceptable overall despite
the omission of these areas.
2. Protester's contention that the evaluation of performance risk was flawed
because the awardee's prior experience is not relevant to the solicited
effort is denied where the record shows that the solicitation anticipated
reviewing a broader level of prior experience than the solicited effort, and
also shows that the assessment of the awardee's past performance was
reasonable and consistent with the evaluation criteria.
3. Challenge to the agency's review of price realism is denied where the
agency reasonably concluded that the offerors's proposed hourly labor rates
were sufficient to attract and retain quality professionals by comparing the
proposed rates to Bureau of Labor Statistics rates, as the solicitation
advised.
DECISION
ENMAX Corporation protests the award of a contract to Carlisle Research,
Inc. pursuant to request for proposals (RFP) No. F19628-98-R-0445, issued by
the Department of the Air Force for engineering services in support of the
software development activities of the Materiel Systems Group at Hill Air
Force Base, Utah. ENMAX argues that the Air Force evaluation of Carlisle's
proposal was improper in three areas--the conclusion that the proposal was
technically acceptable, the decision that the proposal presented low
performance risk, and the review of Carlisle's proposed compensation rates.
We sustain the protest.
BACKGROUND
The RFP was issued on September 21, 1998, and limited the competition to
participants in the Small Business Administration's (SBA) section 8(a) small
disadvantaged business program, pursuant to 15 U.S.C. sect. 637(a) (1994). The
RFP contemplated award of an indefinite-delivery, indefinite-quantity,
fixed-price labor-hour contract [1] with a term of 5 years. RFP sect.sect. B.A.,
F.A. The work was priced using on-site and off-site labor rates. The RFP's
evaluation scheme, described in greater detail below, advised that award
would be made using a performance/price tradeoff, which "permits tradeoffs
between price/cost and the past performance evaluation for technically
acceptable proposals." RFP amend. 0002, sect. M.A.8.
Under the RFP's evaluation scheme, proposals were first reviewed for
technical acceptability under three evaluation factors: (1) management of
multi-disciplinary teams in an information technology environment; (2)
sustainment/maintenance in an SEI/CMM [2] environment; and (3)
enhancement/modernization in an SEI/CMM environment. RFP amend. 0002, sect.
M.B.2. Each of these factors was followed by a narrative description of the
scope of the factor, which the evaluators transformed into a [deleted]
checklist; each item on the checklist was graded on a pass/fail basis.
Evaluation Plan, Oct. 15, 1998, at 11. Proposals found to be technically
acceptable were next evaluated for performance risk, using ratings of low,
moderate, or high. RFP amend. 0002, sect. M.B.3.b. In addition, the Air Force
evaluated each offeror's price. The RFP advised that past performance and
price would be equally important, with award to the offeror whose proposal
presented the best combination of performance risk and evaluated price. Id.
sect. M.A.8.
After receiving and reviewing eight proposals, the Air Force concluded that
[deleted] of them, including those submitted by ENMAX and Carlisle, were
technically acceptable. The agency next concluded that [deleted] of the
acceptable proposals (again including those submitted by ENMAX and
Carlisle), presented low performance risk based on the past performance
review. Of the [deleted] technically acceptable, low-risk proposals,
Carlisle had the lowest evaluated price [deleted] followed by ENMAX
[deleted]. Based on this information, the agency selected Carlisle for
award. Memo for Record, Feb. 10, 1999, at 3.
DISCUSSION
In its challenges to the Air Force's evaluation of Carlisle's proposal,
ENMAX focuses on whether Carlisle has gained sufficient technical expertise
and relevant past performance from its prior contract experience. Carlisle's
predominant experience has been in providing support for the [deleted].
Carlisle Technical Proposal, Oct. 29, 1998, at 2. Among other things,
Carlisle maintained and administered the [deleted] Wide Area Network (WAN),
and the Air Force explains that Carlisle "has successfully supported this
system at all engineering levels and project management levels." Air Force
Reply to ENMAX Protest, Agency Report, Tab 2, attach. 11, at 5.
Technical Acceptability of Carlisle's Proposal
ENMAX contends that Carlisle's proposal did not meet the minimum technical
requirements set forth in the RFP, and that the Air Force improperly
concluded that the proposal was technically acceptable. Specifically, ENMAX
claims that Carlisle has failed to demonstrate 2 or more years of experience
in the ORACLE, UNIX, NT, and Powerbuilder computing environments. [3]
In considering a protest challenging an agency's evaluation of proposals, we
will examine the record to determine whether the agency's judgment was
reasonable and consistent with stated evaluation criteria and applicable
statutes and regulations. ESCO, Inc., B-225565, Apr. 29, 1987, 87-1 CPD para.
450 at 7. We have reviewed the pleadings, the solicitation, the proposals,
and the evaluation materials, and, based on the record before us, we are
unable to conclude that the evaluation was reasonable.
To adequately discuss ENMAX's challenge in this area, we begin with a more
detailed review of the solicitation's instructions to potential offerors. As
stated above, the RFP identified three evaluation factors, and in the
narrative immediately preceding the recitation of these factors, the RFP
explained, in relevant part, that:
Each factor, identified below, will be considered equally important and be
rated as either pass or fail. The technical evaluation team will use
relevant experience to correlate with specific factors below. Only those
offerors which receive a "pass" evaluation on all three factors will be
eligible for award. The offeror shall ensure that each and every area of
each factor has been addressed. Since the technical area is rated as
pass/fail, failure to do so may result in failure of the technical proposal.
RFP amend. 0002, sect. M.B.2. The evaluation factors followed the above-quoted
narrative, and each factor was followed by its own narrative expanding on
the scope of the factor.
The requirement that the agency review an offeror's experience in ORACLE,
UNIX, NT, and Powerbuilder, arises from the third of the three technical
evaluation factors, which is set forth below:
FACTOR 3: ENHANCEMENT/MODERNIZATION IN A SEI/CMM ENVIRONMENT
The Government will evaluate the ability to direct the overall system(s)
migration from legacy mainframe architecture to an open architecture,
client/server environment. The Government will evaluate experience of
offeror (2 plus years) in client/server development in ORACLE, UNIX, NT, and
Powerbuilder environments and emerging technologies such as web enabled
applications development and large scale (over 1 terabyte) data depot/data
warehouse projects. The Government will evaluate offeror's experience in
transitioning a system(s) to a shared data environment, containing standard
data elements and facilitating global access via a distributed computing
environment (a form of data warehousing) and experience in data management
to include data standardization, modeling, integration, architecture,
design, and data extraction/loading. The Government will evaluate the
ability to transition a system to the Integration and Runtime Specification
(I&RTS) in accordance with Defense Information Infrastructure (DII) Common
Operating Environment (COE) compliance, and successful DII certification.
The Government will evaluate offeror experience in successfully reusing
software code components, modules, objects and/or segments when
transitioning or enhancing a system to include maintenance of software code
libraries and the offeror's ability to recognize and apply proven
state-of-the-industry technologies to evolve existing system(s).
Id. (emphasis added).
As described above, the agency evaluators generated a [deleted] checklist
from the requirements identified in the RFP. Of relevance here is the
[deleted] item on the checklist:
[deleted].
1. Evaluation Plan for Engineering Services Support, Oct. 15, 1998,
attach. 1. The checklist items were graded on a pass/fail basis. Since
Carlisle's proposal does not identify experience with [deleted], ENMAX
argues it was unreasonable to conclude that Carlisle should receive a
grade of "pass" under this checklist item. Thus, ENMAX argues it was
unreasonable to conclude that the Carlisle proposal was technically
acceptable.
As a preliminary matter, we note that it is the solicitation, not the
internal evaluation plan, that is the touchstone for whether offerors have
been treated fairly in an evaluation. Requirements stated in evaluation and
source selection plans are not disclosed to offerors, and thus, do not give
outside parties any rights. Mandex, Inc.; Tero Tek Int'l, Inc., B-241759 et
al., Mar. 5, 1991, 91-1 CPD para. 244 at 7. Our review here is limited to the
RFP language, not the evaluation checklist items that are the focus of the
protester's arguments.
Carlisle's technical proposal shows that, at best, Carlisle has experience
in the [deleted]. [4] Carlisle Technical Proposal, supra, at 13-14. We see
no evidence to support a conclusion that Carlisle has any experience in
[deleted], other than the proposal's assertion that the [deleted] WAN
"incorporate[s] the same principles" as these systems. Id. at 14.
Nonetheless, the evaluators gave Carlisle a rating of "pass" under the
"[d]emonstrated experience (2 plus years) in client/server development in
ORACLE, UNIX, NT, and Powerbuilder environments" [deleted].[5]
The next issue is whether Carlisle's failure to show the required experience
in all four computing environments should have led the agency to conclude
that the proposal was technically unacceptable under the third evaluation
factor (a finding which, under the terms of the RFP, would have resulted in
an overall rating of technically unacceptable). The RFP does not specify
that failure to meet any one of the technical areas under an evaluation
factor would result in a finding of technical unacceptability under the
factor overall. In this regard, while the RFP does require offerors to
address each and every area within each factor, it says only that failure to
do so "may result in failure of the technical proposal." RFP amend. 0002, sect.
M.B.2. While, given this language, we cannot conclude that failure to
address any one area must result in a finding of technical unacceptability,
it clearly could have affected the evaluators' conclusion that Carlisle was
technically acceptable. Accordingly, we conclude that the evaluators had to
consider whether Carlisle's failure to demonstrate the required experience
made it technically unacceptable under the third evaluation factor, and we
sustain the challenge to the evaluation in this area.
Past Performance
ENMAX argues that it was unreasonable for the Air Force to award Carlisle a
low performance risk rating because Carlisle's past performance has limited
relevance to the effort covered by the solicitation here. According to
ENMAX, the heart of the current effort is writing code for a large scale
management information system (MIS) designed using ORACLE, UNIX, NT, and
Powerbuilder, while Carlisle's past experience is less broad and has been
largely limited to maintaining the [deleted] WAN. In addition, ENMAX argues
that the survey questionnaire used by the Air Force to collect past
performance information was inadequate.
With respect to ENMAX's challenge to the agency's conclusions about
Carlisle's past performance, our review consists of examining the record to
determine whether the agency's judgment was reasonable and consistent with
stated evaluation criteria and applicable statutes and regulations.
Aerospace Design & Fabrication, Inc., B-278896.2 et al., May 4, 1998, 98-1
CPD para. 139 at 12. For the reasons set forth below, we conclude that the
evaluation was reasonable.
The record here shows that Carlisle provided [deleted] past performance
references it deemed relevant to this effort. All of these references were
sent surveys, and all of them completed and returned the surveys. Each
survey consisted of 17 questions for which the reference rated the offeror
as "above average," "average," "below average," or "not observed." Surveys,
Agency Report, Tab 6, attach. 3. A response of "above average" was assigned
3 points; a response of "average" was assigned 2 points; a response of
"below average" was assigned 1 point. Memo to the File, Evaluation of Past
Performance, Dec. 17, 1998, at 1. Responses of "not observed" were not
counted further. These responses were then averaged across the survey, and
translated to adjectival ratings as follows: 2.5 to 3.0, low risk; 1.5 to
2.4, moderate risk; and, 0 to 1.4, high risk. All [deleted] of Carlisle's
references rated Carlisle above 2.5 and all [deleted] indicated they would
hire Carlisle in the future. Surveys, supra.
In evaluating Carlisle's past performance information, the Air Force noted
that some of the work cited by Carlisle was not precisely the type of work
covered by the solicitation. In this regard, the agency stated:
[deleted].
Air Force Reply to ENMAX Protest, Agency Report, Tab 2, attach. 11, at 14.
We start our analysis by noting that there is a difference between ENMAX's
claims about the skills that lie at the heart of this effort, and the RFP's
guidance regarding the evaluation of past performance. Specifically, while
ENMAX correctly asserts that the solicitation emphasized the agency's desire
for proposals showing past performance of comparable complexity, and with
similar technical requirements, RFP amend. 0002, sect.M.B.3.c, the solicitation
also stated that it would consider the offeror's past experience with:
Information technologies management, systems modernization, operations and
maintenance in a software development environment to include the ability to
provide the necessary organization, experience, accounting and operational
controls, technical skills, and equipment.
Id. sect. M.B.3.c.(1). Nothing in this language limits offerors to precisely the
same effort as covered by the solicitation here; rather, the solicitation
anticipates review of a fairly broad range of experience in developing
software. [6]
In addition, we see nothing about Carlisle's references, their responses, or
the Air Force's assessment of the relevancy of the references, which
precludes a reasonable determination that Carlisle's proposal presents low
performance risk. First, there is nothing in the record to support a
conclusion that Carlisle does not have experience that falls within the
broader range of software development experience highlighted in the past
performance portion of section M of the RFP, and quoted above. In addition,
to the extent that the referenced experience is somewhat different from the
scope of the solicited effort, the Air Force recognized the differences and
concluded that Carlisle's experience was sufficiently close to be relevant.
Finally, the actual responses received were overwhelmingly favorable.
Accordingly, we conclude that the agency reasonably assigned Carlisle's
proposal a low performance risk rating based on these references. See Apache
Enters., Inc., B-278855.2, July 30, 1998, 98-2 CPD para. 53 at 3.
With respect to ENMAX's argument that the survey questions used here were
inadequate to assess the relevance of Carlisle's experience, ENMAX claims
that the survey questions should have been more tailored to uncover
information about the similarity between the effort solicited here, and the
effort covered by the survey. Generally, past performance surveys do not
request information about the relationship between the solicited effort and
the prior contract; in fact, to do so would substantially increase the
burden on individuals asked to complete such surveys, as they would need
information about the solicited effort. Instead, each offeror's past
performance proposal was required to include an explanation of how the
identified contracts are relevant to the solicited effort. RFP amend. 0002,
sect. L.F.3.d. This information was then reviewed by the agency, which was in
the best position to make judgments about the relevance of earlier projects.
In addition, our review of the past performance survey provides no basis to
conclude that it was unreasonable, improper, or failed to discern the
information which it sought. As explained above, the survey consisted of a
total of 17 questions for rating the offerors (and a small number of
additional questions). Also, the survey was broken into areas designed to
correspond with the three technical evaluation factors in the RFP. Surveys,
supra. The questionnaire covered the kind of information routinely sought by
such questionnaires, including information about whether the agency would
like to do business with the company in the future. Given that the record
shows that the questionnaires were adequate to survey prior customers of
Carlisle, and that the ratings reflected the judgment of cognizant
contracting officials with specific knowledge of the contracts in question,
we find that the questionnaires used here were reasonably designed to
generate information of sufficient reliability to support the Air Force's
assessment of past performance. See Pacific Ship Repair and Fabrication,
Inc., B-279793, July 23, 1998, 98-2 CPD para. 29 at 4.
Price Evaluation
ENMAX argues that the Air Force failed to perform an adequate price realism
evaluation because the agency simply compared each offeror's proposed hourly
labor rates with standard rates compiled by the government. According to
ENMAX, this approach, without more, was insufficient to determine the
realism of proposed prices, and violates the scheme set forth in Federal
Acquisition Regulation (FAR) section 52.222-46, "Evaluation of Compensation
for Professional Employees."
As an initial matter, price realism ordinarily is not considered in the
evaluation of proposals for the award of a fixed-price contract because the
government's liability is fixed and the risk of cost escalation is borne by
the contractor. The Centech Group, Inc., B-278715, B-278715.2, Mar. 5, 1998,
98-1 CPD para.108 at 5. However, because there is a risk of poor performance in
certain circumstances--such as when a contractor attempts to provide
products or services at little or no profit, id. at 5-6, or when a
contractor fails to offer sufficient compensation levels to obtain and keep
qualified professionals, see Research Management Corp., B-237865, Apr. 3,
1990, 90-1 CPD para. 352 at 8--an agency may in its discretion provide for a
price realism analysis in the solicitation of fixed-price proposals. The
Centech Group, Inc., supra, at 6. The nature and extent of an agency's price
realism analysis is a matter within the sound exercise of the agency's
discretion. Cardinal Scientific, Inc., B-270309, Feb. 12, 1996, 96-1 CPD para.
70 at 4.
The RFP here anticipated award of a fixed-price labor-hour contract (with
minor cost reimbursement elements not germane to this discussion), and
advised that the agency would evaluate the prices of all technically
acceptable proposals for realism, reasonableness, and completeness. RFP
amend 0002, sect. M.B.4.c.-d. The RFP also included the text of the
above-referenced, standard FAR clause on compensation of professional
employees, which provides that "[t]he government will evaluate the
[compensation] plan to assure that it reflects a sound management approach
and understanding of the contract requirements." FAR sect. 52.222-46(a). In
essence, ENMAX argues that the standard FAR clause with its anticipated
two-pronged review--i.e., a review of an offeror's management approach, and
a review of its understanding of the contract requirements--promised
something more than the Air Force provided. We disagree.
The RFP's guidance on this issue was not limited to the FAR clause, but also
included the following:
In accordance with FAR 52.222-46, "Evaluation of Compensation for
Professional Employees," [the] cost realism evaluation will include a review
of salary and fringe benefit information provided in the proposal. It is the
Government's intent to use Department of Labor, Bureau of Labor Statistics
(BLS) data in support of the Professional Clause, 52.222-46, to ensure that
it is reasonable for the offeror to attract and retain quality profesionals.
If salaries and fringe benefits proposed appear unrealistically low, the
proposal may be rejected.
RFP amend. 2, sect. M.B.4.e (emphasis added); see also sect. L.F.4. In addition, the
RFP directed offerors to a website where the BLS data could be found. Id. sect.
L.F.4. Offerors were advised that the website contained a cross-referenced
matrix linking the labor categories in the RFP to the BLS labor hour rates.
Questions and Answers on RFP, question 23, p.7.
In our view, ENMAX should have been on notice from the terms of the RFP,
together with the structure of the solicitation's pricing, that the Air
Force might not look beyond a comparison of the proposed rates to the BLS
rates. The solicitation required only that offerors submit a labor-hour rate
for each of the labor categories. By comparing the proposed rates to the BLS
rates for each category, the agency concluded that Carlisle's proposed rates
were reasonable, realistic, and sufficiently high to hire and retain the
necessary personnel in the Ogden, Utah area. This conclusion was both
reasonable, and consistent with the RFP's stated evaluation scheme. [7] Air
Force Response to ENMAX Protest Concerning Government's Salary Data, Agency
Report, Tab 2, attach. 22 at 3.
We also do not view the agency's conclusions as inconsistent with the review
anticipated by the FAR clause. By comparing an offeror's proposed hourly
rates with the BLS rates, the agency was able to reasonably conclude that
the offeror was not endangering successful performance by taking liberties
with professional compensation. In addition, the agency was also able to
conclude that the offeror understood that the complexity of the effort
requires paying appropriate professional rates. Little more was needed to
allow the agency to make a judgment about proposed prices--especially in
light of the competitive, and fixed-price nature of this procurement, and in
light of the fact that an offeror's understanding was also gauged by the
technical proposal. Accordingly, we conclude that the agency's price
evaluation was reasonable and was not an abuse of its discretion. See
Cardinal Scientific, Inc., supra.
RECOMMENDATION
We recommend that the agency reevaluate Carlisle's technical proposal taking
into account whether the proposal should be considered acceptable overall
given Carlisle's failure to show experience in [deleted], of the computing
environments identified under the third technical factor. If, at the
conclusion of this reevaluation, the contracting officer determines that
Carlisle's proposal is not technically acceptable, we recommend that the
agency terminate the award to Carlisle, and award to the offeror whose
proposal presents the best value under the evaluation scheme here.
We also recommend that the protester be reimbursed the reasonable costs of
filing and pursuing its protest, including attorneys fees. 4 C.F.R. sect.
21.8(d)(1). In accordance with 4 C.F.R. sect. 21.8(f)(1), ENMAX's certified
claim for such costs, detailing the time expended and the costs incurred,
must be submitted directly to the agency within 60 days after receipt of
this decision.
The protest is sustained.
Comptroller General
of the United States
Notes
1. The RFP also anticipated the payment of the contractor's travel costs and
certain other direct costs on a cost-reimbursable basis; government
estimates were used for the evaluation of these costs. RFP amend. 0002, sect.
M.B.4.c.
2. SEI/CMM is defined elsewhere in the RFP as the Software Engineering
Institute's Capability Maturity Model. RFP attach. 1, Statement of
Objectives at 1.
3. In its pleadings, ENMAX points to five areas where it argues that the Air
Force should have concluded that the Carlisle proposal was technically
unacceptable. We have reviewed each of these areas and--with the exception
of whether Carlisle demonstrates experience in each of the environments
identified above--we disagree. For example, while we agree with ENMAX that
Carlisle's experience managing the [deleted] WAN may not be sufficient to
establish experience with a "large scale (over 1 terabyte) data depot/data
warehouse project" (RFP, amend. 0002, sect. M.B.2 (factor 3)), we do not agree
that the RFP required such experience. Instead, the RFP specified that the
agency would evaluate experience in emerging technologies, and cited a large
scale data depot/data warehouse project as an example of such experience.
Id. Since Carlisle's proposal offered other examples of experience with
emerging technologies it was properly found acceptable.
4. We say "at best" because the protester has correctly observed that
Carlisle claims it has experience in client/server development in [deleted]
because the [deleted] "WAN is built in an [deleted] environment." Carlisle
Technical Proposal, supra, at 14. Instead, as ENMAX points out, [deleted] is
a database application that exists within the [deleted] WAN. As the WAN
administrator for a system that operates [deleted], it is not clear that
Carlisle has experience in software development using [deleted].
5. Because there are no contemporaneous explanations of these pass/fail
ratings, we are unable to discern whether the evaluators were willing to
accept experience in only [deleted] of the four areas (or perhaps, in only
[deleted] of the four areas), or whether they failed to recognize that
Carlisle's proposal did not show 2 years of experience in [deleted]. We have
no basis to conclude, on our own, that experience in all four areas was not
required by the RFP, given that the RFP set forth the four areas in the
conjunctive ("experience . . . in ORACLE, UNIX, NT, and Powerbuilder"), RFP
amend. 2, sect. M.B.2 (emphasis added), and that experience in these enumerated
environments could well be critical to the successful performance of the
contractor. Also, as stated above, while the checklist is not dispositive of
this issue, its [deleted] consideration of this experience on a pass/fail
basis provides further evidence that the agency evaluators considered
experience in all four areas an important requirement under the factor.
6. In this regard, we note that the desired experience described in the past
performance section of the RFP is broader than the experience set forth in
the technical evaluation factors. Compare RFP amend. 0002, sect. M.B.3.c.(1)
with RFP amend. 0002, sect. M.B.2, Factors 1-3.
7. In its comments on the agency report, ENMAX also argues that the BLS rate
is too low to hire sufficiently qualified personnel for these services.
ENMAX Comments, Mar. 15, 1999, at 10. Given the clear language in the
solicitation stating the agency's intent to use BLS data to review
professional compensation, any challenge to the use of the BLS data for this
purpose had to be raised prior to the initial closing date for submission of
proposals to be considered timely under our Regulations. 4 C.F.R.
sect. 21.2(a)(1) (1998).