TITLE: J&J Maintenance, Inc., B-284708.2; B-284708.3, June 5, 2000
BNUMBER: B-284708.2; B-284708.3
DATE: June 5, 2000
**********************************************************************
J&J Maintenance, Inc., B-284708.2; B-284708.3, June 5, 2000
Decision
Matter of: J&J Maintenance, Inc.
File: B-284708.2; B-284708.3
Date: June 5, 2000
James J. McCullough, Esq., and Catherine E. Pollack, Esq., Fried, Frank,
Harris, Shriver & Jacobson, for the protester.
Stephen G. Anderson, Esq., Baker, Donelson, Bearman & Caldwell, for Day &
Zimmermann Services, an intervenor.
Nicholas P. Retson, Esq., and Raymond M. Saunders, Esq., Department of the
Army, for the agency.
Peter A. Iannicelli, Esq., and Michael R. Golden, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision.
DIGEST
Protest of source selection decision is sustained where the record does not
establish the reasonableness of the evaluation or the cost/technical
tradeoff underlying the source selection.
DECISION
J&J Maintenance, Inc. protests the Army's award of a contract to Day &
Zimmermann Services (D&Z) pursuant to request for proposals (RFP) No.
DAKF11-98-R-0011. [1] The protester contends that the decision to award to
D&Z on the basis of its higher-priced proposal was flawed because the agency
unreasonably downgraded the protester's proposal and evaluated proposals
unequally, giving D&Z's proposal higher ratings in a number of areas even
though J&J's proposal contained similar or better features.
We sustain the protest.
Issued on June 8, 1998, the RFP solicited proposals for maintenance and
repair of family housing and for operation of a "self-help" center at Fort
Polk, Louisiana. RFP amend. 7, sect. C.1.1. The RFP contemplated award of a
fixed-price requirements contract to include a 1-month phase-in period, a
1-year basic contract period, and two optional 1-year periods. Id., sect. B.
The contract was to be awarded on the basis of best value. The evaluation
factors (and their relative weights) were quality ([deleted] percent), price
([deleted] percent), and past performance ([deleted] percent). RFP
amend. 0009, sect.sect. M.2, M.5; Agency Report, Tab Q, Source Selection Decision,
at 1. Within the quality factor, management and technical were equally
important subfactors. Management approach, staffing and qualifications,
subcontracting plan, and phase in/phase out were listed as elements of the
management subfactor. Resources, quality control/corrective action program,
and methodology were listed as elements of the technical subfactor. RFP
amend. 9, sect. M.6. Only quality and past performance were given point scores;
price was evaluated for reasonableness and balance. Id., sect. M.7. The RFP
required the quality portion of each offeror's proposal to be presented
orally, with slides, and indicated that the agency intended to award the
contract without discussions. RFP amend. 0007, sect. L.13.1.
Six firms, including J&J, the incumbent contractor, submitted proposals by
the February 10, 1999 closing. Each offeror made an oral presentation in
April, and revised price proposals were received in August. [2] After
evaluation, two proposals were rated unsatisfactory, while the four
acceptable offers were ranked as follows:
Offeror Quality/Past Adjectival Total Price
Performance Rating
Score
D&Z [deleted] [deleted] [deleted]
Offeror B [deleted] [deleted] [deleted]
Offeror C [deleted] [deleted] [deleted]
J&J [deleted] [deleted] [deleted]
Agency Report, Tab P, Combined Pre-Negotiation Objective/Price Negotiation
Memorandum, at 10-14. After reviewing the evaluations, the source selection
official (SSO) decided to award the contract on the basis of initial
proposals without discussions. Noting that D&Z's proposal had the highest
score under the quality factor, the SSO determined that D&Z's proposal
represented the best overall value to the government and that it would be
worth the additional expenditure to have D&Z, rather than J&J, perform the
work. Agency Report, Tab Q, Source Selection Decision Document, at 1, 3-4.
Accordingly, the contract was awarded to D&Z on February 14. After a
debriefing, J&J filed this protest.
Our Office will question an agency's evaluation and selection decisions only
where they violate a procurement statute or regulation, lack a reasonable
basis, or are inconsistent with the stated evaluation and selection scheme.
See B. Diaz Sanitation, Inc., B-283827, B-283828, Dec. 27, 1999, 2000 CPD
para. 4 at 6. We cannot perform a meaningful review of an agency's evaluation
and source selection if the agency record lacks adequate documentation to
support the evaluation of proposals and the selection decision. Biospherics
Inc., B-278508.4 et al., Oct. 6, 1998, 98-2 CPD para. 96 at 4.
In this regard, we rely primarily on the documentation required by the
Federal Acquisition Regulation (FAR). Specifically, FAR sect. 15.305(a) requires
agencies to document in the contract file the relative strengths,
deficiencies, significant weaknesses, and risks supporting proposal
evaluation. Of particular relevance to this case, FAR sect. 15.102(e) requires
the contracting officer to maintain a record of oral presentations to
document what the agency relied upon in making the source selection
decision. The source selection authority selects the method of recording the
oral presentations, and FAR sect. 15.102(e) gives the following examples:
videotaping, audio tape recording, written record, Government notes, copies
of offeror briefing slides or presentation notes. In addition, FAR sect. 15.308
requires that a source selection decision be documented, and that the
documentation include the rationale for any business judgments or tradeoffs
made or relied on.
Even where an agency has acted improperly (such as by failing to maintain
adequate documentation of its decisions), we will not sustain the protest
unless the improper action prejudiced the protester, that is, there is a
reasonable possibility that, but for the agency's actions, the protester
would have had a substantial chance of receiving the award.
McDonald-Bradley, B-270126, Feb. 8, 1996, 96-1 CPD para. 54 at 3; see
Statistica, Inc. v. Christopher, 102 F.3d 1577, 1581 (Fed. Cir. 1996). In
this protest, we are faced with both an evaluation that is unsupported as to
many elements and an undocumented cost/technical tradeoff, in which the
protester's and awardee's proposals were quite close in terms of their
technical ratings (D&Z's [deleted] points versus J&J's [deleted]) and their
price (approximately [deleted] for D&Z versus [deleted] for J&J). Because
the protester's price was lower, any but a de minimis change to the
technical rating of the protester's proposal creates the reasonable
possibility that the proposal would have been selected for award.
Accordingly, even though the evaluation defects discussed below relate to
only a small number of points (or even fractions of points), we view them,
collectively, as establishing that the agency's actions prejudiced the
protester.
In this procurement, the proposals addressing the quality factor consisted
of oral presentations, briefing slides, and resumes. No written technical
proposals were allowed. [3] While the RFP stated that the agency could
videotape and/or record the oral presentations, the agency did not do so.
RFP amend. 9, at L-6. In this case, the offerors' slides and the evaluators'
notes are the only record of what was included in the oral presentations. We
recognize the SSO's discretion to decide the method and level of detail of
the record of an oral presentation and that the use of offerors' slides and
government notes are two of several possible methods of documenting an oral
presentation under FAR sect. 15.102(e). Here, however, the slides and notes do
not present sufficient information to determine if the evaluations of J&J's
and D&Z's proposals were conducted in a reasonable and equal manner. The
slides provide only an outline--in most cases, the slides include general
headings only--and do not describe what was included in the approximately
2-hour oral presentations and question-and-answer sessions. [4] The
evaluators' notes are not summaries of the presentation but, for the most
part, are selective comments which are sketchy and provide almost no
elaboration, or even a description, of what was contained in the portion of
the oral presentation being commented upon by the evaluator.
In addition, the consensus evaluation ratings consist only of numerical
scores for each offer in each evaluation element, without any accompanying
narratives to show how the panel reconciled the divergent opinions of the
individual evaluators on any particular facet of an oral presentation or how
the panel achieved a consensus rating for any element of the evaluation.
There is no evidence in the consensus ratings sheets (or elsewhere in the
record) that the evaluators discussed the strengths and weaknesses of the
proposals or used any other method to reconcile their differences to arrive
at the consensus scores. Agency Report, Tabs I, J, Consensus Ratings for
Quality Proposals (D&Z and J&J, respectively).
We understand that oral presentations are an effective way to streamline the
source selection process and to enhance the agency's understanding of an
offeror's approach, and we do not believe that the FAR requirement for
documentation described above necessarily limits this flexibility. FAR
sect.sect. 15.102(e) and 15.308 do not require a particular method of establishing a
record of what was said by offerors during oral presentations or place an
excessive burden on contracting agencies in recording oral presentations.
Whatever method is chosen, FAR sect.sect. 15.102(e), 15-305(a) and 15-308 do
establish an obligation to provide a reasonably adequate record of such
presentations and the evaluation thereof. Such a record permits our Office
to perform a meaningful review of the agency's selection decision. Here, the
oral presentations constituted the offerors' entire technical proposals (the
only written portions of proposals related to past performance). The record
of the oral presentations and the evaluation is so sketchy, that we have no
means to determine, based upon the record before us, the reasonableness of
the agency's selection.
In discussing the following five examples, we necessarily examine the notes
and comments made by individual evaluators, because those notes (and the
offerors' slides) are the only record available to us.
First, regarding evaluation of the quality control/corrective action
program, D&Z's proposal received a [deleted] consensus evaluation score
([deleted] points) than J&J's proposal ([deleted] points) from the
evaluation panel. Agency Report, Tab I, Consensus Rating for Quality
Proposals (D&Z), at 1-2; Tab J, Consensus Rating for Quality Proposals
(J&J), at 1-2. J&J contends that the agency unfairly and inexplicably rated
D&Z's proposal higher than J&J's proposal in this area. Supplemental Protest
at 12, 23; Protester's Supplemental Comments at 10-12.
J&J points out that at least one evaluator made a very favorable comment
regarding J&J's quality control program, stating,"[deleted]" Id. at 10;
Agency Report, Tab H, Individual Evaluator Worksheets--J&J Quality Proposal,
at 18. J&J also points out that at least one evaluator made what J&J
interprets as less positive comments regarding D&Z's quality control
program, "[deleted]" and "[deleted]." Protester's Supplemental Comments
at 10; Agency Report, Tab G, Individual Evaluator Worksheets--D&Z Quality
Proposal, at 8. J&J believes that its quality control program was superior
to D&Z's and, therefore, should have been given at least as high a rating.
Protester's Supplemental Comments at 11.
The Army argues that the protester is merely disagreeing with the evaluation
and has not shown that the evaluation and selection should be overturned.
Agency Supplemental Report at 6. Neither the record nor the Army's
post-protest filings explain why J&J's proposal was rated lower than D&Z's
regarding the quality control programs, and the agency offers no way to
reconcile those ratings with the evaluators' comments noted by the
protester.
Second, regarding evaluation of management approach, the record shows that
D&Z's proposal received a [deleted] consensus evaluation score ([deleted]
points) than J&J's proposal ([deleted] points) from the evaluation panel.
Agency Report, Tab I, Consensus Rating for Quality Proposals (D&Z), at 1-2;
Tab J, Consensus Rating for Quality Proposals (J&J), at 1-2. J&J challenges
the agency's position that the protester's proposal contained a weakness
because J&J "[deleted]." [5] Agency Report, Tab T, Debriefing Slides, at 14.
J&J asserts that its oral presentation emphasized the use of [deleted]. [6]
Among other things, J&J points out that its [deleted] J&J states that it
demonstrated the [deleted] during its oral presentation, explained how the
[deleted], and answered questions about [deleted] at the end of its oral
presentation. Supplemental Protest at 10-11.
The record shows that one evaluator apparently downgraded J&J's proposal in
evaluating management approach, commenting that J&J "[deleted]," and that
the same evaluator, when evaluating the methodology for accomplishing tasks
element commented that J&J's "[deleted]." Agency Report, Tab H, Individual
Evaluator Worksheets--J&J Quality Proposal, at 30, 35. Another evaluator
appeared to be favorably impressed with J&J's [deleted], commenting,
"[deleted]." That same evaluator gave J&J's proposal a [deleted] score when
evaluating cost control measures; however, due to the change from a hybrid
type to a fixed-price type of contract, cost control was not considered in
the final evaluation. Id. at 22.
The Army contends that its evaluators were aware that the [deleted], and
points to several notes in the individual evaluators' scoresheets as
evidence of this fact. Agency Supplemental Report at 5-6. [7] The agency
does not otherwise justify the ratings assigned to the two competing
proposals, and the agency offers no way to reconcile those ratings with the
evaluators' comments noted by the protester.
A third example concerns J&J's approach to [deleted]." The agency perceived
this approach as a weakness of the proposal, stating: "[deleted]." Agency
Report, Tab T, Debriefing Slides, at 15.
J&J states that it explained during its oral presentation that it would
[deleted] J&J also states that it is currently successfully using [deleted].
The protester alleges that the Army ignored this information in the
evaluation. Supplemental Protest at 16.
The record shows that one evaluator, while recognizing that the idea had
merit, questioned whether J&J's [deleted] concept could be applied at Fort
Polk, because "[deleted]." That evaluator downgraded the proposal on the
evaluation of resources. Agency Report, Tab H, Individual Evaluator
Worksheets--J&J Quality Proposal, at 34. However, at least two other
evaluators made favorable comments regarding J&J's approach. One
specifically noted the concept as an advantage of the proposal, giving it a
[deleted] score in the methodology for accomplishing tasks element, stating:
"[deleted]." Id. at 27. Another also noted that, using this method, J&J
would be "[deleted]" and gave J&J [deleted] rating on methodology for
accomplishing tasks. Id. at 11, 18.
The Army responds only that its evaluators noted in several places that J&J
would use [deleted], so they clearly did not ignore J&J's oral presentation
on this point, but the Army states that, in the opinion of at least one
evaluator, the concept might not be as advantageous as J&J thinks. The Army
again contends that J&J is merely disagreeing with the Army's judgment and
has provided no basis for finding that judgment unreasonable. Agency
Supplemental Report at 8. The agency offers no other support for the ratings
given or explanation for the contrast between the evaluators' positive
comments and the consensus ratings.
The fourth instance concerns the agency perception that J&J's quality
proposal had a weakness because it was "[deleted]." Agency Report, Tab T,
Debriefing Slides, at 14. J&J states that it explained during its oral
presentation that it was [deleted]. J&J alleges that the Army ignored this
portion of its oral presentation and downgraded its proposal on the
resources element. Supplemental Protest at 15-16. J&J also asserts that the
Army has failed to explain why J&J's approach provides any greater risk than
D&Z's approach, since J&J contends that [deleted] , and therefore it should
have been rated at least equal to D&Z on this element. Protester's
Supplemental Comments at 13.
The record includes a reference to one evaluator criticizing J&J's having
its [deleted], and indicates that this criticism affected the evaluator's
evaluation of both the resources and methodology for accomplishing tasks
elements. Agency Report, Tab H, Individual Evaluator Worksheets--J&J Quality
Proposal, at 34, 35. The only other evaluator that specifically noted this
approach gave J&J much higher ratings on the same two elements. Id. at 19,
26, 27. The Army response is simply that its evaluators took note of the
protester's approach and determined it risky, and that the protester merely
disagrees with the evaluators. Agency Supplemental Report at 7-8.
As with the other examples, we cannot tell if the evaluation of this aspect
of J&J's oral presentation was reasonable, or if the agency treated J&J and
D&Z equally, because there is no adequate record of either J&J's or D&Z's
oral explanation of their [deleted], and because the evaluation record
provides no explanation of how the evaluators reconciled their different
opinions on this point.
A final example concerns D&Z's proposal receiving a higher rating than J&J's
on the phase in/phase out element of the management subfactor; D&Z's
proposal received a consensus score of [deleted] points, while J&J's
proposal received a consensus score of only [deleted] points. Agency Report,
Tab I, Consensus Rating for Quality Proposals (D&Z), at 1-2; Tab J,
Consensus Rating for Quality Proposals (J&J), at 1-2.
The protester asserts that, since it is the incumbent contractor, it has a
significant advantage over D&Z because J&J will not have to phase in. The
protester alleges that J&J and D&Z were not treated equally in the
evaluation and that its proposal should have been rated at least equal to
D&Z's. Second Supplemental Protest at 7; Protester's Supplemental Comments
at 22-23. In this regard, the offerors' slides show that both J&J and D&Z
addressed the phase in/phase out element. While D&Z's [deleted] on this
element includes [deleted], J&J's [deleted] also indicate that it [deleted].
Agency Report, Tab E, D&Z's Quality Proposal Slide No. 14; Tab F, J&J's
Quality Proposal Slides Nos. 13, 14.
[Deleted] evaluators made favorable comments about J&J's oral presentation
on this point and gave it relatively high scores. One evaluator wrote:
"[deleted]." Another commented: "[deleted]." [Deleted] evaluator stated:
"[deleted]." [Deleted] evaluator gave J&J [deleted] score and wrote:
"[deleted]." Agency Report, Tab H, Individual Evaluator Worksheets--J&J
Quality Proposal, at 5, 15, 24, 32.
The Army responds that, even though J&J was the incumbent and would not have
to go through a phase-in period, it was still required to address the
phase-out period. The Army states that D&Z addressed both phases of the
contract in sufficient detail to impress the evaluators and argues that J&J
merely disagrees with the evaluation. Army Second Supplemental Report at 4.
Again, because there is no adequate evaluation record, we have no way to
determine whether the evaluation was fair and reasonable. Further, since the
consensus evaluation worksheets include no narrative comments or
explanation, we cannot determine how the evaluators reconciled their
different opinions and evaluation comments on this element of the
evaluation.
As illustrated by these examples, we conclude that the Army failed to
maintain an adequate record of the oral presentations, as required by FAR
sect. 15.102(e), or adequate documentation of the relative strengths,
deficiencies, significant weaknesses, and risks supporting proposal
evaluation, as required by FAR sect. 15.305(a). In addition, there is no
meaningful evaluation record that supports, explains and reconciles the
individual evaluator scores and comments.
We also conclude that the source selection document does not meet the
requirements of FAR sect. 15.308. That provision requires that the selection
decision be documented and that the documentation include the rationale for
any tradeoffs made or relied on. Here, while the source selection document
lists several of the perceived advantages of D&Z's oral presentation, it
does not address any of the perceived advantages or disadvantages of J&J's
presentation. Instead, the SSO merely compares the overall scores of the two
offers in the quality and past performance evaluations and their total
prices. [8] Agency Report, Tab Q, Source Selection Decision, at 1-4. Because
of the SSO's apparently exclusive reliance on point differentials in the
selection decision, we conclude that the source selection decision, which
required a tradeoff between J&J's lower-rated, but lower-priced proposal and
D&Z's, failed to include documentation showing the rationale for the
tradeoff made between those two proposals. See Teltara Inc., B-280922,
Dec. 4, 1998, 98-2 CPD para. 124 at 3-4.
For the reasons set out above, we sustain the protest. We recommend that the
Army (1) allow offerors to make new oral presentations and to submit revised
written portions of their proposals; (2) make an adequate record of the
revised oral presentations and proposal evaluations; (3) conduct
discussions, if deemed necessary; and (4) make a new source selection
decision. If the agency determines that the contract should be awarded to an
offeror other than D&Z, the contract with D&Z should be terminated for the
convenience of the government and the contract awarded to the offeror
selected. [9] We also recommend that the protester be reimbursed the
reasonable costs of filing and pursuing the protest, including attorneys'
fees. 4 C.F.R. sect. 21.8(d)(1) (2000). The protester should submit its
certified claim for costs, detailing the time expended and the costs
incurred, directly to the contracting agency within 60 days after receipt of
this decision. 4 C.F.R. sect. 21.8(f)(1).
The protest is sustained.
Comptroller General
of the United States
Notes
1. Previously, J&J had protested the Army's failure to solicit and evaluate
revised technical proposals after amending the RFP's evaluation scheme.
Initial Protest at 6-9. We dismissed that earlier protest (reference No.
B-284708) as untimely on March 10, 2000.
2. Initially, the RFP contemplated award of a hybrid cost plus award
fee/fixed-price contract. RFP sect. L.9. After proposals were submitted and oral
presentations made for the quality proposal, the RFP was amended to state
that a fixed-price contract would be awarded; offerors were requested to
submit revised price proposals, but no revisions to technical proposals or
additional oral presentations were allowed. RFP amend. 7, at 2, 3.
3. J&J's also submitted an index and a 19-page executive summary as parts of
its proposal, but these items were not evaluated because they were neither
solicited nor authorized. The agency only evaluated each offeror's oral
presentation, slides, and resumes. Agency Report, Tab F, Quality Proposal
(Slides)--J&J; Memorandum from the Contracting Officer to Army Contract
Appeals at 1 (May 9, 2000).
4. Each offeror was allowed to present up to 30 slides during the oral
presentation. RFP amend. 7, sect. L.13.2(g)(3)
5. The RFP required the contractor to have a computerized system for
processing and recording work-related data and for generating contract
reports. RFP sect. C.4.6.1.
6. [deleted].
7. The Army fails to explain why, during the debriefing, J&J was told that
one of the perceived weaknesses of its proposal was its [deleted].
8. The SSO simply states that "J&J Maintenance, although offering the lowest
total price, does not represent the degree of expected performance specified
by the solicitation's weighting of [deleted] for the non-price factors. The
numerical differences in the Quality evaluations between Day & Zimmermann
and J&J Maintenance are significant, and not to be dismissed as trivial or
otherwise ‘virtually the same.' Day and Zimmermann presents an offer
rated as ‘[deleted],' while the lower-priced J&J, based on the point
differential, rates [deleted]." Agency Report, Tab QQ, Source Selection
Decision, at 3-4.
9. J&J contends that D&Z should be precluded from any further participation
in this procurement because the Army improperly released J&J's protected
initial protest letter, which contained some proprietary information, to D&Z
directly. Supplemental Protest at 7-9; Protester's Supplemental Comments
at 29-30. The contracting officer, who admits that the improper release
occurred, reports that the Army promptly retrieved the protected materials
from D&Z. Contracting Officer's Statement at 3. The Army also reports that
it gave J&J very similar information about D&Z's proposal (including D&Z's
price, relative quality score, and several of its perceived technical
advantages) during the debriefing, and concludes that J&J has therefore not
been prejudiced. Agency Supplemental Report at 3-4. While the release was
clearly improper, we see no basis to punish D&Z because of the Army's error,
particularly in light of the limited evidence of prejudice to J&J.