TITLE: Olympus Building Services, Inc., B-285351; B-285351.2, August 17, 2000
BNUMBER: B-285351; B-285351.2
DATE: August 17, 2000
**********************************************************************
Olympus Building Services, Inc., B-285351; B-285351.2, August 17, 2000
Decision
Matter of: Olympus Building Services, Inc.
File: B-285351; B-285351.2
Date: August 17, 2000
Ruth E. Ganister, Esq., Rosenthal and Ganister, for the protester.
Mark R. Warnick, Esq., General Services Administration, for the agency.
Jacqueline Maeder, Esq., and Paul I. Lieberman, Esq., Office of the General
Counsel, GAO, participated in the preparation of the decision.
DIGEST
Agency's proposal evaluation and resulting competitive range determination
were unreasonable where offerors' experience, staffing and authority
standards, and management approach/resources were evaluated by the
mechanical and otherwise unsupported application of undisclosed source
selection plan standards, which resulted in an irrational evaluation
outcome.
DECISION
Olympus Building Services, Inc. protests the exclusion of its proposal from
the competitive range under request for proposals (RFP) No.
GS06P-99-GXC-0021, issued by the General Services Administration (GSA) for
janitorial and related services at the Thomas F. Eagleton Courthouse, St.
Louis, Missouri. Olympus contends that the exclusion was the result of GSA's
improperly downgrading Olympus's proposal on the basis of the mechanical
application of arbitrary evaluation criteria that were not disclosed in the
RFP.
We sustain the protest.
The solicitation, issued September 16, 1999, provides for the best value
award of a fixed-price contract for a base year with four 1-year options.
RFP sect.sect. B.2, M.2.a.
The RFP identifies the following evaluation factors, listed in descending
order of importance (with the two subfactors of equal value):
1. Experience
2. Past Performance
3. Staffing and Authority Standards
4. Management Approach
a. Quality Control
b. Resources
RFP sect. M.2.b.
Under GSA's source selection plan (which was not disclosed in the RFP), each
factor and subfactor was assigned a possible point score and a weight, for a
possible total weighted score of 1,000, as indicated below:
Points Weight Total Possible
Score
Experience 50 10 500
Past Performance 50 6 300
Staffing and Authority 60 2 120
Standards
Management Approach
Quality Control 20 2 40
Resources 20 2 40
Total 1,000
Agency Report, Tab 3, Source Selection Plan, at 4; Contracting Officer's
Statement at 4.
The RFP advised offerors to address each of the above-listed evaluation
factors and subfactors in their proposals, and provided general guidelines,
discussed below, regarding the information that offerors were expected to
include under each factor. RFP sect. L.2.
[Deleted] proposals, including Olympus's, were received by the November 15
closing date. Contracting Officer's Statement at 2. The proposals were
evaluated and scored by the source selection evaluation board (SSEB), which
applied specific criteria contained in the source selection plan. Id. at
4-5. The five highest technically rated offers included two proposals rated
"excellent," with weighted point scores of [deleted], and three proposals
rated "poor," one with a weighted point score of [deleted] and two with
weighted point scores of [deleted]. [1] Id. at 5. Olympus's proposal was
assigned an overall "poor" rating based on its weighted point score
of [deleted]. Olympus's evaluated price was the second low of these five,
and was substantially lower than the prices of the two highest-scored
proposals. Because there was a "natural break" in technical scores between
the two highest-rated proposals and the next three proposals, the agency
determined to include only the two highest technically rated proposals in
the competitive range. Supplemental Agency Report at 4. By letter dated
April 19, GSA notified Olympus that its proposal was eliminated from the
competitive range and after a May 8 debriefing, Olympus filed this protest
with our Office. [2]
The protester argues that GSA improperly applied undisclosed standards in
evaluating the experience, staffing and authority standards, and management
approach factors which resulted in the downgrading and exclusion from the
competitive range of Olympus's proposal. Protester's Comments at 2-10.
Olympus points out that while the solicitation "generically" described what
GSA wished to see in a technical proposal, the source selection plan, which
was not disclosed to offerors, contained a specific evaluation methodology
that included "undisclosed benchmarks" that offerors had to include in
proposals in order to achieve high scores. Id. at 4-5. Olympus contends that
the agency was required to disclose these criteria because an offeror whose
proposal did not meet these undisclosed benchmarks had little, if any,
chance of being considered for award and, therefore, these benchmarks
effectively served as "pass/fail" or "accept/reject" criteria. Id. at 2.
TECHNICAL EVALUATION
Experience
The solicitation provided no specific explanation of how the technical
evaluation factors--including experience--would be evaluated. The sole
reference to experience in the solicitation (other than the above-cited
listing among the evaluation criteria) consists of the instructions in
section L of the RFP advising offerors of the requirement to submit a
minimum of three but not more than five references "of comparable type
facility at least 600,000 SF [square feet] of one (1) year duration within
the last five (5) years." RFP sect. L.2.A. Olympus listed references for five
facilities in its proposal. The SSEB evaluated each reference in accordance
with the source selection plan, which provided that each reference would be
rated on a scale of 0 to 10 points, with an "unacceptable" rating earning 0
to 4 points; a "poor" rating earning 5 to 6 points; a "good" rating earning
7 to 8 points; and an "excellent" rating earning 9 to 10 points. The plan
further provided that to be rated "good," a reference had to be judged
comparable and be 600,000 to 719,000 square feet. To be rated "excellent," a
reference had to be judged comparable and be 720,000 or more square feet.
Agency Report, Tab 3, Source Selection Plan, at 6. The scores for each
reference were totaled, and the total score was divided by the number of
references provided to determine the average score. The average score was
then multiplied by 5 (the maximum number of references) to determine the raw
score. Id. This raw score was multiplied by 10 to arrive at the weighted
score.
The agency determined that four of Olympus's five references were for
comparable facilities. Two of these references were for space that met or
exceeded the 720,000 square foot minimum for an "excellent" rating. These
two references were evaluated as "excellent" and each received 10 points.
Agency Report, Tab 5, Olympus's Scores, at 11-12, 19-20. A third comparable
reference was for a facility of approximately 700,000 square feet which,
because it was in the high end of the 600,000 to 719,000 square foot range
for a "good" rating, was evaluated as "good" and received 8 points. Id. at
23-24. The fourth reference was for a facility of 610,000 square feet. It
was evaluated as comparable space and, because it was in the low end of the
"good" range, was awarded 7 points. Id. at 15-16. Olympus's fifth reference
was evaluated as "not comparable" space because it was not a high rise
building but consisted of several buildings, the largest of which was
approximately 300,000 square feet. This reference was rated "unacceptable"
and awarded 1 point. Id. at 7-8. Based on these five scores and using the
above-explained formula that was detailed in the source selection plan,
Olympus's proposal received a total of 360 out of 500 possible points under
the experience factor. [3] Id. at 4.
Olympus argues that its proposal was unfairly downgraded, which essentially
eliminated it from consideration, as a result of the application of the
undisclosed source selection plan formula. Protester's Comments at 5-7. The
protester argues that offerors should have been advised that scores would be
averaged and that the agency was awarding its top rating and score only for
a building with 720,000 or more square feet, rather than merely indicating
the 600,000 square feet minimum requirement. Id. at 5. Olympus contends that
had it been informed of the evaluation scheme it would not have provided all
the references it included in its proposal. Id. at 6.
As a general rule, a contracting agency need not specifically identify the
subfactors comprising the evaluation criteria if the subfactors are
reasonably related to the stated criteria. FMS Corp., B-255191, Feb. 8,
1994, 94-1 CPD para. 182 at 6; KMS Fusion, Inc., B-242529, May 8, 1991, 91-1 CPD
para. 447 at 9. Agencies also need not disclose evaluation guidelines for rating
proposal features as more desirable or less desirable since agencies are not
required to inform offerors of their specific rating methodology. ABB Power
Generation, Inc., B-272681, B-272681.2, Oct. 25, 1996, 96-2 CPD para. 183 at 4.
However, the particular method of proposal evaluation utilized must provide
a rational basis for source selection and be consistent with the evaluation
criteria set forth in the solicitation. Brown & Root, Inc. and Perini
Corp.--A Joint Venture, B-270505.2, B-270505.3, Sept. 12, 1996, 96-2 CPD para.
143 at 9. Here, the undisclosed source selection plan methodology used by
GSA in computing scores for experience is unreasonable and does not provide
a rational basis for the resulting evaluations.
Specifically, under the solicitation, while an offeror's experience was to
be evaluated on the basis of at least three facility references, offerors
were invited to submit up to five references. Olympus submitted five
references; three of these--satisfying the minimum number required by the
solicitation--were scored as "excellent" or in the high "good" range because
they were deemed comparable and ranged from approximately 700,000 to 1.4
million square feet. However, the agency went on to essentially deduct
points for the two references that were for facilities substantially less
than 720,000 square feet or were judged "not comparable." Thus, Olympus lost
3 points for one of these references (which met the stated 600,000 square
feet criterion) and 9 points for the other. All of the scores were added and
averaged.
The application of this formula resulted in Olympus's proposal being
evaluated as of a lesser quality simply because, in addition to its
experience on three fully comparable buildings, Olympus also indicated
experience on two unnecessary, smaller, "non-comparable" buildings. In other
words, while Olympus's score for the five references was 360, had the
protester's score been based only on the three most comparable references,
it would have received an additional 105 points for a total score of 465.
[4] Crediting only these additional points would increase the protester's
total point score to 849, for an overall "good" rating. In our view, GSA's
mechanical formula for scoring the experience factor inappropriately
penalized Olympus for including essentially extra references, which resulted
in an arbitrary score that did not bear a meaningful relation to the
offeror's actual experience. SDA Inc., B-248528.2, Apr. 14, 1993, 93-1 CPD para.
320 at 8-12.
GSA argues that the fact that Olympus included one reference the agency
judged as "not comparable" warranted the reduced score because it
demonstrated that the protester "did not have the requisite knowledge to
discern comparable facilities . . . ." Supplemental Agency Report at 6.
GSA's position is that by submitting this one non-comparable reference,
Olympus "failed to demonstrate its understanding of the types of
requirements (e.g., staffing, supplies, materials and level of effort)
needed to clean the St. Louis Federal Courthouse." Id.
This argument is without merit. Although GSA could conceivably have included
a criterion that was tailored to evaluate offerors' understanding or ability
to discern the importance of experience with projects of size and
requirements similar to the current procurement, such a criterion was not
present here under the experience factor. The RFP identified this evaluation
criterion as an assessment of the offerors' experience, not as an ability to
identify comparable projects. Moreover, Olympus did provide four directly
comparable facility references--one more than the minimum required by the
solicitation. Nothing in GSA's contemporaneous record of its evaluation of
Olympus's proposal suggests a concern that Olympus "failed to demonstrate
its understanding of the types of requirements" needed under this
solicitation. Indeed, the only weaknesses noted in the rating sheets
concerning the reference GSA judged to be non-comparable were that the
reference was "not comparable type space," "not comparable to Class A
[office space]" and "multi type space, mixed use -- not comparable to the
courthouse." Agency Report, Tab 5, Olympus's Scores, at 7-8. [5] Further,
under the past performance factor, which is actually relevant to
understanding the requirements in that it evaluates actual performance,
Olympus's proposal received a score of 282 out of a possible 300, warranting
an assessment of "excellent." Id. at 4. The agency's argument in its protest
report that Olympus's listing of non-comparable experience reflected, or was
reasonably evaluated as, a lack of understanding of the requirements is thus
not supported by the record.
The agency also argues that it averaged the scores for the references
submitted because, without averaging, an offeror submitting five minimal or
marginal references would be rated higher than an offeror submitting three
superior comparable references. Supplemental Agency Report at 7. This
argument ignores the fact that simply averaging the three highest scored
references would accomplish the same result without penalizing an offeror
for including additional experience beyond the minimum requirement. In this
regard, GSA admits that its "formula may, under limited circumstances, yield
what appear to be inequitable results . . . ." Id. at 6. In sum, the
undisclosed evaluation methodology for scoring and rating experience was
irrational and resulted in the Olympus proposal being inappropriately
downgraded.
Staffing and Authority Standards
Olympus next challenges the agency's evaluation of its proposal under the
staffing and authority standards factor under which Olympus's proposal
received a weighted score of 70 (35 x 2) out of a possible total of 120
points. Again, the solicitation provided no explanation of how this factor
would be evaluated. Rather, the solicitation instructed each offeror to
include the project manager's resume and to define the "Project Manager's
level of authority in terms of monetary limits." RFP sect. L.2.C. The source
selection plan provided that a "good" rating and a score of 31 to 45 points
would be assigned for a project manager who, among other things, has a level
of authority from $2,500 to $24,999; an "excellent" rating and 46 to 60
points would be assigned a level of authority over $25,000. [6] Agency
Report, Tab 3, Source Selection Plan, at 8.
Olympus included a resume for its proposed project manager, listing
education, skills, knowledge and experience. The proposed project manager
had experience managing an 800,000 square foot building. Olympus also listed
the project manager's monetary level of authority as $2,500. Agency Report,
Tab 4, Initial Source Selection Report, at 6. In its evaluation of this
factor, GSA found that the level of authority was at the lower end of the
range and, deducting points primarily on this basis, assigned the protester
35 points and a "good" rating under this criterion. Id.
Olympus argues that the solicitation should have disclosed the required or
desired monetary levels of authority for the proposed project manager. The
protester contends that the solicitation merely advised offerors to submit
the monetary level of authority, but failed to explain that this subfactor
would "dramatically affect the points allocated to an offeror" on this
criterion or that fewer points will be awarded if the manager's monetary
level of authority was not equal to GSA's arbitrary "secret standard."
Protester's Comments at 7-8, 9.
GSA states that it required the project manager's level of monetary
authority to evaluate the manager's ability to quickly address and resolve
on-site problems and to respond to emergency situations. Supplemental Agency
Report at 8. The agency contends that the $2,500 to $24,999 standard "was
neither arbitrary nor capricious," but was "deemed appropriate for a project
of this size." Id. GSA argues that it did not impose mandatory monetary
limits because "it recognized offerors would ‘parrot' the solicitation
rather than submit an independently and intelligently arrived at offer." Id.
at 9. The agency argues that this standard allowed it "to rationally
distinguish and discriminate among competing proposals." Id.
In our view, GSA's application of the source selection plan used an
undisclosed criterion, the predetermined monetary level of authority for the
project manager, to downgrade proposals whose proposed monetary levels of
authority were less than that level. Contrary to the agency's claim, the
record shows that GSA mechanically applied this standard and did not, as it
asserts in its report, "rationally distinguish and discriminate among
competing proposals." The evaluation documents do not contain a single
instance where the agency analyzed an offeror's proposed monetary level of
authority to examine strengths and weaknesses of the proposed level within
the context of the proposed management approach, staffing, resources, or
experience of the offeror. The initial source selection report for all
proposals contains only brief factual statements, such as: "Contained
spending authority of $5,000," "Manager's spending authority was $2,500.00,"
and "Dollar authority is limited to $2,500.00," in the evaluation summaries
of this criterion. Agency Report, Tab 4, Initial Source Selection Report, at
2-4. While we do not question the agency's need to evaluate an offeror's
ability to quickly respond to problems and emergencies, which GSA explains
that this criterion was intended to measure, the solicitation failed to
indicate the agency's intention. Rather, GSA arbitrarily and mechanically
applied an undisclosed standard or estimate, which, in conjunction with the
other evaluation improprieties discussed in this decision, effectively
denied Olympus a fair opportunity to compete.
Management Approach
Finally, Olympus complains that the agency did not disclose the specific
estimate it used to evaluate the resources subfactor under management
approach. Olympus's proposal received 6 of 20 possible points, for a
weighted score of 12 of 40 possible points under this subfactor. As with the
other factors, the RFP here advised offerors as to the information they
should submit in their proposals without providing any explanation of the
evaluation methodology. Specifically, the RFP advised offerors that they
must "submit manhour requirements, with assignment of square footage per
hour to be cleaned by productive workers. Offeror submits a listing of
quantities and types of equipment, supplies and materials to be used in
performance of the contract." RFP sect. L.2.D.2. The source selection plan
specified that an offeror that listed quantities and types of equipment,
supplies and materials and proposes staff-hour requirements and square
footage assignments of 2,700 to 3,100 "are within industry standards" and
would be rated "good" and awarded 11 to 15 points. An offeror proposing
staff-hours requirements or square footage assignments in excess of 3,100
and with no listing of proposed quantities and types of equipment, supplies,
and materials would be rated "poor" and awarded 6 to 10 points. [7] Agency
Report, Tab 3, Source Selection Plan, at 9.
GSA found that the protester's proposal "[c]ontained listing of supplies and
equipment along with quantities. Square foot assignments proposed were
[deleted] per hour vs. industry standards of 2700-3100 per hour." [8] Agency
Report, Tab 4, Initial Source Selection Report, at 6. The agency also noted
that Olympus proposed fewer productive hours than the government estimate.
Id. The Olympus proposal was rated "poor" on this subfactor and awarded 6
points.
Olympus objects that offerors were not advised to support their staffing
levels and that the agency did not consider any reasons or circumstances of
a particular offeror "which may have justified a higher or lower per man
hour estimate of productivity." Protester's Comments at 11.
In response, GSA explains that this factor was utilized to determine if
"offerors understood the amount of resources and manpower needed to
successfully perform the contract requirements." Supplemental Agency Report
at 9. GSA explains that its estimate was based on its own analysis of the
cleaning requirements, results from the Cleaning Maintenance and Management
Survey, the Sanitary Supply Association Cleaning Time Estimator and the GSA
Custodial Management Desk Guide. Agency Report at 7-8. GSA argues that
agencies may rely on their own estimates of required staffing levels and
contends that if it advised offerors of the level of staffing needed, the
agency "would have no reasonable or rational way of determining whether the
offeror clearly had the knowledge, skills and ability to perform the
requisite services." Supplemental Agency Report at 10.
While an agency may rely on its own estimates of the staffing levels
necessary for satisfactory performance when negotiating a fixed-price
contract, we have found that it is improper for a contracting agency to
reject fixed-price proposals simply because the offerors' estimated
staff-hours differ significantly from the government's estimate, where the
government's estimate was not disclosed to the offerors and the agency
failed to conduct discussions with the offerors concerning the discrepancy.
Allied Cleaning Servs., Inc., B-237295, Feb. 14, 1990, 90-1 CPD para. 275 at
3-4. Because estimates are no more than informed guesses which may have
little or no applicability to other companies, an agency's absolute reliance
on estimates could have the effect of arbitrarily and unfairly penalizing an
innovative or unusually efficient firm. Id. at 4.
Here, GSA mechanically downgraded the protester's proposal because its
proposed staff-hours reflected an hourly cleaning rate that was not within
the range of the government's undisclosed estimate. In fact, the record
shows, in this regard, that several of the offerors' estimated staff hour
levels differed substantially from the government's estimate. Agency Report,
Tab 4, Initial Source Selection Report, at 2-15.
In our view, a reasonable evaluation of this factor would have taken into
consideration each offeror's individualized approach and would have
recognized the possibility that the hourly cleaning rate and the resulting
proposed staff-hours reasonably could vary. Here, as noted above, Olympus
asserts that its innovative approach using [deleted] provided efficiencies
which GSA did not consider, because its scoring methodology did not take
into account any differences in technical approach that might have offered
efficiencies. We offer no opinion on whether or not the protester's
contention about its innovative approach is correct; nor on this record
could we do so. However, in these circumstances, the agency's failure to
consider each offeror's approach by mechanical application of an undisclosed
hourly square footage rate was unreasonable.
In sum, the arbitrary and inflexible approach reflected by GSA's mechanical
evaluation method does not represent a rational evaluation of the offerors'
experience, staffing and authority standards, and management
approach/resources. Had the criteria been disclosed and applied in a
rational manner, Olympus's proposal, which was substantially lower-priced
than the two competitive range proposals, might well have received a
technical score above the 900 point "natural break" which the agency used to
determine which proposals to include in the competitive range.
Accordingly, we sustain the protest. We recommend that the agency amend the
solicitation to provide a rational and reasonably disclosed evaluation
scheme, request and evaluate revised proposals and make a new source
selection decision. If a proposal other than MMMM's is selected for award,
the agency should terminate the contract awarded to that firm. We also
recommend that the protester be reimbursed the reasonable costs of filing
and pursuing its protest, including attorneys' fees. 4 C.F.R. sect. 21.8(d)
(2000). The protester should submit its claim for costs, detailing and
certifying the time expended and cost incurred with the contracting agency
within 60 days after receipt of this decision. 4 C.F.R. sect. 21.8(f)(1).
The protest is sustained.
Robert P. Murphy
General Counsel
Notes
1. The source selection plan provided that proposals with total scores of
900 or more would be rated "excellent," 800 to 899 would be rated "good,"
500 to 799 would be rated "poor," and equal to or below 499 would be rated
"unacceptable." Agency Report, Tab 3, Source Selection Plan, at 18.
2. The agency held discussions with the two competitive range offerors on
April 24, and requested best and final offers (BAFO) by May 4. Contracting
Officer's Statement at 4 n.1. After evaluation of BAFOs, the decision to
award to Mitch Murch's Maintenance Management Company (MMMM) was made on May
16. Agency Report, Tab 30, Source Selection Final Report, at 4. By letter
dated June 30, GSA notified our Office that the agency had determined that
urgent and compelling circumstances required contract award; MMMM began to
perform in July.
3. Specifically, Olympus received scores of 10, 10, 8, 7 and 1, for a total
of 36 points. Using GSA's formula, the 36 points were divided by 5 (the
number of references) to compute Olympus's average score of 7.2. The average
was multiplied by 5 (the maximum number of references) to determine the raw
score of 36. This score was multiplied by 10 (the weight for the experience
factor) for a total of 360.
4. Using only Olympus's three highest rated references, 10, 10 and 8,
Olympus's total score would have been 28. The protester's average score
would then equal 9.3 (28 divided by 3) and its raw score would be 46.5 (9.3
times 5). The raw score times 10 (the weight of the experience factor)
equals 465.
5. Olympus argues that the agency improperly downgraded one reference
because it was for a facility comprised of many buildings rather than a
single building. The protester points to RFP language requiring references
"of comparable type facility" to support its position that references did
not have to be single buildings. RFP sect. L.2.A. The protester's interpretation
of the RFP is not reasonable because it renders meaningless the RFP's
numerous references to "building," for example, in its request for the
building name and the building gross square footage. RFP sect. L.2.B. Read as a
whole, we agree with the agency that the RFP did call for references of
comparable single building facilities.
6. The agency also evaluated whether the project manager's resume included
information regarding the manager's education, knowledge, experience and
skills and if the experience was within the past 5 years in a 600,000 or
larger square foot building. Agency Report, Tab 3, Source Selection Plan, at
7-8.
7. An offeror would be awarded an "excellent" rating and 16 to 20 points if
it listed quantities and types of equipment, supplies and materials,
proposed square footage assignment within the 2,700 to 3,100 industry
standard and submitted proposed staff-hour requirements and hours broken out
by daily and periodic requirements. Agency Report, Tab 3, Source Selection
Plan, at 9.
8. Olympus's proposal did not state a figure for the square footage to be
cleaned in one hour, so the agency used the [deleted] square foot rate
submitted for the protester's [deleted] as the average. Contracting
Officer's Statement at 7. In its protest, Olympus argues that it had
proposed to clean [deleted] square feet per hour, which was feasible because
of its innovative use of [deleted], whose work was augmented [deleted], and
that this figure could be calculated from its proposal documents. In
responding to the protest, the agency reevaluated the protester's proposal
and discovered that the [deleted] square foot figure could be derived from
the proposal, but determined that under the source selection plan criteria,
this would not change Olympus's proposal's score on this criterion. Id.