BNUMBER: B-280595
DATE: October 23, 1998
TITLE: Keane Federal Systems, Inc., B-280595, October 23, 1998
**********************************************************************
DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective
Order. This redacted version has been approved for public release.
Matter of:Keane Federal Systems, Inc.
File: B-280595
Date:October 23, 1998
Jason I. Hewitt, Esq., Eric J. Marcotte, Esq., and Carl J.
Peckinpaugh, Esq., Winston & Strawn, for the protester.
Jessica C. Abrahams, Esq., and Lynda Troutman O'Sullivan, Esq., Miller
& Chevalier, and Helaine G. Elderkin, Esq., Computer Sciences
Corporation, for Computer Sciences Corporation; Mark H. Neblett, Esq.,
Philip J. Davis, Esq., and Rand L. Allen, Esq., Wiley, Rein &
Fielding, and Joseph T. Casey, Esq., Electronic Data Systems
Corporation, for Electronic Data Systems Corporation; and Carla D.
Craft, Esq., Paul N. Wengert, Esq., Lars E. Anderson, Esq., and Thomas
J. Madden, Esq., Venable, Baetjer, Howard & Civiletti, and Bucky
Mansuy, Esq., Lockheed Martin Corporation, for Lockheed Martin
Technical Services, Inc., intervenors.
Robert J. Brown, Esq., and Michael K. Cameron, Esq., Department of
Justice, for the agency.
Tania Calhoun, Esq., and Christine S. Melody, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision.
DIGEST
1. Protest that contracting agency improperly evaluated protester's
proposal with respect to management and cost issues is denied where
the record shows that the evaluation was reasonable and consistent
with the evaluation criteria.
2. Protest that contracting agency's best value analysis improperly
distorted the solicitation's evaluation scheme and resulted in an
irrational award decision is denied where, in accordance with the
solicitation's terms, the agency's best value analysis consisted of
the development of discriminators between competing proposals, the
weighing of those discriminators between competing proposals, and the
selection for award of those proposals based upon the quantity,
significance, and applicability of the superior features proposed.
DECISION
Keane Federal Systems, Inc. protests the award of contracts to
Computer Sciences Corporation (CSC), Electronic Data Systems
Corporation (EDS), and Lockheed Martin Technical Services, Inc. (LM)
under request for proposals (RFP) No. HQ- 97-13, issued by the
Department of Justice, Immigration and Naturalization Service (INS),
in support of its Service Technology Alliance Resources (STARS)
program, under which INS intends to meet all of its information
technology needs into the next century. Keane primarily contends that
INS improperly evaluated its proposal with respect to management and
cost and conducted an irrational and arbitrary best value analysis
that improperly distorted the solicitation's evaluation scheme.
We deny the protest.
BACKGROUND
The STARS program is the result of INS's desire to form an alliance
with information technology contractors to obtain a full range of
technological solutions to support the agency in achieving its mission
objectives and information technology needs into the 21st century.
RFP sec. C.1.1. This solicitation, issued on September 16, 1997, is the
vehicle by which INS planned to obtain the services of its STARS
contractors.
To fulfill the STARS program strategy INS intended to award one
contract for system management and integration services; three hybrid,
indefinite-delivery performance contracts[1] to three other firms for
systems development, implementation, operations, and maintenance
services; and one contract to a fifth firm for independent
verification and validation services. RFP sec. C.1.2. All work to be
performed under the STARS contracts will be assigned by task orders
issued over 1 base year, with up to 4 option years available. RFP sec.
B, C.1.2. This protest concerns the award of the performance
contracts.
The performance contractors will be responsible for, among other
things, ongoing systems engineering and integration to support
implementation of a comprehensive information systems architecture;
systems maintenance and sustaining engineering; systems management;
designing, developing, installing, operating, and maintaining existing
and planned communications and electronic systems; and assisting INS
with system definition and providing development, installation,
operation, maintenance of existing and planned end-user automated data
processing support systems or services, user training, and video
services. RFP sec. C.4.
Since software development is the heart of the STARS program, the RFP
required offerors to possess a documented Software Engineering
Institute (SEI) Software Capability Maturity Model (CMM) at level 2 or
higher.[2] Offerors not meeting this mandatory requirement were not
eligible for evaluation or award--their proposals were to be rejected.
Offerors with documented Software CMM levels higher than 2 were to be
given credit in the overall best value analysis.[3] RFP sec. L.7.2.1.,
M.2.
Awards were to be made to the offerors whose proposals represented the
greatest overall value to the government, cost and other factors
considered. RFP sec. M.3.2. The solicitation set forth five evaluation
factors: management, experience and past performance, technical,
cost, and other business factors. RFP sec. M.4.
Within the management factor, the contract management subfactor was
more important than the corporate management subfactor; the RFP also
set forth numerous elements to be considered under each management
subfactor. RFP sec. M.4.1(a), M.4.3.1. The experience and past
performance factor consisted of two equally important subfactors,
experience and past performance. RFP sec. M.4.1(a). Within the
technical factor, each sample task response was considered to be
equally important. RFP sec. M.4.1(b). All of these factors and their
components were to be adjectivally rated. RFP sec. M.4.2.
The management and experience and past performance factors were
considered to be equally important, and these two factors were
significantly more important than the technical factor. RFP sec.
M.4.1(a). Collectively, the management, experience and past
performance, and technical factors represented the performance
capability factors, and the performance capability factors were
significantly more important than the cost factor. RFP sec. M.4,
M.4.1(c).
Costs and other business factors were not to be adjectivally rated.
Information in the cost volumes was to be evaluated to determine the
total proposed contract cost, cost realism, reasonableness, and cost
risk. RFP sec. M.4.4. The cost factor was significantly more important
than the other business factors. RFP sec. M.4.1(d). The other business
factors were to be evaluated and considered and were also critical to
INS's responsibility determination. RFP sec. M.4.4.2.
Offerors' management and experience and past performance volumes were
to be evaluated by the management, experience & past performance
evaluation committee (MEPPEC). Offerors' technical volumes were to be
evaluated by the technical evaluation committee (TEC). Finally,
offerors' cost and other business factors volumes were to be evaluated
by the business evaluation committee (BEC). At the conclusion of the
evaluation process, each committee was to submit its findings to the
source selection advisory council (SSAC), which was to review the
committees' findings; conduct comparative analyses of the proposals;
and present its recommendations to the source selection authority
(SSA). The SSA was to make the final source selection decision. RFP sec.
M.3.1.
The determination of greatest value was to be made by comparing the
differences in the value of the management, experience and past
performance, and technical factors with differences in the costs
proposed. INS would not make an award at a significantly higher
overall cost to achieve only slightly superior performance capability.
INS planned to make its greatest value assessment through the
development of trade-off analyses and other analytic studies that
involve the assessment of benefits of superior performance capability
features versus the added cost. RFP sec. M.3.3.
INS received proposals from five offerors, each consisting of a prime
contractor and numerous subcontractors. The contracting officer
included all five proposals in the competitive range based upon the
initial evaluation. INS conducted extensive written and oral
discussions with each offeror and received best and final offers
(BAFO) in April 1998. Each evaluation committee reviewed the BAFOs
and revised their final reports as appropriate. The BEC listed both
the offerors' proposed costs, corrected for any errors, and their most
probable costs, inclusive of the BEC's cost realism adjustments.
The SSAC was given the committees' final reports, each of which
consisted of detailed documentation of the strengths and weaknesses of
each proposal in connection with each evaluation factor, subfactor,
and element. The SSAC voted unanimously to accept the findings of the
committees. SSAC Recommendation
at 27. The SSAC decided to consider offerors' proposed corrected
costs as their evaluated costs since it believed the BEC's cost
realism adjustments reflected upper
bounds on quantifiable risks rather than corrections for
unrealistically low cost proposals.[4] Id. The final relevant
evaluation results were as follows:[5]
CSC EDS LM Keane
Management Outstanding Outstanding Outstanding Good
Contract ManagementOutstandingOutstandingOutstanding Good
Corporate ManagementOutstandingOutstandingOutstandingOutstanding
Experience and Past PerformanceGoodGood Good Outstanding
Experience Outstanding Outstanding Outstanding Outstanding
Past Performance Good Good Good Outstanding
Technical Outstanding Good Acceptable Outstanding
Proposed Costs (Millions)
$272
$296
$244
$283
The SSAC used various methods to rank the offerors but achieved
different results depending on the method used. In any event, the
SSAC decided that a further ranking of proposals was unnecessary since
it required a detailed best value analysis. Id. at 29. The best
value working group (BVWG) was appointed by the SSAC Chair to conduct
this analysis. As discussed in detail below, the BVWG began its task
by identifying 53 potential discriminators based upon a review of the
evaluation committee reports. BVWG Report, Exhibit 1. The BVWG next
reviewed each proposal to ascertain which of the potential
discriminators were actual discriminators--superior features and
risks--and arrived at a list of 12 superior features and 5 risks.
BVWG Report, Exhibit 2.
After undertaking a detailed review of each proposal, the BVWG
determined that CSC's proposal offered 11 superior features and 1
risk; EDS's proposal offered
9 superior features and 1 risk; LM's proposal offered 6 superior
features and 1 risk; and Keane's proposal offered 5 superior features
and 2 risks. SSAC Recommendation at 31. The BVWG compared these
results with the offerors' proposed costs and reached some
conclusions.
First, Keane's proposal did not present a sufficient number of
features to warrant paying a $39 million premium (the difference
between Keane and LM's proposed costs) for slightly fewer features and
added cost risk. Second, LM's proposal offered a significant number
of superior features, was the lowest cost offer, and had very little
cost and performance risk. Third, CSC was the highest-ranked offer
based on the performance capability factors and its proposal, ranked
third based on cost, offered a significantly greater number of
superior features than any other offeror with little cost or
performance risks. Fourth, EDS's proposal offered superior features
that were virtually identical to those proposed by CSC, its proposed
costs were $24 million higher than CSC's, and the superior features
found in EDS's proposal significantly exceeded those found in Keane's
proposal with a cost that was slightly higher. Id. at 32.
The BVWG recommended that awards be made to CSC, EDS, and LM, and the
SSAC adopted this recommendation. Id. at 33-35. The SSA decided to
award the contracts to CSC at a maximum value of $817.9 million; to
EDS at a maximum value of $890.2 million; and to LM at a maximum value
of $734.1 million.[6] In his Source Selection Decision, the SSA
stated:
The CSC proposal offers the most performance capability, third
lowest cost, and represents the greatest value to the Government
of all offerors.
The EDS proposal offers high performance capability and a
significant number of superior features. Although the EDS
proposal carries a cost premium over the unsuccessful offeror's
proposals, it represents the second greatest value to the
Government.
The [LM] proposal offers high performance capability, a number of
superior features, and is the lowest cost proposal. As such it
represents the third greatest value to the Government.
In making this determination, I have considered the rankings,
ratings, and recommendations of the [SSAC] and evaluation
committees and have used them as the basis to arrive at my own
independent decision.
Keane filed this protest after it received its debriefing. Keane
primarily argues that INS improperly evaluated its management volume
with respect to the contract management subfactor; improperly
evaluated its cost information in assessing it several cost risks; and
conducted an irrational and arbitrary best value analysis that
improperly distorted the evaluation scheme.
EVALUATION OF MANAGEMENT VOLUME
Evaluating the relative merits of competing proposals is a matter
within the discretion of the contracting agency since the agency is
responsible for defining its needs and the best method of
accommodating them, and must bear the burden resulting from a
defective evaluation. Advanced Tech. and Research Corp.,
B-257451.2, Dec. 9, 1994, 94-2 CPD para. 230 at 3. Where an evaluation is
challenged, we will examine the agency's evaluation to ensure that it
was reasonable and consistent with the evaluation criteria.
Professional Software Eng'g, Inc., B-272820, Oct. 30, 1996, 96-2 CPD para.
193 at 4. A protester's disagreement with the agency's conclusion
does not itself establish that the agency acted unreasonably. Id.
Keane argues that if EDS's proposal was rated outstanding under the
management factor its proposal should have received the same rating.
Keane points to the fact that EDS's proposal was rated "outstanding"
under the management factor even though it had two fewer major
strengths and four fewer minor strengths than did Keane's proposal
under the contract management subfactor, and the fact that Keane's
proposal was rated "outstanding" under the corporate management
subfactor.
As an initial matter, as discussed below, the source selection here
did not depend upon these adjectival ratings or the mere number of
strengths or weaknesses found in given proposals. Keane's narrow
focus on these matters is meaningless because it fails to consider the
underlying nature of the strengths found in each proposal and their
relative importance to the evaluation as a whole. See Innovative
Logistics Techniques, Inc., B-275786.2, Apr. 2, 1997, 97-1 CPD para. 144
at 9. Keane's only substantive challenge to the evaluation is its
assertion that its proposal and EDS's proposal were treated unequally,
citing several examples. We address three of these.
Keane complains that INS evaluated EDS's proposal as having a major
strength for "High Visibility of STARS Program within EDS" because its
proposed STARS program is four organizational levels from the EDS
chairman, while Keane's proposed STARS program, three organizational
levels from its chairman, was not evaluated as a strength. The record
shows that EDS's evaluated strength was not attributable to the mere
placement of the STARS program within its organization, but to many
additional factors as well. Final MEPPEC Report at II-41-42 and
Appendix 3 at D-2. As a result, we have no basis to question INS's
evaluation.
Keane incorrectly contends that the MEPPEC's final report erroneously
indicated that EDS's proposal was rated as having two major strengths
for staffing when only one major strength appears in the backup
documentation. The record shows that, from the beginning, EDS's
proposal was considered to have two major strengths for staffing.
Final MEPPEC Report, Appendix 2 at D-18 and Appendix 3 at D-12.
Keane contends that the MEPPEC improperly rated EDS's proposal as
having a minor strength for its goal of awarding 16 percent of its
annual revenue to small businesses and its commitment to a minimum
set-aside of 10 percent, whereas Keane's proposal was rated as having
the same minor strength for its proposal to allocate 28.1 percent of
its work to small businesses and to a minimum set-aside of 12.5
percent. While there are differences between the two proposals, Keane
has provided no reason for us to conclude that these differences
merited a materially different assessment.
EVALUATION OF COST VOLUME
The BEC's evaluation of Keane's proposal disclosed three significant
cost risks associated with the status of its purchasing system, its
proposed indirect rates, and its cost accounting standards (CAS)
status. Keane argues that each of these cost risks is unreasonable.
We address each in turn.
One of the other business factors was "approval of business systems."
Offerors were required to state whether all of their business systems
which required government approval, such as their purchasing systems,
were approved or, if not, to explain the status of their approval.
RFP sec. L.7.7.6.6. In this regard, contractors with approved purchasing
systems generally need not obtain administrative contracting officer
(ACO) consent for cost-reimbursement subcontracts, while contractors
without approved purchasing systems generally do need to obtain such
consent. Federal Acquisition Regulation (FAR) sec. 44.201-2. In
considering whether to grant consent, the ACO is to consider numerous
factors associated with the way the prime contractor selected the
subcontractor; "particularly careful and thorough consideration" is
necessary when the prime contractor's purchasing system is inadequate
or when subcontracts are proposed on a cost-reimbursement basis.
FAR sec. 44.202-2.
To assist the BEC with its cost proposal evaluation, offerors were
required to furnish a copy of each cost proposal and the analysis made
thereof for each subcontract valued at $100,000 or more. Offerors
were required to have each of these subcontractors submit a full set
of cost forms documenting various elements of their proposed costs.
RFP sec. L.7.7.5.1.
Keane's initial proposal did not advise INS whether its purchasing
system was approved and, in addition, the BEC found that it had not
been given sufficient cost information to evaluate Keane's
subcontractors. During discussions, Keane was asked to address the
status of its purchasing system and to provide sufficient information
on its proposed subcontractors to allow for their evaluation. Keane's
answers did not specifically address the status of its purchasing
system and provided only general information concerning its
subcontractors. Keane's Response to Deficiency Reports BEC-DR-346,
BEC-DR-410.
Amendment No. 0005 revised RFP sec. L.7.7.5.3. to specifically instruct
offerors that:
All prime contractors must show, or as a minimum explain what
analysis (cost analysis, price analysis, etc.) was performed on
subcontractor proposals to determine the reasonableness of the
subcontractor pricing. If the prime contractor has an approved
purchasing system, the prime must affirmatively state whether
they did or did not adhere to their own purchasing system
requirements when evaluating the subcontractors proposals.
Keane's BAFO provided some additional explanation of its subcontractor
agreements and arrangements, BAFO Cost Volume at V-B-14 through
V-B-22, but the BEC concluded that these general and unsupported
statements failed to meet the requirements of RFP sec. L.7.7.5.3. Keane
also advised that it did not have an approved purchasing system. See
BAFO Other Business Factors Volume at VI-17.
The BEC found that Keane was the only offeror that did not provide
required documentation of the process by which it selected
subcontractors in accordance with FAR Part 44, and that its analysis
was not sufficient to permit the contracting officer to authorize
approval of subcontracting at the time of award. Final BEC Report at
2. This lack of information, which might not have been necessary had
Keane possessed an approved purchasing system, represented significant
risk, particularly since Keane planned to subcontract out 40 percent
of the contract effort. The contracting officer would have to approve
each subcontractor until Keane's purchasing system was reviewed and
approved. Id. at 18-19. Keane's assertion that it identified each
subcontractor for the government to evaluate, making its lack of an
approved purchasing system irrelevant, is wholly unresponsive to INS's
concerns, and we conclude that the BEC's assessment was reasonable.
Next, in evaluating Keane's initial proposal the BEC was concerned
that the firm's proposed indirect rates declined over the period of
performance and were both significantly lower than historical rates
and unsupported. During discussions, Keane was asked to explain and
support its proposed declining rates. Keane generally asserted that
it had experienced declining rates due to increased business volume
and that these rates would continue to decline with the addition of
STARS business. Keane's Responses to Clarification Requests
BEC-CR-623, BEC-CR-620. One of Keane's subcontractors was also
advised, during discussions, that its proposed indirect rates were
unrealistically low and unsupported, and its response was similarly
vague and unsupported. Id., BEC-CR-167, BEC-CR-445, BEC-CR-166,
BEC-CR-628, BEC-CR-146. These explanations did not persuade INS that
either firm's proposed indirect rates were reliable, and the BEC
repeated its concerns during oral discussions.
Keane's BAFO provided no substantive support for its proposed indirect
rates but only generalized statements about its projected growth which
Keane itself termed "optimistic." BAFO Cost Volume at V-B-5 through
V-B-7. Accordingly, the BEC concluded that Keane's indirect rates,
and those of its subcontractor, represented a significant cost risk.
The issuance of any competitive task order on a cost-reimbursement
basis would require that these indirect rates be routinely checked
with the Defense Contract Audit Agency to enable the contracting
specialist to monitor trends. Final BEC Report at 18-20.
Keane's argument that the BEC mechanically adjusted the firm's--and
its subcontractor's--indirect rates without considering the positive
impact that receiving the STARS contract would have on lowering their
historical indirect rates is simply untrue and fails to address INS's
concerns regarding the lack of support for these proposed indirect
rates. Given the unsupported discrepancy between Keane's actual
indirect rates and its proposed indirect rates, we see nothing
unreasonable about INS's evaluation. See Booz-Allen & Hamilton, Inc.,
B-275934.2, May 29, 1997, 97-1 CPD para. 222 at 5.
Finally, since the performance contracts are subject to CAS coverage,
FAR Appendix sec. 9901.306, the RFP asked offerors to submit their CAS
Disclosure Statements--written descriptions of their cost accounting
practices and procedures-- as part of their cost volumes. RFP sec.
L.7.7.5.3., K.3.21. In addition, approval of an offeror's accounting
system was one of the business systems specified under the "approval
of business systems" factor discussed above. RFP sec. L.7.7.6.6.,
M.4.4.2.6. In its initial proposal Keane advised that it had not
submitted a CAS Disclosure Statement because it had not previously
been required to do so, and that it would submit the required
statement in 1998. Initial Proposal Volume VI at VI-17.
Business units selected to receive a CAS-covered contract of $25
million or more are required to submit a CAS Disclosure Statement
before award. FAR Appendix sec. 9903.202-1(b)(1). As a general matter,
contracting officers are not to award a CAS-covered contract until the
ACO has made a written determination that a required Disclosure
Statement is adequate. FAR sec. 30.202-6(b). Accordingly, during
discussions, the BEC asked Keane about its CAS compliance status and
was advised that the firm would submit its disclosure statement
concurrent with award of this contract. Keane's Response to
Clarification Request BEC-CR-619. The BEC repeated its inquiry during
oral discussions, and expressed concern about the time it takes to
review such statements.
The BEC gave Keane's proposal a passing rating under the approval of
business systems factor since it had not previously been required to
submit a CAS Disclosure Statement.[7] However, since the contracting
officer may generally not award a
CAS-covered contract until the ACO has determined that the required
Disclosure Statement is adequate, which it could not do here, and
since the performance contracts contemplated the issuance of
high-dollar value cost-reimbursement task orders, the BEC found the
fact that Keane did not have approval of its cost accounting standards
to be a significant risk because the government did not know if its
cost accounting practices would meet government requirements. Final
BEC Report at 18-19. Keane's argument that it is CAS-compliant does
not minimize the agency's concerns about whether it actually is
CAS-compliant, or its concern regarding the firm's ability to obtain a
review of its CAS Disclosure Statement in a timely fashion. Under the
circumstances, we cannot conclude that INS's concerns here were
unreasonable.
While Keane is correct that the awardees' proposals evidence various
CAS
non-compliance issues, its implication that their proposals should
have been found to present cost risks as well is unsupported. Keane
is incorrect when it argues that INS ignored these matters with
respect to the awardees. The record shows that the agency reviewed
the issues, conducted discussions to gain more information, and
ultimately concluded that the risks, if any, were not significant.
Keane has given us no reason to think that the risks, if any,
presented by these proposals were equivalent to the risk presented by
its own.
As for Keane's assertion that these areas of risk were undisclosed
evaluation criteria, the consideration of risk involved in an
offeror's proposed approach is inherent in the evaluation of
proposals. DIGICON Corp., B-275060, B-275060.2,
Jan. 21, 1997, 97-1 CPD para. 64 at 4. Moreover, the RFP here
specifically informed offerors that the cost risks associated with
their proposals would be evaluated, and that their CAS status, methods
of selecting subcontractors, purchasing systems, and indirect rates
would be reviewed. Thus, the agency's consideration of risk here was
not improper. Id. at 5. Finally, Keane's argument that INS failed to
conduct meaningful discussions on these issues is without support.
Aside from the fact that applicable regulations and RFP provisions put
Keane on notice of most of these matters, as noted above, the record
is replete with evidence that INS conducted meaningful discussions in
each of these areas.[8]
BEST VALUE ANALYSIS AND SOURCE SELECTION DECISION
INS took particular care to inform offerors how their proposals were
to be evaluated against the RFP's terms and how their proposals were
to be selected for award. The RFP put offerors on notice that a set
of evaluation committees was to review, analyze, evaluate, and rate
the various volumes of their proposals, gave them the evaluation
roadmap that each committee planned to follow, and instructed them
that the committees' findings were to be given to the SSAC. The RFP
put offerors on notice that the SSAC's role was to review the findings
of the evaluation committees, conduct comparative analyses of the
proposals, and present its recommendations to the SSA, who would make
the final source selection decision.
In this regard, offerors were told that awards were to be made to
those offerors whose proposals represented the greatest overall value
to the government, cost and other factors considered. Section M.3.3.
of the RFP told offerors precisely how the "greatest overall value"
was to be determined:
The determination of greatest value will be made by comparing the
differences in the value of the Management, Experience & Past
Performance, and Technical Factors with differences in the costs
proposed. These evaluation factors, when combined, are
significantly more important than cost. However, the Government
will not make an award at a significantly higher overall cost to
the Government to achieve only slightly superior performance
capability. The Government will make this assessment through the
development of trade-off analyses and other analytic studies that
involve the assessment of benefits of superior performance
capability features (e.g., economic benefits clearly attributable
to increased INS productivity, probability of successful contract
performance, and/or unique and innovative approaches or
capabilities) versus the added cost. Overall cost to the
Government may become the ultimate determining factor for award
of the contracts as proposals become more equal based on the
other factors. The degree of equality between the Offerors'
proposals will be measured by the quantity, significance, and
applicability of the superior features proposed.
The record shows that INS followed the RFP's instructions.
Each evaluation committee reviewed, analyzed, evaluated, and rated the
proposals of each offeror. The evidence of their labor is found in
hundreds of pages of detailed documentation which contains
factor-by-factor, subfactor-by-subfactor, and element-by-element
analysis of how each proposal measured up to the solicitation's
requirements. Each evaluation report culminated in a set of
adjectival ratings or adjusted costs, as applicable. The SSAC
reviewed these findings and convened the BVWG to assist with the
comparative analysis of proposals. The process followed for the best
value analysis and source selection is set forth in the BVWG's Report
and the SSAC's Recommendation, and was further illuminated by the
MEPPEC Chair (a BVWG member), the SSAC Chair, and the SSA at a
hearing.
The BVWG's goal was to identify the differences--discriminators--betwe
en proposals that would be important to the INS mission and/or program
objectives, and that would be expected to affect INS's operation under
the contract. SSAC Final Recommendation at 30. To ascertain the
universe of potential discriminators, the BVWG prepared a matrix which
listed each evaluation factor, subfactor, and element as set forth in
section M of the RFP, and contained a column for each offeror. The
BVWG reviewed each committee's final evaluation report,
factor-by-factor, to identify superior features and risks--potential
discriminators--found in the offerors' proposals. The BVWG inserted
each potential discriminator into the matrix under the appropriate
evaluation factor, and inserted a plus (for a superior feature) or a
minus (for a risk) in the column of each firm whose proposal contained
the potential discriminator. Id.; BVWG Exhibit 1; Hearing Transcript
(Tr.) at 26-34, 70-71. The BVWG identified 53 potential
discriminators.
Next, the BVWG set out to determine which of these potential
discriminators were actual discriminators--features that would have a
significant impact on INS's mission productivity, effectiveness, and
cost. The BVWG reviewed each proposal, factor-by-factor, to ensure
that it fully understood what was offered and why it did or did not
add value. The evidence of their labor is found in more than 100
pages of documented analysis which links each discriminator to an RFP
requirement, discusses the relevant proposal feature, and analyzes the
value of that feature, if any. BVWG Exhibit 2 Backup Documentation;
Tr. at 34-35, 37-39. To summarize its findings the BVWG prepared the
following matrix which lists each actual discriminator and inserts
pluses or minuses, as applicable, with respect to each proposal:
Qualitative DiscriminatorsCSCEDS LM Keane
CMM Level +
Understanding/Embracing of STARS Program+++ +
Location/Facilities + + + +
Program Visibility + +
Corporate Support for the STARS Performance Team+++
Merit of Proposed Subcontractors++
Senior Program Manager + + +
Management Team Composition+ +
Quality Assurance Role and Responsibility+++ +
Corporate Resources + + +
Degree of Corporate Commitment+
Sample Tasks + +
Performance Risk
Direct Labor Costs-Risk
Indirect Rates (Prime)-Risk -
Subcontractor Costs-Risk- - -
Distribution of Work Effort-Risk -
Net Totals: 10 8 5 3
Each SSAC member was given the BVWG's full report, and the SSAC
received an extensive debriefing on the reasons various proposals did
or did not offer the various discriminators. SSAC Recommendation at
30-33; Tr. at 42-43, 89, 125-30. The SSAC concurred with the BVWG's
recommendation and provided an extensive debriefing to the SSA. As
part of his debriefing the SSA was given and reviewed the SSAC's Final
Recommendation, which contained the BVWG's report, as well as the
BVWG's detailed analysis of proposals. Tr. at 136, 143-46, 175-78.
Keane argues that this best value analysis improperly disregarded and
distorted the evaluation scheme set forth in section M of the RFP and
improperly ignored many of Keane's major strengths as found by the
lower-level evaluation committees. According to Keane, since the SSAC
unanimously voted to accept the findings of the committees, it should
have made award based on their adjectival ratings.
In a negotiated procurement with a best value evaluation plan,
adjectival ratings are only guides to assist contracting agencies in
evaluating proposals; they do not mandate automatic selection of
particular proposals. Chemical Demilitarization Assocs., B-277700,
Nov. 13, 1997, 98-1 CPD para. 171 at 6. Source selection officials, which
includes officials at an intermediate level, are not bound by the
recommendations or evaluation judgments of lower-level evaluators,
even though the working level evaluators may normally be expected to
have the technical expertise required for such evaluations. PRC,
Inc., B-274698.2, B-274698.3, Jan. 23, 1997, 97-1 CPD para. 115 at 7.
Source selection officials have broad discretion in determining the
manner and extent to which they will make use of the technical and
cost evaluation results, and their judgments are governed only by the
tests of rationality and consistency with the stated evaluation
criteria. Id.; Chemical Demilitarization Assocs., supra.
INS did not discard the lower-level evaluation results and, along with
them, the RFP's evaluation scheme. The record shows that INS analyzed
those results and drew from them a list of discriminators with which
to make its best value analysis, in accordance with the
solicitation.[9]
The solicitation explicitly advised offerors that INS planned to
assess the benefits of superior features versus added cost. RFP sec.
M.3.3. Moreover, where detailed technical proposals are sought and
detailed technical evaluation criteria are used to enable the agency
to make comparative judgments about the relative merits of competing
proposals, offerors are on notice that qualitative distinctions will
be made. Computer Sys. Dev. Corp., B-275356, Feb. 11, 1997, 97-1 CPD para.
91 at 6.
Information regarding specific proposal advantages or disadvantages is
the type of information that agencies should make available to source
selection officials to enable them to reasonably determine whether and
to what extent adjectival evaluation ratings indicate meaningful
differences in proposals and the resulting value of such differences.
Israel Aircraft Indus., Ltd., MATA Helicopters Div.,
B-274389 et al., Dec. 6, 1996, 97-1 CPD para. 41 at 7. Such
considerations are the essence of any best value source selection
decision, F2M-WSCI, B-278281, Jan. 14, 1998, 98-1 CPD para. 16 at 8, and
the BVWG's designation of discriminators was simply and properly a
tool to help assess which proposals represented the best value. See
Engineering and Professional Servs., Inc., B-262179, Dec. 6, 1995,
95-2 CPD para. 266
at 5.
Keane complains that the BVWG's selection of discriminators improperly
distorted the evaluation scheme by "abandoning" the experience and
past performance evaluation factor. Keane points out that the
management factor is represented by numerous discriminators. In
contrast, the experience and past performance factor, which was just
as important as the management factor, is represented by only one
discriminator, and that a negative one--performance risk. According
to Keane, the selection of discriminators gave disproportionate weight
to the management factor and other, less important factors, and failed
to give Keane credit for its superior past performance rating.
The record shows that INS did not "abandon" the experience and past
performance factor, but that the BVWG simply did not consider
favorable past performance to be a discriminator. As a BVWG member
stated at the hearing, past performance instills confidence that a
company will probably do well but does not in and of itself guarantee
that a company will do well, whereas poor past performance is a good
indicator of performance risk. Tr. at 92-96, 102-106.
There is no requirement to have a discriminator for each evaluation
factor, or to have an equivalent number of discriminators for equally
important evaluation factors. See Computer Sys. Dev. Corp., supra.
Moreover, whenever equal factors are considered, the fact that one is
chosen as more valuable does not mean that the relative weights of the
evaluation factors have been changed or that one has been abandoned.
It simply means that one has become the discriminator between
competing proposals. Calspan Corp., B-258441, Jan. 19, 1995, 95-1 CPD para.
28 at 14. There is also no requirement that award discriminators be
the most heavily weighted factors. Research for Better Schools, Inc.,
B-270774.3, June 17, 1996,
96-2 CPD para. 41 at 8. So long as the less heavily weighted criteria
have been disclosed to the offerors in the RFP, as they were here,
there is nothing improper in their becoming the discriminator where
competing proposals are evaluated as equal in the more heavily
weighted ones. Duke/Jones Hanford, Inc., B-249367.10,
July 13, 1993, 93-2 CPD para. 26 at 10 n.8.
While Keane's proposal was rated "outstanding" under the past
performance subfactor, to the other offerors' "good" ratings, the BVWG
considered the offerors' past performance to be approximately the
same. Tr. at 100. Indeed, our review of the MEPPEC's findings show
virtually identical strengths for all offerors, and there is no basis
for us to conclude that the minor concerns with respect to the
awardees' past performance amounted to a material distinction. As a
result, we cannot conclude that the agency improperly omitted a
positive discriminator for past performance here.
Keane also asserts that INS improperly failed to credit the firm for
offering two discriminators, one for a superior senior program
manager, and one for possessing a CMM level higher than 2. With
respect to the first, Keane merely argues that it received a major
strength for its senior program manager and, thus, should have been
credited with this discriminator. Since Keane does not address the
BVWG's detailed rationale for crediting other offerors with this
discriminator, and not itself, we have no basis to question the
agency's assessment. See BVWG Exhibit 2 Backup Documentation; Final
MEPPEC Report, Appendix 3 at C-11; Appendix 2 at D-18; Appendix 3 at
F-9. With respect to the second, Keane's allegation is untimely.
On the day of its debriefing Keane was given redacted copies of BVWG
Exhibits 1 and 2. The first potential discriminator listed on exhibit
1 was "CMM Level . . . Higher than Level 2, per RFP." As noted above,
the RFP specifically provided that offerors with documented Software
CMM levels higher than 2 were to be given credit in the overall best
value analysis. RFP sec. M.2. There was no plus in Keane's column for
this discriminator. This discriminator was abbreviated and carried
over onto exhibit 1; again, there was no plus in Keane's column for
this discriminator.
Despite Keane's knowledge on that day that offerors with CMM levels
higher than 2 were to be given credit in the best value analysis; its
belief that it offered a
CMM level 3; and its knowledge that it was not given credit for having
a CMM level higher than 2 in the best value analysis as set forth in
BVWG exhibits 1 and 2, Keane failed to raise this issue until its
comments on the agency report. In view of the detail available to
Keane when it filed its protest, it was required to raise the issue
then and its failure to do so renders the issue untimely. Global
Eng'g & Constr. Joint Venture, B-275999.4, B-275999.5, Oct. 6, 1997,
97-2 CPD para. 125 at 4.
Finally, Keane argues that the award decision was irrational and
arbitrary because it reduced the entire source selection decision to a
tradeoff between the net totals of superior features versus cost
risks. As discussed above, while the BVWG exhibits list net totals,
and the SSAC's recommendation and source selection decision speak in
terms of the numbers of features, the record is abundantly clear that
the award decision was not based on a simple "scorecard" as Keane
suggests. Instead, the SSAC and SSA relied on these numbers as a
shorthand for the well-documented, reasonable findings of the BVWG;
the source selection decision was clearly based on the BVWG's findings
as a whole. See Israel Aircraft Indus., Ltd., MATA Helicopters Div.,
supra, at 7; Tr. at 111-17, 147-48, 178-80.
The rationale for the source selection decision is clearly outlined in
the full BVWG Report, the SSAC Recommendation, and the source
selection decision. As discussed above, the rationale is rational,
supported by the evaluation record, and
consistent with the evaluation criteria. As a result, there is no
basis to object to INS's selection of CSC, EDS, and LM for contract
award, and no basis to object to INS's decision not to award a
contract to Keane. Calspan Corp., supra, at 14.
The protest is denied.
Comptroller General
of the United States
1. INS intended to award three performance contracts but reserved the
right to award more or fewer such contracts based on the evaluation
results. RFP sec. C.1.2.
2. The CMM is a tool for assessing and evaluating the maturity of an
organization's software processes. The model, which identifies five
levels of maturity, provides for software process assessments--or
self-assessments--that allow organizations to implement improvement
programs and for software capability evaluations that allow evaluators
to identify the risks of selecting among different contractors for
award. Research Analysis and Maintenance, Inc., B-272261, B-272261.2,
Sept. 18, 1996, 96-2 CPD para. 131 at 4 n.1.
3. Since INS evaluated Keane's proposal and considered it for award,
and since we conclude that the evaluation and source selection
decision were reasonable, we need not resolve the question, raised by
INS during the pendency of the protest, whether Keane's proposal
actually met this mandatory requirement. As discussed further below,
Keane's allegation that INS improperly failed to credit it with
possessing a documented CMM level 3 in the best value analysis is
untimely.
4. As a result, our consideration of INS's cost realism analysis need
only address those areas of cost risk that became important in the
best value analysis.
5. The ratings assigned to the adjectivally rated other business
factors are not at issue here, and all offerors were rated as passing
the remaining other business factors.
6. These values are based upon the RFP's overall contract maximum
value, which was computed by multiplying each offeror's total
evaluated costs by a factor of three. This formula assumes the
possibility that each performance contractor might perform all of the
work ordered under the performance contracts. RFP sec. H.10.2.2.
7. As revised by amendment No. 0005, offerors were to be given passing
ratings if none of their business systems that required approval had
significant outstanding citations of non-approval that might affect
their eligibility for a cost-type contract, and if the offeror had no
significant CAS violations that might affect their eligibility for a
cost-type contract. RFP sec. M.4.4.2.6.
8. In its comments Keane pointed out that the BEC found six
significant cost risks in EDS's proposal but the BVWG only considered
one of these to be a discriminator. In response to our request, INS
provided a detailed explanation for the BVWG's decision, which Keane
failed to address in its subsequent response. As a result, we
consider the issue to have been abandoned and will not consider it.
IT Corp.,
B-258636 et al., Feb. 10, 1995, 95-1 CPD para. 78 at 8 n.11.
9. The 17 discriminators selected by the BVWG are not, as Keane
insists, undisclosed evaluation criteria. All are drawn directly, and
many verbatim, from section M of the solicitation.