TITLE:  KPMG Consulting LLP, B-290716; B-290716.2, September 23, 2002
BNUMBER:  B-290716; B-290716.2
DATE:  September 23, 2002
**********************************************************************
KPMG Consulting LLP, B-290716; B-290716.2, September 23, 2002

   DOCUMENT FOR PUBLIC RELEASE                                                
The decision issued on the date below was subject to a GAO Protective      
Order.  This redacted version has been approved for public release.        

   Decision
    
Matter of:    KPMG Consulting LLP
    
File:             B-290716; B-290716.2
    
Date:              September 23, 2002
    
William A. Roberts, III, Esq., David M. Southall, Esq., Timothy W. Staley,
Esq., and Jonathan L. Kang, Esq., Wiley, Rein & Fielding, for the
protester.
Thomas P. Humphrey, Esq., John E. McCarthy, Jr., Esq., Elizabeth W.
Newsom, Esq., and J. Catherine Kunz, Esq., Crowell & Moring, for
International Business Machines Corporation, an intervenor.
John F. Ruoff, Esq., Department of Defense, for the agency.
Henry J. Gorczycki, Esq., and James A. Spangenberg, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision.
DIGEST
    
1.  Under a request for quotations (RFQ) issued in a competitive
procurement under the Federal Supply Schedule, where the RFQ does not
contain a late quotation clause, the contracting agency may accept
quotation modifications received prior to the source selection if
acceptance will not prejudice the other competitors.
    
2.  In a competitive procurement under the Federal Supply Schedule program
with a stated *best value* evaluation plan whereby the technical factor is
significantly more important than all the other factors combined and price
is the least important factor, the contracting agency's decision to select
a substantially higher-priced quotation, even though the lower-level
evaluation did not produce ratings that showed a significant overall
difference in technical merit between the quotations, is reasonable where
the higher-level evaluators and source selection authority reasonably
identified significant advantages in technical merit in the higher-priced
quotation, consistent with the evaluation criteria stated in the
solicitation, which justified an award based on the higher-priced
quotation.
DECISION
    
KPMG Consulting LLP protests an award to International Business Machines
Corporation (IBM) under request for quotations (RFQ) No. MDA210-02-T-0003,
issued by the Department of Defense (DOD), Defense Finance and Accounting
Service (DFAS), for the development and maintenance of a DOD-wide
Financial Management Enterprise Architecture (FMEA) for the Office of the
Under Secretary of Defense (Comptroller).
    
We deny the protest.
    
The RFQ, issued November 29, 2001, contemplated a single-award blanket
purchase agreement (BPA) to cover orders under General Services
Administration (GSA) Federal Supply Schedule (FSS) contracts for 5 years,
or until expiration of the GSA schedule, whichever comes first. 
    
The RFQ set forth a *best value* evaluation plan considering the following
evaluation factors listed in descending order of importance:  technical,
management, past performance, and price.  The RFQ stated that the
technical factor *is more important than the other factors combined.*  RFQ
at 9.  Under the technical factor, the RFQ identified four equally
weighted subfactors:  (1) understanding the problem, (2) solution
approach, (3) integrated solution, and (4) risk mitigation.  Under the
management factor, the RFQ identified three equally weighted subfactors: 
(1) management approach, (2) delivery schedule, and (3) personnel.[1]  RFQ
at 5‑6, 9.  Under the price factor, the RFQ indicated that the cost
of the initial task order, which constituted the largest portion of the
FMEA, would be evaluated, although this was *not susceptible to a high
degree of estimation, but primarily will consist of labor.*  A government
estimate of labor hours for various categories was stated in the RFQ, and
vendors were requested to submit *applicable proposed discounted labor
rates* to be applied to these estimates to evaluate the vendors' prices. 
RFQ at 7, 12.
    
The RFQ, as amended, stated that the agency might require oral
presentations, and that such presentations and corresponding materials
would be considered by the agency in making a final award selection. 
However, oral presentations, if held, would not be considered discussions,
and the agency reserved the right to make award without discussions.  RFQ
at 13 (added by amend. 0001).
    
The RFQ stated that *[r]esponses shall arrive by January 25, 2002.*  RFP
amend. 0003 at 1.  No late submission clause was included in the RFQ. 
Several vendors, including KPMG and IBM, submitted quotations by that
date.
    
Also on January 25, a source selection plan (SSP) for this acquisition was
approved identifying the source selection authority (SSA), the source
selection evaluation board (SSEB) chairperson, and the other SSEB
members.  Agency Report, Tab M-2, Initial SSP (Jan. 25, 2002), at 2, app.
A.  On January 28, the SSEB convened to evaluate quotations.  The SSEB
evaluated each submission using evaluation worksheets provided in the SSP
identifying the evaluation factors and subfactors stated in the RFQ, and
providing evaluation criteria for each subfactor under the technical and
management factors and for the past performance factor.  The worksheets
provided for assigning a numerical score for each criterion using a
three‑point scale with two points assigned if the quotation was
*responsive,* one point for quotations needing *clarification,* and zero
points for *nonresponsive* quotations.  The worksheets also provided space
for describing strengths and weaknesses.  The worksheets provided for
totaling the numerical scores under each subfactor for the technical and
management factors and for the past performance factor, and assigning
color/adjectival ratings*blue/exceptional, green/acceptable,
yellow/marginal, and red/unacceptable--based on various point ranges
stated for each subfactor or factor.  Agency Report, Tab M-2, Initial SSP
(Jan. 25, 2002), apps. C, D.
    
On February 5, the SSEB chairperson reported the evaluation results to the
contracting officer.  The SSEB had assigned numerical scores and
color/adjectival ratings for each subfactor and factor as provided for in
the SSP.  KPMG's and IBM's quotations were both rated blue/exceptional
overall with IBM's numerical score exceeding KPMG's by a very narrow
margin.  Agency Report, Tab J-2, Initial SSEB Report (Feb. 5, 2002) at 2.
    
Also on February 5, the Director of DFAS appointed as SSA a person
different from the one identified in the initial SSP.[2]  However, the
Director states that there was miscommunication regarding who should be
appointed the SSA and that the individual appointed, though discussed as a
potential SSA, had not been recommended for appointment.  Agency
Submission (Aug. 7, 2002), encl. 2, Affidavit of the DFAS Director.  On
February 22, the Director rescinded his initial appointment and appointed
as SSA the Deputy Under Secretary of Defense (Financial Management), the
same person identified as the SSA in the initial SSP.  Id.; Agency Report,
Tab T-3, SSA Appointment Memo (Feb. 22, 2002).
    
Subsequently, the SSA established a source selection advisory board
(SSAB).  Agency Submission (Aug. 7, 2002), encl. 1, Affidavit of SSA.  On
March 1, the SSEB chairperson and the contracting officer briefed the SSAB
on the SSEB's evaluation.  The contracting officer also identified the
evaluated prices of the quotations--approximately $33 million for IBM's
quote and approximately $20 million for KPMG's--and recommended award to
KPMG.  The SSAB reviewed the quotations and determined that material
differences existed between them, and that the SSEB evaluation lacked
sufficient detail to permit the SSAB to identify significant
discriminators among the quotations or to assess their relative merit
considering price and technical factors.  On March 11, the SSAB
recommended that the SSEB reevaluate quotations using evaluation
worksheets with a larger numerical rating scale (i.e., 0-unacceptable,
1-marginal, 3-acceptable, and 5-exceptional), and provide additional
narrative descriptions of strengths and weaknesses at the subfactor
level.  The contracting officer agreed with the SSAB and revised the SSP
accordingly.  Contracting Officer's Statement at 6; Agency Report, Tab
I-3, Memo from SSAB to the Contracting Officer (Mar. 11, 2002); Tab I-4,
Briefing Slides (Mar. 1, 2002).
    
On March 18, the contracting officer received an unsolicited notice from
IBM that the vendor was removing a team member from its quotation. 
Contracting Officer's Statement at 6.  The team member, Arthur Andersen
LLP, had notified IBM that GSA had suspended Arthur Andersen from all
contracting except ongoing subcontract work, and that, since an agreement
for Arthur Andersen to be a subcontractor under IBM's quotation had not
been executed prior to the suspension, Arthur Andersen was withdrawing
from IBM's quotation.  Agency Report, Tab O-8, Letter from Arthur Andersen
to IBM (Mar. 18, 2002).  IBM's notice to the agency included the following
statements:
    
Please note there will be no impact to IBM's ability to perform in
accordance with our previously submitted proposal.  In particular, there
will be no impact to our technical solution, price or schedule.  The sole
Arthur Andersen individual identified in the key personnel section of our
proposal will be replaced with an individual of equal or superior
qualifications.
In assembling our team, we deliberately sought a breadth and depth of
talent including two *Big Five* [public accounting] firms, Arthur Andersen
and KPMG LLP.[3]  As a result, the void created by the withdrawal of
Arthur Andersen will be completely addressed by IBM and its other team
members.  In fact, we have conferred with our teammates and confirmed that
the scope previously allotted to Arthur Andersen can be absorbed by IBM
and its team members.
Agency Report, Tab O-7, Letter from IBM to Agency (Mar. 18, 2002).  The
contracting officer forwarded IBM's notice to the SSEB without comment. 
Contracting Officer's Statement at 6.
    
On March 25, the SSEB completed the reevaluation requested by the SSAB and
reported the results to the contracting officer.  The SSEB assigned
overall ratings of green/acceptable to both IBM's and KPMG's quotations. 
With one exception, each of these quotes received ratings of
green/acceptable under every factor and subfactor. 
    
The lone exception was that the SSEB rated KPMG's quotation
yellow/marginal under the first technical subfactor--understanding the
problem.  Agency Report,
Tab J-1, SSEB Memo to the Contracting Officer (Mar. 25, 2002), at 10.  The
SSEB stated that, although KPMG addressed certain aspects of that
subfactor *very well,* the quotation *did not evidence a thorough
understanding of the complexity and scope of the problem . . . lacked
information recognizing data standardization efforts and the relationship
of the effort to the joint Operational Architecture, [and] was somewhat
weak on describing the organizational complexities associated with the
FMEA.*  Id. at 11.  In contrast, the SSEB stated that IBM's quotation
under this same subfactor demonstrated *an excellent understanding of the
scope and complexities, both organizationally and functionally, associated
with the effort,* and addressed the *historical challenges of DOD in
pursuing financial reform.*  Id. at 7.
    
The SSEB's general summary of the quotations reflects that both vendors
were found to have relevant experience and *very acceptable* quotations. 
The SSEB did identify a significant difference between the two
quotations.  It found that KPMG's quote emphasized a *partnership* with
the agency with several actions contingent upon agency approval, and
focused on *system compliance and identification of systems that will be
part of the FMEA solution.*  In contrast, the SSEB found that IBM's quote
was *built on a clear understanding of the magnitude and complexity of the
FMEA* and its impact on DOD, that IBM viewed the project as a
*transformation* within the agency that was larger than merely redesigning
current business processes, and that IBM expressed an intent to assume a
*very strong leadership/'take charge' role* applying experience and
knowledge from its own successful corporate transformation that IBM stated
rivaled the scope and complexity of the FMEA.  Id. at 7, 11.
    
On March 28, the SSAB began review of the SSEB reevaluation and the
underlying quotations.  That same day, the contracting officer briefed the
SSA (with the SSAB in attendance) on the SSEB reevaluation.  The
contracting officer again recommended award to KPMG.  Contracting
Officer's Statement at 9-10.
    
On April 4, KPMG and IBM made oral presentations to the SSA and the SSAB. 
Following the oral presentations, the SSAB issued a memo to the SSA that
summarized the SSEB evaluation, and identified areas where the SSAB's
review showed significant differences between the two quotations. 
Contracting Officer's Statement at 10.  The memo also presented the SSA
with both the color/adjectival ratings and numerical scores from the
SSEB's reevaluation, which indicated that the quotations had the same
overall color/adjectival rating, green/acceptable, with similar overall
numerical scores--KPMG's quotation received 3.54 on a scale of 5 and IBM's
quotation received 3.48.  Agency Report, Tab I-1, Memo from the SSAB to
the SSA (Apr. 4, 2002), attach. 1, at 1-2.  However, as detailed below,
the SSAB determined that there were significant discriminators between the
quotations in four areas. 
    
The first discriminator, *understanding the problem,* was evaluated as
arising under the first technical subfactor of the same name.  The SSAB
considered this issue key to the successful development of the FMEA, since
a complete understanding of the problem is critical to performing the
subsequent tasks.  Of the shortcomings apparent in KPMG's quotation, as
identified in the SSEB's evaluation, the SSAB considered most noteworthy
KPMG's shortcomings in understanding DOD's previous attempts at financial
reform.  Although KPMG had been assisting the agency for some time on
related components of this problem, and thus should have the capability to
demonstrate an understanding of the agency's financial management
problems, its quotation did not identify all relevant components of the
problem, and when it did generally identify a problem area, KPMG failed to
elaborate on it.  The SSAB considered this weakness to be exacerbated by
KPMG's claim that it would improve upon the RFQ's required
milestone/timeframe requirements for completing the FMEA.  The SSAB
considered the RFQ timeframes to be aggressive and that, considering the
complexity of the problem, even the most capable team may not be able to
meet those timeframes.  The SSAB considered the disjoint--the incongruity
between KPMG's shortcomings in identifying the problem and its claim that
it would reduce timeframes--as evidencing that KPMG's quotation was unduly
optimistic and did not evidence an understanding of the enormity and
complexity of the agency's problems.  Agency Report, Tab I-1, Memo from
the SSAB to the SSA (Apr. 4, 2002), at 2-3.
    
In contrast, the SSAB found that IBM's quotation not only elaborated on
issues that KPMG addressed only minimally, but also identified and
discussed key points that KPMG missed; for example, IBM detailed the flaws
in the current financial management processes, as well as the forces, both
internal and external, that affect those processes and will affect
attempts to change those processes.   Id. at 3. 
    
With regard to this discriminator, the SSAB determined that IBM
demonstrated a firm understanding of the scope and difficulty of the
problems, and KPMG did not.  Given KPMG's apparent lesser understanding of
the problem, the SSAB concluded that, compared to IBM, KPMG likely will
*[r]equire more work on the part of the government to overcome the KPMG
shortcomings, [r]equire more time 'getting up to speed' before KPMG
develops concrete quotations for developing architectures, and [i]ncrease
the risk KPMG will not develop a workable solution in the required
timeframe.*  Id.
    
The second discriminator, *financial management transformation,* was
evaluated as arising under the first three technical
subfactors--understanding the problem, solution approach, and integrated
solution.  The SSAB identified this difference between the vendors' plans
for business process reengineering as being closely related to their
identification of the problem.[4]  The SSAB recognized that DOD's current
procedures, processes and systems are ineffective.  The essential
difference between the vendors' solutions is that KPMG's solution focuses
on modernizing the agency's existing systems to achieve compliance with
applicable standards, whereas IBM goes beyond that, addressing the
extensive transformation that will be needed within DOD to make successful
implementation of the system possible.  The SSAB concluded that *KPMG['s]
approach has less chance of success . . . produc[ing] the substantive
reengineering [that DOD] needs in order to resolve the root causes of
DOD's financial management problems.*  Id. at 3-4.
    
The third discriminator, *leading vice partnering,* was evaluated as
arising under the first management subfactor, management approach, and the
second technical subfactor, solution approach.  The SSAB stated that the
FMEA effort is a *momentous undertaking, requiring an unprecedented effort
by the contractor* to assess and recommend solutions to the agency's
business and financial management processes.  This represents a *financial
management transformation* for which solution approaches that *rely
heavily on involvement by government personnel are unlikely to be
successful, because [DOD] lacks sufficient experience in tackling such
problems,* and because the *disparate DOD communities are typically
resistant to working together and reluctant to compromise their
requirements for the requirements of others.* 
    
The SSAB determined that KPMG's quotation relies on forming a
*partnership* with DOD to develop the FMEA, which essentially relies on
DOD to lead the transformation, rather than on outside
expertise/leadership to overcome the entrenched communities within the
agency.  The SSAB found that this did not represent a dynamic approach
that will push DOD to achieve goals that it has been unable to do in
previous attempts. 
    
On the other hand, the SSAB found that IBM's quotation was built upon the
vendor's experience and lessons learned from that firm's earlier
development of an enterprise architecture and transformation of its own
and its customers' financial management systems, and providing the
leadership for DOD to develop the FMEA.  The SSAB characterized IBM's
approach as one of *bringing in an outside consultant to tear apart an
organization and rebuild it*--a *take charge* management approach, which
the SSAB stated was *the approach needed here.*  In making this
assessment, the SSAB disagreed with an SSEB assessment of risk that IBM's
*take charge* approach so limited interaction with the agency as to be
detrimental to DOD's interests, finding that the SSEB concern arose from
the frequency of meetings contemplated in the quotation, rather than any
actual exclusion of agency consultation.  Indeed, consistent with the SSEB
evaluation, the SSAB found that IBM proposed necessary *collaborative
processes with DOD to include full-time representation from DOD's
functional experts [and other] designated DOD points of contact.*  The
SSAB believed that the SSEB's caution arising from its risk assessment
kept the SSEB from rating IBM higher than it did and properly identifying
a meaningful advantage of IBM's quotation.  Id. at 4-5.
    
The fourth discriminator, personnel, was evaluated as arising under the
third management subfactor of the same name, as well as under the price
factor.  The SSAB first noted a limitation in the price evaluation that,
being based on the government's estimate of labor categories and hours,
did not account for differences in the vendors' approaches.  Nevertheless,
the SSAB stated that analysis of the labor rates in each quotation
provided significant information for evaluation.  On average, KPMG's labor
rates were substantially lower than competing quotes and the government
estimate, whereas IBM's quote was in line with both.  The SSAB considered
two possible explanations for KPMG's substantially lower rate--*reduced
overhead or less-qualified personnel.*  The SSAB determined that KPMG's
facilities could not provide the level of savings in overhead costs to
account for the large margin in labor rates, and therefore, KPMG's lower
price indicated that the quotation must be based on providing
*less-qualified personnel to perform the work.*  The SSAB believed that
IBM's higher labor rates reflected IBM's experience with a similar type of
efforts that recognized the need for more experienced, better qualified
personnel.  As part of this analysis, the SSAB also prepared a
side-by-side comparison of the job description information in IBM's and
KPMG's quotations, from which the SSAB determined that, in many areas, the
differences in experience are *sizeable,* and concluded that IBM will
devote more experienced, and better qualified personnel *in virtually all
job categories,* thus increasing the likelihood of success at the
quotation price.  Id. at 5‑6, attach. 2.
    
Here, too, the SSAB referred to the SSEB evaluation, which found that both
vendors would provide adequate key management personnel and rated the
quotations almost identically.  While the SSAB did not disagree with the
SSEB's description of key personnel qualifications, it determined that,
aside from the fact that the SSEB did not have access to the personnel
cost information in the vendors' quotations, the SSEB evaluation did not
consider the overall level of experience and qualifications of offered
personnel generally.  The SSAB determined that the difference in labor
costs and experience/qualification levels was consistent with the
differences in the vendors' solution approaches, and that KPMG's greater
reliance on DOD personnel for performance *shifts cost and risk for
accomplishing the task to the government.*  Id. at 5-6.
    
In conclusion, the SSAB recommended award to IBM because, even at a
significantly higher price, IBM's quotation represented the best value
because, based on the SSAB's comparison of the significant discriminators
between the quotations, the SSAB found that IBM's quote offers the team
most capable of accomplishing the requirements and meeting the timeframes
of the RFQ with the least risk of failure.  Id. at 6-7.
    
On April 5, the SSA issued her source selection decision.  The decision
stated that it was based on the SSA's review of the vendors' quotations
and oral presentations, the evaluation of the SSEB, and the SSAB Memo of
April 4.  The SSA also stated that she met with the members of the SSAB
and discussed their views.  Her decision analyzed the quotations of KPMG
and IBM as follows:
    
I find the quotations of KPMG and IBM to be acceptable but fundamentally
different.  The KPMG proposal is built around systems compliance and
bringing the department's financial management systems into compliance
with law.  The proposal describes substantial collaboration with the
government to execute this approach.  It is rooted in efforts to review
individual systems and to conform them to federal accounting standards. 
It does not, in my judgment, adequately address the business process
reengineering that is fundamental to this effort.  The IBM proposal, on
the other hand, recognizes that business process reengineering is
fundamental to the development of successful enterprise architectures,
addressing compliance as a necessary component.  IBM's proposal utilizes
proven enterprise architectures to enable the design of a more
comprehensive DoD enterprise architecture solution in the available time. 
It does not rely on government leadership or collaboration but on its own
experience gained from developing its own financial management
architecture.
The quotations also reflect important differences in personnel, an element
that was noted in the RFQ as being material to successful execution of
program objectives.  Although both quotations describe acceptable
management personnel, the skill and experience levels of KPMG's personnel
are generally less, often significantly so, than those of the proposed IBM
personnel.  This difference is consistent with the differences in the
proposed labor rates for the two teams.
I find that IBM's direct experience in the development of enterprise
architectures means less learning will be required by IBM to execute this
requirement.  I consider KPMG's collaborative approach to involve more
risk, because the government does not have personnel with the expertise
necessary to participate effectively in a collaborative approach.  Because
the department seeks improved efficiency and not just compliance with
existing standards and laws, I find KPMG's approach less effective than
IBM's and more likely to require additional effort and expense in order to
obtain improved efficiency.  I find that KPMG's less skilled and less
experienced personnel will also likely result in additional cost by
necessitating additional work.
While KPMG's proposed rates are initially attractive, I associate
substantial risk with them.  The government cannot uphold its end of a
collaborative arrangement with KPMG, and this approach, coupled with less
skilled and less experienced personnel, exposes the government to
significant cost risk and unacceptable performance risk.  IBM's proposal
is significantly less risky.  It offers direct and relevant experience in
the development of large-scale enterprise architectures.  It does not rely
excessively on government experience and participation.  It offers higher
skilled and more experienced personnel at rates consistent with the
government's independent estimate.
The lower risk of the IBM proposal comes at a substantially higher cost to
the government.  However, as noted in the evaluation factors, cost in this
procurement is by far the least important of the evaluation factors, while
the technical factor is more important than all of the other factors
combined, including cost.  The reason for this order of importance is the
importance of this requirement to the department.  The success of this
undertaking is essential to the mission of the entire department and
ultimately to the availability of resources to Warfighters.  It is one of
the Secretary's top priorities and is responsive to Congress' growing
concern about how the department uses and controls its resources.
In view of this importance, I find that the higher cost of the IBM
proposal is more than offset by the greater value of significantly reduced
risk and greater chance for success offered by its Technical and
Management proposal.  I conclude that the proposal of the IBM team
represents the best value for the government and that the BPA be awarded
to IBM.
Agency Report, Tab H, Source Selection Decision, at 1-2.
    
On April 9, the agency offered the BPA to IBM, which IBM accepted. 
Following a debriefing, KPMG filed an agency-level protest.  The agency
denied the protest on June 10.  This protest followed.
    
Under the FSS program, an agency is not required to issue a solicitation
to request quotations, but rather may simply review vendors' schedules
and, using business judgment to determine which vendor's goods or services
represent the best value and meet the agency's needs at the lowest overall
cost, may directly place an order under the corresponding vendor's FSS
contract.  10 U.S.C. S: 2302(2)(C) (2000); OSI Collection Servs., Inc.;
C.B. Accounts, Inc., B-286597.3 et al., June 12, 2001, 2001 CPD P: 103 at
4.  If, however, the agency issues an RFQ and thus shifts the burden to
the vendors for selecting the items from their schedules, the agency must
provide guidance about its needs and selection criteria sufficient to
allow the vendors to compete intelligently.  Where, as here, the agency
intends to use the vendors' responses as the basis of a detailed technical
evaluation and cost/technical trade-off, the agency has elected to use an
approach that is more like a competition in a negotiated procurement than
a simple FSS buy, and the RFQ is therefore required to provide for a fair
and equitable competition.  COMARK Fed. Sys., B-278343, B‑278343.2,
Jan. 20, 1998, 98-1 CPD P: 34 at 4-5.  While we recognize that the
procedures of Federal Acquisition Regulation (FAR) Part 15, governing
contracting by negotiation, do not govern competitive procurements under
the FSS program, Computer Prods., Inc., B-284702, May 24, 2000, 2000 CPD
P: 95 at 4, where the agency has conducted such a competition and a
protest is filed, we will review the record to ensure that the evaluation
is reasonable and consistent with the terms of the solicitation and with
standards generally applicable to negotiated procurements.  OSI Collection
Servs., Inc.; C.B. Accounts, Inc., supra at 4-5; Amdahl Corp., B-281255,
Dec. 28, 1998, 98-2 CPD P: 161 at 3.
    
KPMG alleges that IBM's removal of Arthur Andersen from its quotation
cannot be accepted because it is a late proposal modification.  The
protester asserts that because this acquisition more closely resembles a
competition in a negotiated procurement than a simple FSS buy, FAR Part 15
should be applied, in particular FAR S: 15.208, which permits
consideration of quotations, revisions and modification received after the
date for submission stated in the solicitation only in limited
circumstances not present here.  KPMG therefore asserts that IBM's late
quotation modification should have been rejected, and thus IBM's
quotation, based on teaming with Arthur Andersen, a suspended contractor,
should be rejected.  
    
It is well established that the standard for late quotations does not
generally apply to requests for quotations.  An RFQ, unlike a request for
proposals (or an invitation for bids), does not seek offers that can be
accepted by the government to form a contract.  Rather, the government's
purchase order represents the offer that the vendor may accept through
performance or by a formal acceptance document.  DataVault Corp.,
B-248664, Sept. 10, 1992, 92-2 CPD P: 166 at 2.  It follows that language
in an RFQ requesting quotations by a certain date cannot be construed as
establishing a firm closing date for receipt of quotations, absent a late
quotation provision expressly providing that quotations must be received
by that date to be considered.  Instruments & Controls Serv. Co.,
B-222122, June 30, 1986, 86-2 CPD P: 16 at 3.  An agency may consider
*late* quotations or quotation modifications, so long as the award process
has not begun and other offerors would not be prejudiced.  Id.; ATF
Constr. Co., Inc., B‑260829, July 18, 1995, 95-2 CPD P: 29 at 2.
    
Here, the RFQ did not contain a late quotation provision.  Therefore, the
agency was not required to reject a modification received after the date
stated in the RFQ for submission of quotations.  In the case of IBM's
modification of its quotation, the agency received this modification
several days after the SSAB had requested a reevaluation and prior to the
SSEB's completion of that reevaluation, and prior to the oral
presentations and the SSA's award decision.  No competitor was prejudiced
by the agency's acceptance of IBM's modification.  Therefore, the agency
could accept IBM's removal of Arthur Andersen from its quotation.[5]
    
KPMG also protests that the agency failed to evaluate the effect that
removing Arthur Andersen had on IBM's quotation.  The contributions of
Arthur Andersen to IBM's quotation that the protester alleges are
significant are a key personnel position, a steering committee member,
compliance expertise, and experience with DFAS.  The agency responds that
the modification was considered in the reevaluation, but the changes were
not considered material.
    
The agency's evaluation of IBM's quotation does not mention the Arthur
Andersen individual's steering committee membership, that firm's
compliance expertise, or its experience with DFAS, either before or after
the reevaluation and IBM's modification.  This reasonably supports the
agency's position that the modification was not material in the overall
evaluation of quotations. 
    
With regard to the key personnel position, IBM initially identified, and
submitted a resume for, a person from Arthur Andersen for the key
personnel position of data repository manager.  Agency Report, Tab O-1,
IBM Quotation Executive Summary, at 9; Tab O-3, IBM Management Proposal,
at A-23.  Upon removing Arthur Andersen from its quotation, IBM stated
that this would have no impact on its quotation and that the Arthur
Andersen key personnel position would be replaced *with an individual of
equal or superior qualifications.*  Agency Report, Tab O-7, Letter from
IBM to Agency (Mar. 18, 2002).  During IBM's oral presentation, IBM
specifically identified the replacement, an assistant vice president with
another member of IBM's team, and described his experience, which included
significant data repository, DFAS and other DOD experience.  Agency
Report, Tab O-6, IBM's Oral Presentation Slides, at 3.  Therefore, though
not in the traditional format of a resume, IBM did provide a resume for
this individual.  Though the agency did not solicit this information from
IBM, the RFQ stated that the agency would consider the information
presented at oral presentations in the evaluation.  Where the solicitation
states, as here, that oral presentations will be considered in the
evaluation of quotations, the content of the oral presentation is part of
the proposal and should be considered in the agency's selection process. 
Compare Kathpal Tech., Inc.;  Computer & Hi‑Tech Mgmt., Inc.,
B-283137.3 et al., Dec. 30, 1999, 2000 CPD
P: 6 at 12-13, aff'd, Department of Commerce*Request for Modification of
Recommendation, B-283137.7, Feb. 14, 2000, 2000 CPD P: 27, (where
solicitation states that oral presentations will be considered part of the
quotations, agency cannot eliminate quotations from consideration for
award without considering the corresponding oral presentations), with S.
C. Myers & Assocs., Inc., B-286297, Dec. 20, 2000, 2001 CPD P: 16 at 5
(agency properly did not consider proposal revision stated in the oral
presentation where solicitation stated that oral presentations would not
be considered part of the quotations).  Thus, we find no basis to question
the award selection based on IBM's change in this key personnel position.
    
KPMG next alleges that the agency's evaluation and cost/technical tradeoff
are defective because they are not consistent with the evaluation factors
and subfactors stated in the RFP.  KPMG argues that since the SSEB rated
IBM's and KPMG's quotations the same overall on a color/adjectival basis,
and rated KPMG's quote slightly higher numerically, the quotations must be
considered at least equivalent as to the non‑price factors, and
since KPMG's evaluated price is substantially lower than IBM's, the agency
has no reasonable basis to make a cost/technical tradeoff in favor of the
higher-priced quotation. 
    
Point scores and adjectival ratings are only guides to assist source
selection officials in evaluating quotations; they do not mandate
automatic selection of a particular proposal.  Grey Advertising, Inc.,
B-184825, May 14, 1976, 76-1 CPD P: 325 at 9; PRC, Inc., B-274698.2,
B-274698.3, Jan. 23, 1997, 97-1 CPD P: 115 at 12.  Those officials have
broad discretion in determining the manner and extent to which they will
make use of the technical and cost evaluation results, subject only to the
tests of rationality and consistency with the evaluation criteria.  Grey
Advertising, Inc., supra; A & W Maint. Servs., Inc.--Recon., B-255711.2,
Jan. 17, 1995, 95-1 CPD P: 24 at 4.  Where,
as here, higher-level officials determine that the lower-level evaluators'
ratings do not reflect the actual technical differences in quotations and
the award is protested, the agency must show that its ultimate
determination is reasonable, with sufficient detail to permit our Office
to review the determination for reasonableness.  Chemical Demilitarization
Assocs., B-277700, Nov. 13, 1997, 98-1 CPD P: 171 at 6.
    
Here, as indicated above, the SSAB provided a detailed written analysis of
the differences in the quotations that, though generally identified by the
SSEB, were not reasonably reflected in the SSEB evaluation ratings and
scores.  The SSAB found these differences to be significant and of value
to the government, and the bases for its conclusions are well documented
in the SSAB memo to the SSA summarized above.  While KPMG argues that this
analysis ignored certain strengths in KPMG's quotation that were found by
the SSEB, KPMG only points to certain areas where KPMG's quotation
received slightly higher point scores, without persuasively elaborating on
why these differences represent actual technical superiority; this amounts
to mere disagreement with the agency's evaluation which does not render it
unreasonable.  Dual, Inc., B-279295, June 1, 1998, 98-1 CPD P: 146 at 3.
    
KPMG argues, however, that there was no reasonable basis for the
evaluation of the four discriminators identified by the SSAB and SSA and,
in any event, those discriminators represent unstated evaluation factors
that the agency could not properly consider.
    
In reviewing an agency's technical evaluation under an FSS competitive
acquisition, we will not reevaluate the quotations, but (as with protests
of negotiated procurements) will examine the record of the evaluation to
ensure that it was reasonable and in accordance with the stated evaluation
criteria.  Digital Sys. Group, Inc., B-286931, B-286931.2, Mar. 7, 2001,
2001 CPD P: 50 at 7.  It is fundamental, however, that offerors must be
advised of the bases upon which their proposal will be evaluated.  H.J.
Group Ventures, Inc., B-246139, Feb. 19, 1992, 92-1 CPD P: 203 at 4.  This
means that the agency may not consider unstated evaluation criteria that
are not reasonably related to the stated evaluation factors.  Compare MCA
Research Corp., B-278268.2, Apr. 10, 1998, 98-1 CPD P: 129 at 8-9
(agency's identification of relative differences in technical quotations
that are reasonably related to the stated evaluation factors are not
unstated evaluation criteria) and Computer Sys. Dev. Corp., B-275356, Feb.
11, 1997, 97-1 CPD P: 91 at 7 (same) with Lloyd H. Kessler, Inc.,
B-284693, May 24, 2000, 2000 CPD P: 96 at 4-5 (where agency evaluates each
quotation for a specific type of experience and accords that evaluation
significant weight, it is a significant evaluation subfactor that must be
disclosed in the solicitation).  Based on our review, we find the
discriminators identified by the agency were reasonable and did not
represent unstated evaluation factors not reasonably related to those
stated in the RFQ.
    
As noted, the first of the four discriminators was understanding the
problem.  This was a stated subfactor under the technical factor.  The RFQ
states under this subfactor that each quotation shall demonstrate an
understanding of DOD's financial management environment *including issues,
context, and scope.*  RFQ at 5.  Here, the SSEB rated KPMG's quote
yellow/marginal under this subfactor because it failed to evidence a
thorough understanding of the scope and complexity of the problem.  Agency
Report, Tab J-1, SSEB Final Evaluation Report, at 10-11.  The SSEB also
determined that IBM's quote demonstrated an excellent understanding of the
problem and rated IBM's quote higher under the subfactor.  Id. at 7.  The
SSAB concurred in this distinction.  Moreover, the SSAB considered the
performance timeframe required by the RFQ to be difficult to achieve, yet
KPMG claimed that it would perform in a shorter timeframe, which the SSAB
considered unreasonable and supported its conclusion that KPMG did not
demonstrate a complete understanding of the problem.  This determination
is consistent with the stated subfactor and is reasonable.
    
The next two discriminators were financial management transformation and
leading (IBM) versus partnering (KPMG).  The SSAB identified the former
discriminator as arising under the first three stated technical
subfactors, and the latter as arising under the first management subfactor
and the second technical subfactor.  These  discriminators stem from the
difference in the way these vendors identified the problem.  IBM,
identifying the problems as being of broader scope and complexity than
KPMG, had a more expansive solution and management approach to address the
problems. 
    
The protester alleges that terms used by the SSAB and SSA to refer to this
evaluated difference--financial management *transformation,* a *take
charge* management approach, and *collaboration* as it relates to leading
versus partnering--are unstated evaluation criteria not reasonably related
to those stated in the RFQ.  The first of these terms was used by the
agency to describe the difference between the vendors in the scope of
business process reengineering.  As noted above, business process
reengineering was expressly called for under the RFQ and both vendors
addressed it in their quotations.  As characterized by the agency, KPMG's
approach was to modernize the existing processes to achieve compliance,
whereas IBM's approach was to go beyond achieving compliance and provide
for extensive transformation, recognizing the difficulty that the agency
has had in previous attempts to improve its financial management systems. 
    
The *take charge* management approach was a description of IBM's
leadership approach, and *collaboration* was a term the SSA used to
further summarize the SSAB's comparison of the two vendors' approaches. 
These terms essentially characterize the greater extent to which IBM's
quotation provided for the contractor to assume responsibility for interim
decision-making and performance of other tasks.  We do not think that this
terminology in any way created new evaluation criteria; it merely
described differences between quotations that spanned many of the stated
subfactors.  
    
The protester contends that the repeated use of the term *collaboration*
in the source selection decision shows that the SSA favored
non-collaboration with the government, which is contrary to the terms of
the RFQ, which clearly called for a collaborative effort.  See, e.g., RFQ,
Performance Work Statement MDA210-02-T-0003, at 4-7, 10-13.  A cursory
reading of the SSA's decision (quoted above) could give support to the
protester's concerns, since the source selection decision indeed said that
KPMG's quotation contemplated *substantial collaboration* and that IBM's
does *not rely on government leadership or collaboration*; that KPMG's
*collaborative approach [involves] more risk*; and that the agency *cannot
uphold its end of a collaborative arrangement with KPMG.*  Looking at
these statements in isolation, however, distorts the SSA's summary of the
SSAB's detailed analysis of the real differences between the quotations. 
Contrary to the protester's contention, IBM's quotation does not
contemplate a non-collaborative approach with the government.  In fact,
the SSAB noted, in defusing SSEB concerns about IBM contemplating less
frequent meetings than might be desirable, that IBM's quotation provides
for extensive interaction with relevant government personnel.[6]  See
Agency Report at 12-13; Tab O-2, IBM Quotation's Technical Proposal at 2,
36, 38-39, 47, 78‑79, 101, 103, 105, 109.  Reading the SSA's
selection decision in its entirety, together with the more detailed SSAB
memo to the SSA, it is apparent that the point that was being made in the
source selection decision with its references to *collaboration* was that
the nature of KPMG's collaboration was more dependent on agency personnel
and direction, whereas the nature of IBM's collaboration was a
consultative relationship with fewer demands on government resources.  We
think this is consistent with the stated RFQ subfactors that address the
vendors' solutions and management plans.
    
The final discriminator concerned personnel, which the SSAB identified as
arising under both the personnel subfactor and the price factor.  The
source selection decision stated that a risk of increased costs was
associated with KPMG's less skilled and experienced personnel.  This risk
was identified by the SSAB as originating with its questions concerning
why KPMG's labor rates and overall price were so much lower than the
government estimate and the other vendors' prices.  The SSAB memo to the
SSA stated that the structure of the RFQ and quotations did not facilitate
a detailed analysis of prices, so the SSAB identified factors that could
have a substantial impact on price.  The only factor that the SSAB thought
could conceivably account for the significant labor rate differences was
KPMG relying on less experienced personnel in the labor categories. 
Agency Report, Tab I‑1, SSAB Memo to the SSA (Apr. 4, 2002), at 5. 
    
After reaching this conclusion, the SSAB turned to the labor category
descriptions included in each quotation and concluded that IBM's
descriptions contemplated significantly higher experience levels on
average than did KPMG's.  Id. at 6, attach. 2.  KPMG objects to this
latter comparison because the schedules address experience levels
differently in that KPMG's schedules generally identify a minimum years of
experience figure for each labor category whereas IBM's generally identify
a maximum figure.  While we agree that this comparison was not a
definitive indicator of relative experience because of the different
nature of the vendors' schedules, this comparison does support the
agency's otherwise reasonable assumption to explain KPMG's price variance
from the agency's estimate and other competitors' prices.  Even now, we
find no more reasonable explanation than KPMG proposing less experienced
and less qualified personnel to account for KPMG's substantial price
advantage.  We therefore find the agency's assessment of the resulting
risk on performance is reasonable.
    
In conclusion, we find, based on our review, that the SSAB's and the SSA's
analysis of the quotations and the lower-level evaluation results, and
their identification of significant differences that were not reflected in
the lower-level evaluation results, were reasonable.  As a result of their
review, the SSAB and SSA essentially determined that the SSEB's evaluation
ratings and scores did not reasonably reflect the relative merits of the
quotations of KPMG and IBM.  The resulting record of the relative merits
prepared by the SSAB for the SSA, and on which the SSA's source selection
decision rests, is very detailed and sufficient to support the source
selection decision. 
    
Contrary to the protester's allegations, the cost/technical tradeoff
presented in the source selection decision was reasonable and consistent
with the terms of the solicitation, and with the standards of making such
decisions in negotiated procurements in general.  The SSAB and SSA were
well aware of the substantial difference in the evaluated prices of these
two quotations, but also reasonably concluded there was a significant
advantage in merit offered by IBM's identification of the problem, its
solution approach, and its management plan for performance.  The agency
reasonably determined that this advantage rendered IBM's quotation more
likely than KPMG's to result in success on a challenging project that is
of great importance to DOD.  This was consistent with the RFQ's evaluation
scheme, which provided that the technical factor was significantly more
important than all of the other factors combined, the management factor
was the second-most-important factor, and price was of least importance. 
The agency also reasonably determined that there was a risk that KPMG's
apparent price advantage would decrease in the face of performance.  In
sum, we find the agency's selection of IBM's quotation at a significantly
higher price to be reasonable.
    
The protest is denied.
    
Anthony H. Gamboa
General Counsel
    
    

   ------------------------

   [1] The management factor also contained a discrete fourth subfactor,
security, which was to be evaluated only as acceptable or unacceptable.
[2] Although an SSA was identified in the initial SSP, the record does not
show that one was appointed until February 5.
[3] KPMG LLP is a corporate entity different from the protester.
[4] The RFQ does not state or define the term *business process
reengineering.*  However, the performance work statement states the
following as a key objective of the development of the FMEA:
Reengineer the Department's financially related business processes to
ensure routine availability of reliable, accurate and timely financial
management information upon which to make the most effective business
decisions.
RFQ, Performance Work Statement MDA210-02-T-0003, at 4.
[5] Since this properly accepted quotation modification from IBM was not
solicited by the agency, it did not constitute improper or unequal
discussions with only one vendor.  Also, we need not decide whether IBM's
proposal would have been unacceptable if it had maintained Arthur Andersen
as a subcontractor.
[6] For this reason, we reject KPMG's argument that IBM's quotation is
technically unacceptable for failing to meet the RFQ's colloboration
requirements.