TITLE: B-298833.4; B-298833.5, IBM Global Business Services, March 1, 2007
BNUMBER: B-298833.4; B-298833.5
DATE: March 1, 2007
*******************************************************************
B-298833.4; B-298833.5, IBM Global Business Services, March 1, 2007
DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective
Order. This redacted version has been approved for public release.
Decision
Matter of: IBM Global Business Services
File: B-298833.4; B-298833.5
Date: March 1, 2007
Thomas Humphrey, Esq., John E. McCarthy, Jr., Esq., David Z. Bodenhiemer,
Esq., Rebecca K. Lee, Esq., Steven H. Talkovsky, Esq., and Amy Laderberg
O'Sullivan, Esq., Crowell & Moring; Todd Hutchen, Esq., IBM Global
Business Services, for the protester.
Carl J. Peckinpaugh, Esq., and Helaine G. Elderkin, Esq., Computer
Sciences Corporation, for the intervenor.
Brent Curtis, Esq., Richard C. Bean, Esq., Maj. Angela L. Penny, and Maj.
Erik A. Troff, Department of the Air Force, for the agency.
David A. Ashen, Esq., and John M. Melody, Esq., Office of the General
Counsel, GAO, participated in the preparation of the decision.
DIGEST
In procurement for system integrator to deploy commercial off-the-shelf
software product suite and change management techniques, quotation was
reasonably evaluated as high risk where vendor intended to involve too few
government personnel in change management process to accomplish important
change management activities; protester's defense of its ratio of
government personnel to contractor consultants is undercut by fact that
ratio was significantly lower than that in typical commercial
implementations.
DECISION
IBM Global Business Services protests the Department of the Air Force's
issuance of a task order to Computer Sciences Corporation (CSC), under
request for quotations (RFQ) No. FA8770-05-Q-0022, to serve as the
Expeditionary Combat Support System (ECSS) System Integrator (SI). IBM
challenges the terms of the solicitation and the evaluation of quotations.
We deny the protest.
The RFQ was issued, in accordance with Federal Acquisition Regulation
subpart 8.4, to vendors holding Federal Supply Schedule Department of
Defense Enterprise Resource Planning (ERP) Systems Integration Services
Blanket Purchase Agreements. The RFQ called for an SI to deploy a
commercial off-the-shelf (COTS) software product suite and change
management techniques, which would replace legacy logistics systems and
antiquated business processes. The SI is required to configure, integrate
and implement the COTS software, as well as provide leadership and
resources to redesign and transform Air Force logistics operations so as
to improve logistics results and reduce support costs. The RFQ generally
contemplated a phased approach. The initial phase encompasses preparing
the organization for change and defining how the selected COTS product
suite will be configured to meet the agency's requirements, which will
include such activities as: (1) change management, that is, the process by
which executives, middle managers, and end users are prepared to accept
changes in business processes; (2) blueprinting, that is, creating
descriptions of the Air Force's business processes and determining how the
COTS product suite will support the agency's business needs; and
(3) continuing efforts of "Pathfinders" to prototype COTS implementations
and to prove that they can work in an Air Force logistics environment. In
the second phase, the SI will undertake implementation of ECSS, performing
all work necessary to configure and test the ECSS solution in preparation
for deployment; in the third phase, the SI will undertake deployment of
ECSS into a live production operation; and in the fourth phase, the SI
will transition ECSS into Air Force managed sustainment. ECSS Request for
Quote sections 3, 4, 5; ECSS Task Descriptions; ECSS Statement of
Objectives (SOO); Agency Comments, Feb. 10, 2007, at 11 n.10, 41 n.30, 57.
Award was to be made to the vendor whose quotation represented the "best
value" to the government based on three evaluation factors: (1) mission
capability/quote risk (including subfactors for program execution, change
management, and program management); (2) past performance; and (3) price
(comprised of the value of the task order plus the value of any
government-furnished personnel and property). Under each mission
capability subfactor, quotations were to be rated X+ for exceeding
minimum performance or capability objectives in a beneficial way, X for
meeting minimum objectives, X- for not clearly meeting minimum
objectives, or F for failing to meet minimum objectives. In addition,
quote risk ratings were assigned at the subfactor level. Mission
capability/quote risk and past performance, combined, were significantly
more important than price.
CSC and IBM submitted quotations. After conducting extensive discussions
with the vendors, the agency requested final quotation revisions (FQR). On
September 8, 2006, the Air Force issued a task order to CSC on the basis
that its quotation represented the best value to the government. IBM
thereupon filed several protests with our Office challenging the award
(B-298833, B-29883.2, B-298833.3). The agency subsequently determined to
take corrective action, including reevaluation of quotations, and we
therefore dismissed the protests. After reevaluating quotations, the
agency determined that CSC's quotation was more advantageous than IBM's
under each of the evaluation factors, and remained the best value. The
ratings were as follows:
+------------------------------------------------------------------------+
| |IBM |CSC |
|----------------------------------+--------------+----------------------|
|Mission Capability | | |
|----------------------------------+--------------+----------------------|
|Program Execution |X /low risk |X /moderate risk |
|----------------------------------+--------------+----------------------|
|Change Management |X /high risk |X+ /low risk |
|----------------------------------+--------------+----------------------|
|Program Management |X /low risk |X+ /low risk |
|----------------------------------+--------------+----------------------|
|Past Performance |Confidence |Significant Confidence|
|----------------------------------+--------------+----------------------|
|Evaluated Price | | |
|----------------------------------+--------------+----------------------|
|Task Order Value |$747.0 million|$627.8 million |
|----------------------------------+--------------+----------------------|
|Value Government Furnished |$51.1 million |$118.9 million |
|Personnel | | |
|----------------------------------+--------------+----------------------|
|Value Government-Furnished |$0 million |$5.0 million |
|Property | | |
|----------------------------------+--------------+----------------------|
|Total Price |$798.1 million|$751.7 million |
+------------------------------------------------------------------------+
CSC's quotation was rated X+ under the change management and program
management subfactors for exceeding ECSS objectives, while IBM's quotation
was rated only X. Further, IBM's approach to change management was rated
as high risk due to IBM's request for an insufficient number of government
personnel to be furnished to support its approach; this was found likely
to cause significant disruption of schedule, significant increased cost,
and significant degradation of performance. Source Selection Decision
(SSD) at 4-5.
Both quotations were rated X under the program execution subfactor. In
this regard, IBM planned to initially deploy ECSS to agency Air Logistic
Centers, while CSC planned to initially deploy ECSS to Air Force bases.
IBM's quotation was evaluated as exceeding ECSS objectives by offering
full operational capability 4 months earlier than specified in the
notional solicitation schedule (and 5 months earlier than CSC), which
would potentially accelerate cost savings from retiring legacy systems and
reducing inventory. CSC's "bases first" approach of deploying a complete
set of process functionality to end-users at each base also was favorably
evaluated on the basis that it would make ECSS available to 70 percent of
the end users up front and (unlike under IBM's approach) in a single
deployment per site, which would eliminate the need for dual transactions
by users.[1]
As for past performance, the agency assigned CSC's quotation a rating of
significant confidence and IBM's a rating of only confidence, on the basis
that CSC had better demonstrated the capability to develop and deploy
large-scale, highly complex COTS/ERP integrated business solutions
comparable to the magnitude and complexity of the ECSS. On the other hand,
the agency found that IBM had demonstrated the skills and capabilities
necessary to transform small-to-medium-scale business operations using
COTS/ERP applications and industry best practice, but determined that IBM
(unlike CSC) had not demonstrated the capability to deploy large-scale,
highly complex COTS/ERP solutions reasonably comparable to the magnitude
and complexity of the ECSS.
Given CSC's advantages under the non-price factors and its lower evaluated
price, the Air Force again made award to CSC. IBM thereupon filed this
protest.
In reviewing protests against allegedly improper evaluations, it is not
our role to reevaluate proposals. Rather, our Office examines the record
to determine whether the agency's judgment was reasonable and in accord
with the RFP criteria and applicable procurement statutes and regulations.
See Rolf Jensen & Assocs., Inc., B-289475.2, B-289475.3, July 1, 2002,
2002 CPD para. 110 at 5. Here, based on our review of all of IBM's timely
arguments, we find no basis for questioning the award decision. We discuss
several of IBM's arguments below.
GOVERNMENT-FURNISHED PERSONNEL
IBM raises various arguments concerning the role government-furnished
personnel were to play in the SI contractor's efforts under the ECSS
program. In this regard, the RFQ made clear the agency's view that a
successful implementation of ECSS would require a partnership between the
agency and the SI. According to the RFQ, "[t]he ECSS COTS Product Suite
represents industry best practices which, when combined with the
experience and knowledge of the Air Force Logistics Community, ECSS
project team, and Systems Integrator (SI), will provide the best
opportunity to achieve the objective as described in the RFQ documents."
RFQ, Best Value Selection Factors, sect. 4.1. Likewise, the RFQ (as
amended after the receipt of initial quotations) indicated that,
consistent with the "governing umbrella philosophy of Enterprise Resource
Planning System Integration Services[,] . . . the parties anticipate a
unique sharing of concerns, knowledge and ideas via a partnership whereby
Government personnel are made available to provide a source of
information, and to work side-by-side with the contractor." RFQ amend. 5.
In order to account for vendors' varying approaches to making use of
government personnel in designing and implementing ECSS, the RFQ required
vendors to complete a Government Personnel Matrix, to be incorporated into
the awarded order as attachment 14, entitled "Government Personnel";
vendors were to specify by contract year the number of government
personnel in each of three experience levels (senior, "Mid," and junior)
and the percentage of their time that would be required under the vendor's
proposed approach for each ECSS activity (e.g., program management, change
management, business blueprinting). According to the RFQ, attachment 14
"defines the maximum amount of Government Personnel available, by skill
level, per contract year." RFQ amend. 5. For each category of government
personnel for each contract year, the matrix specified a cost per
full-time equivalent (FTE) that would be added to the vendor's quoted
price for purposes of the evaluation in order to "eliminate any
competitive advantage resulting from the vendor's proposed use of
Government Personnel." RFQ amend. 4; RFQ, Best Value Selection Factors,
attach. 14.
The RFQ further provided that "[i]t is incumbent upon the Government to
make the Government Personnel available [in accordance with] . . .
Attachment 14, Government Personnel"; according to the RFQ, the contractor
"shall be entitled to an equitable adjustment should the Government fail
to make the required personnel available [in accordance with] . . .
Attachment 14." RFQ amend. 5. However, the RFQ also provided that the
contractor's request for government personnel shall be filed so as to
allow a lead time of 60 days in order to ensure availability, and that
[b]ecause Government resources availability varies by location, the
request shall allow flexibility in scheduling. Flexibility may require
allowance for alternate dates of availability within the same contract
year, a variance in labor mix or other variances.
Id. In addition, the government reserved the "right to unilaterally
substitute qualified Advisory and Assistance Services (A&AS) contractors,
with similar skill levels, for Government Personnel." Id.
The RFQ further provided for government personnel to be available to
support the contractor's ECSS effort, stating as follows:
The Government recognizes that Subject Matter Experts (SMEs) are
essential to the success of the ECSS program. Key functional SMEs from
across the Air Force Logistics Enterprise will be integrated into the
PMO [Program Management Office]. The Government anticipates that the
SMEs' team will be comprised of approximately three (3) GS-15 or O-6
personnel, twelve (12) GS-14 or O-5 personnel, and twenty (20) GS-13 or
O-4 personnel. At appropriate times these SMEs will be augmented with
additional functional experts from Air Staff and the MAJCOMs.
Statement of Objectives sect. 4.1.2. Vendors were to exclude from
attachment 14, and thus from the price evaluation, government personnel to
be furnished in support of its ECSS effort who were from the ECSS PMO or
from the Logistics Transformation Office (LTO). Attach. 13.
CSC listed in its attachment 14 a total of 1,202.6 government FTEs, spread
over 7 years, that would be required for its ECSS effort; the agency added
an adjustment of $118.9 million (at the rates specified by the agency in
attachment 14) to its evaluated price to reflect the cost of providing
these government personnel. In addition, pursuant to the above RFQ
provisions, CSC's quotation specified a total of approximately 163
unpriced government FTEs, spread over 7 years, which would be required
from the PMO/LTO. In contrast, IBM listed 486.3 government FTEs, which
resulted in a $51.1 million adjustment to its evaluated price. IBM did not
specifically enumerate the government FTEs from the PMO/LTO that it would
require.
Government Obligations
As an initial matter, IBM asserts that issuance of a task order to CSC was
improper because the Air Force lacked authority to contractually commit
government personnel to work with CSC on the ECSS effort. This argument is
untimely. As discussed above, the RFQ made it clear that the agency
anticipated "a unique sharing of concerns, knowledge and ideas via a
partnership whereby Government personnel are made available to provide a
source of information, and to work side-by-side with the contractor." RFQ
amend. 5. Further, the RFQ included attachment 14, wherein vendors were to
list the government personnel time that would be required under the
vendor's proposed approach, with the contractor generally "entitled to an
equitable adjustment should the Government fail to make the required
personnel available [in accordance with] . . . Attachment 14." Id. Under
our Bid Protest Regulations, protests based upon alleged improprieties in
a solicitation apparent prior to the time set for receipt of initial
quotations must be filed prior to that time, while alleged improprieties
subsequently incorporated into the solicitation must be protested not
later than the next closing time for receipt of quotations. 4 C.F.R.
21.2(a)(1) (2006). Since it was clear from the solicitation that
government personnel would be made available to work with the SI under the
contract, and since IBM did not protest this aspect of the solicitation
prior to the next closing time, the protest in this regard is untimely.
In any case, IBM's quotation was based on the same approach about which
IBM complains with regard to CSC's quotation--it provided in attachment 14
that the Air Force would be required to furnish 486.3 government FTEs.
Having submitted a quotation incorporating government-furnished personnel,
IBM cannot now assert that it was improper for the agency to accept CSC's
quotation due to its reliance on such personnel. The integrity of the
protest process does not permit a protester to espouse one interpretation
or position during the procurement, and then argue during a protest that
the interpretation or position is unreasonable or otherwise improper.
Northrop Grumman Sys. Corp., B-298954 et al., Jan. 12, 2007, 2007 CPD
para. __ at 8; BST Sys., Inc., B-298761, B-298761.2, Dec. 1, 2006, 2006
CPD para. __ at 6; AAI Eng'g Support, Inc., Nov. 16, 1994, 95-1 CPD para.
2 at 3-4.
IBM further asserts that CSC rendered its quotation unacceptable by
improperly conditioning its performance on the Air Force's furnishing
government personnel in a certain manner. IBM notes in this regard that
CSC included in its proposed Performance Work Statement (PWS) an
"Assumption[] and Constraint[]" that the "change agent roster will remain
stable with limited turnover and replacements. No more than 10% turnover
of change agents per year." CSC PWS at 47.
Here too, however, IBM is challenging an aspect of CSC's quotation that is
similar to its own. In this regard, while CSC's quotation set forth
certain "Assumptions and Constraints," IBM's defined numerous "Government
Roles and Responsibilities." For example, IBM's quotation provided that it
was the Air Force's responsibility to: [REDACTED] See, e.g., IBM PWS at 3,
17, 26-27, 59-61, 67, 86, 93. While perhaps not as specific as some of the
government obligations set forth in CSC's quotation, IBM's definition of
the "Government['s] Roles and Responsibilities" likewise defined
government obligations under the contemplated order, and to that extent
could be viewed as similarly conditioning IBM's quotation. Again, we view
this argument as inconsistent with INM's position during the procurement,
and we therefore will not consider it. To the extent that CSC allegedly
acted inconsistently with the solicitation, so did IBM. See Northrop
Grumman Sys. Corp., supra; BST Sys., Inc., supra; AAI Eng'g Support, Inc.,
supra.
Availability of Government Personnel
IBM asserts that the Air Force did not adequately account for the risk
that it would be unable to furnish the 1,202.6 government FTEs, and the
approximately 163 additional FTEs from the PMO/LTO, specified in CSC's
quotation.[2] In this regard, the protester notes that the Air Force has
already decided to acquire the services of up to 28 contract personnel
under the General Services Administration's Mission Oriented Business
Integrated Services (MOBIS) contract in order to supplement the LTO staff.
More generally, the protester notes that the Air Force has announced plans
to reduce overall staffing by approximately 40,000 active duty personnel
and 18,000 civilian/guard/reserve personnel. The Air Force responds that
it reasonably determined that adequate government staffing would be
available to support either vendor's proposed ECSS approach.
Based on our review of the record, including testimony at the hearing our
Office conducted in this matter, we find nothing unreasonable in this
aspect of the evaluation. As noted by the agency, ECSS is a major program
that will completely transform the Air Force logistics process by
replacing over 400 legacy systems and supporting over 250,000 concurrent
users. The record indicates that, during the course of the procurement,
the Air Force Director of Transformation for logistics, who determined
that the agency could furnish the 400-1,200 government personnel (not
including PMO or LTO staff) the vendors had proposed, repeatedly briefed
the leadership of the nine Air Force Major Commands involved in the Air
Force logistics process on the extent of the commitment of agency
personnel to ECSS that would be necessary. The director testified that the
ECSS program had received a commitment from the Major Commands to furnish
the requisite support. Hearing Transcript (Tr.) at 65-67, 96-98, 139-42;
ECSS Briefing Index; see, e.g., AF Logistics Transformation eLog21
Briefing Slides, Sept. 13, 2006. Given that the Air Force logistics
workforce consists of at least 160,000 personnel, and the evident
commitment of the Major Commands to furnish the resources necessary to
transform the Air Force logistics process, the agency reasonably could
conclude that the total of 1,365.6 government FTEs called for under CSC's
approach would be made available, especially since only a part-time
commitment would be required for most affected employees. CSC attach.
14.[3]
IBM's Change Management Approach
As noted above, IBM's approach to Change Management was rated as high risk
due to its identification of and request for an insufficient number of
government personnel to be furnished. IBM disputes the agency's
determination that its staffing was inadequate in this regard.
The evaluation in this area was reasonable. The RFQ described change
management--that is, the process by which executives, middle managers, and
end users are prepared to accept changes in business processes--as "the
key to successful implementation and adoption of a logistics enterprise
solution." SOO sect. 3.2. The RFQ explained the importance of change
management as follows: "To succeed, ECSS must gain user buy-in,
demonstrate the value/benefits of common approaches across the Air Force
Logistics Enterprise, and create confidence in the ECSS solution with its
associated processes and supporting organizational changes." Id.
IBM acknowledged in its quotation that "[t]he importance of change
management to the ultimate success of the endeavor simply cannot be
overstated or overemphasized," and that "success is built on a true
partnership with the AF and that active participation of the AF community
in all aspects of the program is critical." IBM Technical Proposal sect.
3.0, Change Management; IBM Response to Evaluation Notice (EN)
A-PE-CR-255. However, the record supports the agency's determination that
IBM's quotation provided insufficient participation by Air Force personnel
in the change management process. In this regard, the agency explains
that, in the proposed sequence of ECSS activities, commencing with
planning and continuing through blueprinting, implementation, deployment,
and transition to sustainment, change management staffing should increase
at the end of planning and the beginning of blueprinting, and then begin
to decline during the last part of deployment and the beginning of
transition to sustainment. The agency found that IBM's approach did not
follow this staffing pattern. Rather, IBM's attachment 14 requested 21.25
government FTEs for participation in change management activities in each
of the first 3 contract years, and then only 17 in year 4, that is, prior
to implementation, 12.75 in year 5, 8.5 in year 6, and none in year 7. In
contrast, CSC's approach more closely followed the desired pattern,
requesting the participation of 4 government FTEs in change management
activities in year 1, 5.6 in year 2, 76.5 in year 3, 133 in year 4, 111.3
in year 5, 26.2 in year 6, and none in year 7. CSC FQR attach. 14.
The agency found as another indication of inadequate involvement of
government personnel in the change management process, the fact that the
425 members of IBM's proposed Stakeholder Engagement Team were to
participate in change management activities for only 5 percent of their
time in years 1-3, 4 percent in year 4, 3 percent in year 5, 2 percent in
year 6, and 0 percent in year 7. The agency found that this level of
participation, after accounting for conferences, meetings and reviews,
would leave little time for true change management activities. Proposal
Analysis Report (PAR) at 30.
Further, the agency determined that another potential approach to
facilitating change management--relying on super users in the user
activities--was not part of IBM's approach as priced in attachment 14. In
this regard, under the RFQ, the SI was to train super users (estimated to
number 5,000), who would then provide support and advice on the new
systems and processes to non-super users. The agency noted that, while
IBM's quotation indicated that "[t]he ECSS program will benefit greatly
from the involvement of the ECSS super users," IBM did not include in its
attachment 14 calculations any time for the super users to provide this
assistance, explaining instead that "the super users will optionally
choose to support their co-workers." RFQ Technical Parameters sect. 3.2;
IBM Response to Evaluation Notice (EN) A-PE-CR-299; PAR at 31. The agency
asked IBM to explain why it characterized super users as "optional
resources" (and accordingly did not list them in attachment 14 for pricing
purposes). IBM responded that, while the ECSS program would "benefit
greatly" from the participation of the super users, "ECSS implementation
and deployment is not dependent on the estimated 5,000 ECSS super-users
serving as local area experts and providing local support." IBM Response
to Evaluation Notice (EN) A-PE-CR-299. The agency ultimately evaluated
IBM's approach as relying on the super users to facilitate change
management, since IBM had listed only limited other government
participation in the change management process. Since, however, the super
users were not priced in IBM's attachment 14, the agency determined that
IBM's approach to super users was further evidence of IBM's inadequate
involvement of government personnel (as priced in attachment 14). Tr. at
1380; Agency Comments, Feb 13, 2007, at 74.[4]
Finally, the agency viewed as still further evidence of IBM's inadequate
involvement of government personnel the fact that IBM's overall ratio of
government personnel to SI contractor personnel was only approximately
0.75:1. According to the agency, a ratio of up to 3:1 was recommended in
large ERP implementations, while the minimum government participation
level was 1:1. PAR at 30-31. IBM disputes the agency's understanding of
the appropriate government/contractor personnel ratio. We note, however,
that IBM itself advised the agency during a February 3, 2005 Industry Day
briefing that "[t]he typical commercial ERP implementation has 3 to
4 clients for every consultant." Feb. 3, 2005 Industry Day Briefing
Slides. According to the protester's briefing, "[m]any public sector
clients do not commit the level of resources required to successfully
implement and sustain their program."
We think the Air Force could reasonably view the above aspects of IBM's
quotation as indicative that its approach (in attachment 14) to government
participation was inadequate. IBM has not shown otherwise. Id. In
particular, given the agency's stated intent to adopt "commercial best
practices," RFQ sect. 3.1, Project Summary/Strategy, and IBM's own
recommendation that the government adopt a more commercial approach to ERP
implementation, IBM's defense of its proposed much more limited level of
government participation is not persuasive. We thus find no basis to
question the agency's determination that IBM's approach to government
participation was inadequate.
IBM asserts that it was misled by the agency as to the expected level of
government personnel participation in the ECSS process. Specifically, IBM
notes that the RFQ stated that "[t]he Government intends for the SI to
provide as many of the necessary project resources as possible." SOO sect.
4.2. IBM further notes that, in response to an IBM question during
discussions as to whether it was "the Government's intention to allow
unlimited use of Air Force resources," the agency responded as follows:
"No, the Government has limited resources and must fully understand the
quantity and timing of personnel required to support each vendor's
approach." Air Force Memo for File, Apr. 20, 2006.
We find nothing misleading here. The statements cited by IBM are general
in nature and, we think, did not warrant an interpretation, essentially,
that the agency intended vendors to minimize their use of government
personnel. In particular, we view the agency's statement--in response to
IBM's inquiry as to whether agency resources were "unlimited"--that the
government has limited resources as no more than a statement of the
obvious. Moreover, IBM's assertion that it was misled ignores the fact
that, in response to its listing of only 146.3 government FTEs in its
initial response to the April 26, 2006 amendment adding the attachment 14
government personnel matrix, the agency advised the firm in a deficiency
notice that "[y]our response to the Government Personnel Template is
unrealistically low." EN A-AD-DR-294. As evidenced in IBM's subsequent
more than three-fold increase in the number of requested government FTEs
listed in its revised attachment 14, ultimately reaching 486.3 FTEs, IBM
clearly understood that proposing too limited government participation in
the ECSS process would be evaluated unfavorably.
PAST PERFORMANCE
As noted above, the Air Force rated CSC's past performance "significant
confidence" and IBM's only "confidence." IBM challenges CSC's rating
primarily on the basis that it failed to reflect CSC's poor performance
under contracts concerning the Internal Revenue Service's (IRS) Business
Systems Modernization (BSM) and Integrated Financial System (IFS)
programs.
The evaluation of past performance, including the determination of the
relevance and scope of vendors' performance history to be considered, is a
matter of agency discretion, which we will not question unless it is
unreasonable or inconsistent with the solicitation criteria, or
procurement statutes or regulations. National Beef Packing Co., B-296534,
Sept. 1, 2005, 2005 CPD para. 168 at 4. The past performance evaluation
here was unobjectionable.
The RFQ provided for the agency to evaluate the "recent and relevant
performance" of the vendor and its three principal subcontractors,
"focusing on and targeting performance which is relevant to the Mission
Capability subfactors and the requirements of the solicitation." RFQ, Best
Value Selection Factors, sect. 5.1. The RFQ listed several factors that
would be considered, including "[m]anagement actions employed and
indicators/metrics used to achieve success and overcome problems." Id. The
RFQ added that "more relevant past performance efforts will be considered
more heavily in the rating than less relevant past performance efforts."
Id. In addition, the RFQ defined "recent" past performance as follows: "In
determining recency, any past performance within the last 60 months (from
the date of RFQ release) shall be considered." Id.
CSC's past performance rating of significant confidence was based on its
having "demonstrated the requisite capabilities to develop and deploy
large-scale, highly complex COTS/ERP integrated business solutions
comparable to the magnitude and complexity of the ECSS." SSD at 5;
Performance Confidence Assessment Group (PCAG) Final Report at 111. In
particular, the agency noted CSC's strong performance under three contract
efforts that were evaluated as "Very Relevant large-scale, highly complex,
integrated business solutions:" (1) a contract under the Army's Logistics
Modernization Program (LMP), an ERP within a Department of Defense
logistics environment, under which CSC reduced legacy system operations
from more than 1,000 non-integrated applications to a single solution; the
Air Force rated CSC's performance very good (the Army program manager
evaluated CSC's performance as exceptional); (2) a contract under General
Dynamics Land Systems' (GDLS) e-Business program, an ERP within a
commercial logistics domain, under which CSC implemented an Oracle-based
ERP solution; CSC's performance was rated very good by the Air Force; and
(3) a contract under a Dupont Chemical Solutions Enterprise ERP program,
under which CSC implemented a solution within a commercial logistics
domain; CSC's performance was rated exceptional by the Air Force. In
addition, the agency determined that five other CSC contracts and seven
subcontractor contracts were relevant; the quality of performance was
rated by the Air Force as exceptional for six of these, very good for
four, and satisfactory for two--CSC's contracts with IRS for the BSM and
IFS programs. Five other CSC contracts were considered somewhat relevant;
CSC's performance was rated by the Air Force as exceptional for four of
these and very good for the fifth. PCAG at 63-111.
IBM's past performance rating of confidence was based on its having
"consistently delivered products or services, developed transformational
processes, drove best business practices into the operational environment,
and developed COTS and/or ERP applications which improved their client's
business operations." PCAG at 57. However, unlike its findings with regard
to CSC, the agency found "less evidence" that IBM had performed projects
comparable to ECSS, and rated none of IBM's contracts as very relevant.
SSD at 5. The agency did identify three relevant contracts that it
characterized as medium-scale, moderately complex projects: (1) a contract
for a Delta Airlines ERP implementation within a commercial logistics
domain; IBM's performance was rated as satisfactory by the Air Force; (2)
a subcontractor's contract for a Defense Logistics Agency ERP
implementation within a DOD logistics domain; the subcontractor's
performance was rated as very good by the Air Force; and (3) a
subcontractor's contract for an Oracle ERP implementation within a
commercial logistics domain; the subcontractor's performance was rated as
exceptional by the Air Force. Twelve additional contracts performed by IBM
or its subcontractors were found to be relevant; the performance on eight
of these was rated exceptional, on three as very good, and on one as
satisfactory. One IBM contract was considered to be only somewhat
relevant, with IBM's performance rated as satisfactory.
IBM primarily asserts that CSC encountered significant performance
problems under the two IRS contracts, and that it was unreasonable for the
agency to rate CSC's performance on those contracts as satisfactory--the
same rating IBM received for its relevant Delta contract, during which it
experienced no significant performance problems.
This argument is without merit. As noted by the Air Force, while CSC
encountered performance problems on the IRS contracts--as reflected in
questionnaires completed by IRS officials, interviews with IRS officials,
and IRS Contractor Performance Assessment Reports (CPAR)--the record
indicates that CSC's rating reflected the fact that IRS was viewed as
sharing some responsibility for those problems, and the fact that CSC's
performance significantly improved during the contract term. In this
regard, IRS's BSM program was a complex, multi-billion dollar effort
encompassing 17 major modernization projects, for which CSC served as the
prime integrator, responsible for an alliance of subcontractors (including
IBM). The record indicates that CSC initially appeared to be overwhelmed
by the magnitude and complexity of the project, and was late in
recognizing performance problems with several of the subcontractors
(including, especially, IBM's performance with respect to IRS's Customer
Account Data Engine (CADE)). CSC received several unsatisfactory ratings
in IRS CPARs due to resulting delays. The Air Force noted, however, that
there were reports of role confusion between IRS and CSC; that the IRS
commissioner at the beginning of the program had stated that it was a
mistake to believe that any vendor could manage such a huge undertaking
without significant input from IRS; and that IRS's associate chief
information officer had not only concurred with the commissioner's view,
but also attributed some of the failure points to the fact that the
contract model was a flawed approach. The Air Force also was advised that
there was a significant improvement when CSC took over CADE development
from IBM, that IRS was "very happy" with the quality of the CADE end
product, and that CSC received an overall excellent rating for quality (as
well as a fair rating for schedule and a good rating for cost control)
under CSC's most recent BSM CPAR. CSC Performance Documentation, IRS BSM
Contract.
Likewise, the record of CSC's performance on the IRS IFS contract, as
reported to the Air Force, indicated that CSC had successfully overcome
early performance problems. CSC was tasked under the IFS contract with
selecting and deploying a new integrated financial system. The record
indicated that CSC initially failed to assign staff with the requisite
expertise, was late in discovering problems with its subcontractor's
performance, and fell behind the required schedule. IRS IFS Performance
Questionnaire. However, the record also indicated that some of the delay
had resulted from problems with the joint IRS/CSC requirements development
process, and that the schedule established by IRS had been very
aggressive, so that the time CSC required for the project actually
favorably compared to that required for similar projects. Project Notes,
IRS IFS. In any case, the record indicated that CSC's performance
subsequently improved, and that the project was successfully completed.
Indeed, the record indicated that IRS's new integrated financial system
resulted in the agency's receiving its first unqualified audit opinion
from GAO. CSC Performance Documentation, IRS IFS Contract.
In view of CSC's ultimately successful performance on the IRS contracts,
as well as the reported extenuating circumstances (i.e., the magnitude and
complexity of the challenges, IRS having contributed to the performance
problems, and/or IRS having established an overly aggressive schedule),
there is no basis for us to object to the Air Force's rating of CSC's
performance on these contracts as satisfactory.[5]
Given the Air Force's determination that CSC had exceptional or very good
performance on three very relevant contracts, and that IBM's quotation
cited no very relevant contracts, there is no basis for questioning the
ratings in this area.
The protest is denied.
Gary L. Kepplinger
General Counsel
------------------------
[1] IBM also proposed a rapid porting of the existing Pathfinder
capabilities into a production environment, thus providing the potential
for accelerating inventory reductions and improved weapons system
availability. However, because this approach required rapid approval of a
waiver from the Office of the Secretary of Defense, which the agency
viewed as unlikely, this approach was assigned a high risk rating. PAR at
26, 73.
[2] IBM also asserts that the agency failed to consider the risk
associated with having government personnel participate with CSC in the
ECSS effort, including such risks as errors by government personnel in
performing their assigned roles. Again, however, since both vendors'
quotations called for the participation of government personnel in the
ECSS effort as contemplated by the solicitation, we see little fundamental
difference between the quotations in this regard.
[3] Although IBM suggests that there may be greater difficulties
associated with meeting CSC's request for 163 government FTEs over 7 years
from PMO/LTO, we note that the record indicates that ECSS is the most
significant program being undertaken by the 754^th Electronic Systems
Group, which is part of the 554^th Electronic Systems Wing. Given that
these units include approximately 1,800 government personnel (as well as
another 1,600 contractor personnel), we think the agency reasonably could
conclude that sufficient government personnel can be matrixed to the PMO
to enable the agency to furnish the requested resources in this area. Tr.
at 359-64, 504-05.
[4] IBM asserts that CSC likewise failed to price any super user time in
its attachment 14. However, since the documents on which this challenge is
based were furnished to IBM on December 15, 2006, and IBM did not raise
the argument until January 3, 2007, more than 10 days later, this aspect
of the protest is untimely. 4 C.F.R. sect. 21.2(a)(2). In any case, IBM's
argument fails to account for the fact that, while its approach was
evaluated as effectively requiring the efforts of the super users, CSC's
approach was based on significantly greater participation by government
personnel, including a Change Agent Network to serve as change agents in
the user activities, which were priced in CSC's attachment 14. Thus, CSC,
unlike IBM, was not dependent upon super users in order to facilitate
change management. CSC FQR Attach. 14 sect. 2.2.1; Agency Comments, Feb.
13, 2007, at 73-77.
[5] IBM also generally questions the process by which the Air Force
determined the relevance of vendors' contracts. However, IBM has shown
neither that the agency unreasonably determined that CSC's LMP, GDLS and
Dupont contracts were very relevant, nor that any of its own contracts
(that were performed within the 60-month recency window) warranted a
similar very relevant rating. For example, IBM has failed to rebut the
agency's determination that its Delta contract was only relevant (not very
relevant) since, while the contract was for an ERP implementation within a
logistics domain, the agency learned that much of IBM's work under the
contract was performed prior to the 60-month recency window, and that
other work had been performed by another contractor. Tr. at 1444-47;
Agency Comments, Feb. 10, 2007, at 64-65.