TITLE: Abt Associates Inc., B-295449; B-295449.2, March 2, 2005 BNUMBER: B-295449; B-295449.2 DATE: March 2, 2005 ********************************************************************** Decision Matter of: Abt Associates Inc. File: B-295449; B-295449.2 Date: March 2, 2005 Dean M. Dilley, Esq., and Michael J. Schaengold, Esq., Patton Boggs LLP, for the protester. Garry S. Grossman, Esq., and Aaron M. Forester, Esq., Schiff Hardin LLP, for National Opinion Research Center, an intervenor. Elise Harris, Esq., Alex M. Azar II, Esq., and Bruce R. Granger, Esq., Centers for Disease Control and Prevention, for the agency. Paul E. Jordan, Esq., and John M. Melody, Esq., Office of the General Counsel, GAO, participated in the preparation of the decision. DIGEST 1. Protest that discussions were not meaningful due to agency's failure to identify several weaknesses is denied where it is clear that protester was not prejudiced; assigning protester's proposal additional evaluation credit in the areas involved would leave its technical proposal rated lower than awardee's, and awardee's evaluated cost was significantly lower than protester's. 2. Technical evaluation of protester's proposal was reasonable where initial proposal failed to provide sufficient information regarding key personnel's expertise in a particular area and availability of consultants, and agency reasonably determined that revised proposal failed to provide sufficient information to address its concerns. 3. Technical evaluation of awardee's proposal regarding experience with [deleted] was reasonable where, in response to discussion questions on this matter, awardee provided additional information to demonstrate sufficient [deleted] and other comparable experience of various key personnel. 4. Cost realism evaluation of awardee's proposal was unobjectionable where record shows that agency reviewed cost elements of initial proposal, obtained revised cost proposal, and had it reviewed by Defense Contract Audit Agency for realism and possible understatement; source selection authority considered the information in making her award determination; and protester has not shown that upward cost adjustments were warranted. DECISION Abt Associates Inc. protests the award of a contract to the National Opinion Research Center (NORC) under request for proposals (RFP) No. 2004-N-01001, issued by the Centers for Disease Control (CDC), Department of Health and Human Services, for the planning and conducting of health-related surveys. Abt challenges the evaluation of the technical proposals and the evaluation of NORC's cost proposal. We deny the protest. The RFP, which contemplated the award of a cost-reimbursement plus fixed-fee contract, sought proposals to plan and conduct the National Immunization Survey (NIS) and State and Local Area Integrated Telephone Survey (SLAITS) for data years 2005-2007, with options to continue data collection for 2008-2009.[1] The NIS is a random digit dialing (RDD) telephone survey designed to obtain quality and timely data pertaining to the immunization coverage of pre-school children across the United States. SLAITS is a broad-based, ongoing surveillance system available at the state and local levels for tracking and monitoring the health and well-being of children and adults. SLAITS uses the same sampling frame as the NIS, and immediately follows the NIS in selected households, using its sample for efficiency and economy. The contractor is to design and conduct quarterly/annual surveys to collect data from the families of age-eligible children in RDD interviews; collect data from medical providers reported for those children; combine the information from the responding providers and families to produce improved vaccination estimates; perform descriptive analyses of the data collected; and prepare and document computer files of the survey data. Proposals were to be evaluated on the basis of six factors--methodology and approach (50 of 100 available points); personnel/staff expertise and experience (25A points); company experience, past performance and capability (10 points); management plan (10 points); facilities and equipment (5 points); and cost. Cost was to be evaluated for cost realism. Technical strength was considered more important than cost and award was to be made to the offeror whose proposal represented the "best value" to the government. Three offerors, including Abt and NORC, submitted proposals, which were evaluated by the technical evaluation panel (TEP). Based on the results of the initial evaluation, Abt's and NORC's proposals were included in the competitive range. The agency conducted written discussions with Abt and NORC, and obtained a final proposal revision (FPR) from each. After the individual members of the TEP reviewed and scored each of the initial and revised proposals, the TEP met to reach a consensus by reviewing all identified strengths and weaknesses and compiling, verifying, and averaging the individual evaluators' scores. The agency also obtained revised cost proposals and had them reviewed by the Defense Contract Audit Agency (DCAA) for cost realism. The results of the final consensus evaluation are as follows: +------------------------------------------------------------------------+ |A |Abt |NORC | |--------------------------------------+---------------+-----------------| |Methodology/Approach (50) |43 |46.75 | |--------------------------------------+---------------+-----------------| |Personnel/Staff/Experience (25) |22 |23.25 | |--------------------------------------+---------------+-----------------| |Co. Exp./Past Perf./Capability (10) |7.25 |8.25 | |--------------------------------------+---------------+-----------------| |Management Plan (10) |8.25 |9 | |--------------------------------------+---------------+-----------------| |Facilities/Equipment (5) |3.75 |5 | |--------------------------------------+---------------+-----------------| |Total (100) |84.25 |92.25 | |--------------------------------------+---------------+-----------------| |Technical Rating (overall) |Outstanding |Superior | |--------------------------------------+---------------+-----------------| |Initial Proposed Cost |[deleted] |[deleted] | |--------------------------------------+---------------+-----------------| |Revised Proposed Cost |$77,422,221 |$60,322,322 | +------------------------------------------------------------------------+ Based on the DCAA review and its own evaluation of NORC's proposed cost, the agency concluded that the firm could satisfactorily perform at its revised proposed cost. Based on this and the outcome of the technical evaluation, the contracting officer, as source selection authority, concluded that NORC's proposal represented the best value to the agency and awarded it the contract. After receiving a debriefing, Abt filed this protest.[2] MEANINGFUL DISCUSSIONS Abt asserts that its proposal was improperly downgraded based on various weaknesses identified by the agency under criteria 1 (methodology and approach) and 3 (personnel/staff expertise and experience) because the firm was not notified of the weaknesses in discussions. In this regard, when an agency engages in discussions with an offeror, the discussions must be meaningful, that is, they must lead the offeror into the areas of its proposal that require correction or amplification. Hanford Envtl. Health Found., B-292858.2, B-292858.5, Apr. 7, 2004, 2004 CPD P 164 atA 8. Our Office will not sustain a protest unless the protester demonstrates a reasonable possibility that it was prejudiced by the agency's actions, that is, unless the protester demonstrates that, but for the agency's actions, it would have had a substantial chance of receiving the award. McDonald-Bradley, B-270126, Feb. 8, 1996, 96-1 CPD P 54 at 3; see Statistica, Inc. v. Christopher, 102 F.3d 1577, 1581 (Fed. Cir. 1996). While the precise impact from the agency's failure to provide discussions in this area is unclear, it is clear that Abt was not prejudiced. Abt's consensus score for criterionA 1 was 43 out of 50 possible points. Even if Abt, through discussions, were able to improve its proposal score to the maximum 50 points available under the criterion, this would have the effect of increasing Abt's overall score of 84.25 by only 7 points, to 91.25, which would still be lower than the awardee's score of 92.25. Since the awardee's proposed price also was low (by a substantial amount), this increase in Abt's technical score would not change the outcome of the competition. Thus, any failure by the agency to provide adequate discussions in this area did not prejudice Abt. Criterion 3 Abt asserts that the agency failed to provide it with the opportunity to respond to several past performance weaknesses identified by the evaluators. The agency explains that the weaknesses, identified by two of the evaluators, were based on comments on some of Abt's past performance questionnaires. The agency ultimately determined, however, that since Abt's numerical scores on the questionnaires were high, there was no need for a response from Abt. Although the weaknesses remained in the evaluation, the agency explains that the only negative impact was the possible deduction of one point by each of the two evaluators who had expressed concern. Supplemental Agency Report (AR) at 26. Based on the agency's averaging of evaluator scores in the consensus, restoration of the two deducted points would only increase Abt's score by one-half point.[3] Adding this one-half point to Abt's score--as adjusted above under criterion 1--its overall score would be 91.75, still below NORC's overall score. The agency's explanation is consistent with the record, and since Abt has not shown otherwise, we conclude that, given NORC's still higher technical rating and lower cost, Abt was not prejudiced by any error in its past performance evaluation.[4] [5] In reviewing a protest of an agency's proposal evaluation, our review is confined to a determination of whether the agency acted reasonably and consistent with the terms of the solicitation and applicable statutes and regulations. United Def. LP, B-286925.3 et al., Apr.A 9, 2001, 2001 CPD P 75 at 10-11. A protester's mere disagreement with the agency's judgment is not sufficient to establish that the agency acted unreasonably. Command Mgmt. Servs., Inc., B-292893.2, June 30, 2004, 2004 CPD PA 168 at 3. We have reviewed all of Abt's challenges to the evaluation under these criteria and find that none has merit. For example, evaluation criterion 2 covered experience with complex sample surveys, RDD surveys, statistical modeling of non-sampling errors, the design and conduct of record check studies and, as to the education and experience of key personnel, covered publications in these areas, along with specific professional, technical, or scientific accomplishments. RFP at 86. In its initial evaluation, the agency noted that Abt lacked expertise in [deleted] or recognition of when [deleted] needs to be accounted for in a survey's methodology. In discussions, the agency asked Abt to provide additional documentation in support of this expertise. In response, Abt stated that its team possessed [deleted] in the area, summarized the relevant expertise of its team, referred the agency to the original proposal's "extensive listing of papers related to nonsampling errors" authored or co-authored by its staff, and summarized the expertise of its staff members. Abt FPR at 48-49. In Abt's view, its initial proposal and FPR response provided sufficient information to make this identified weakness unreasonable. Abt Initial Comments at 13. However, upon review of Abt's FPR, while the agency acknowledged that Abt had listed its key personnel with their areas of expertise in [deleted] areas, it noted that there was no listing of [deleted] publications in these areas to substantiate the claims of relevant expertise. AR, Tab 8, at 7. In the agency's view, [deleted] publications represent the "gold standard" because the reviewers provide an external and objective evaluation of scientific work. Evaluator Declaration at 3-4. We find nothing unreasonable in the agency's evaluation. The RFP clearly advised offerors that key personnel were to have specific expertise in various areas, including non-sampling errors. While Abt's initial proposal and FPR stated that the key personnel had the requisite expertise, Abt did not provide the additional documentation requested in discussions. In this regard, apart from a description of its personnel's expertise, lists of publications, and the title of one book, neither Abt's initial proposal and FPR, nor its protest submissions, identify which publications document its personnel's expertise in the questioned area. In the absence of the requested documentation, the agency reasonably identified this as a weakness under criterion 2. As another example, with regard to evaluation criterion 3, offerors were required to provide their organizational experience and capacity to acquire qualified staff and to recruit expert consultants in the disciplines essential to the procurement. In addition, offerors were required to express that capacity in instances where it became essential to obtain expert consultants or qualified subcontractors within a very short period of time. RFP at 87. In its initial evaluation, the agency found that Abt had not proposed "deep-water" technical experts on staff in statistical methods for sample surveys. AR, Tab 4, at 10. Noting that Abt had proposed three researchers from universities to serve as potential consultants, the agency expressed concern as to whether these personnel would be available on an as-needed basis, since they did not work for Abt and were documented to have many other commitments. Id. In discussions, the agency pointed to the lack of "deep water" technical experts and asked whether the firm would be able to provide them on an as-needed basis. AR, Tab 6, at 6. In responding to the agency's concern in its FPR, Abt stated that its outside consultants [deleted], and reiterated its proposal of four academically-based experts, each of whom had agreed to provide the NIS with [deleted] percent of a full-time equivalent (FTE) employee. Abt FPR at 56. Upon review of Abt's response, the agency noted that the revised proposal did not address the availability of the consultants on an as-needed basis. AR, Tab 8, at 8. In Abt's view, its response was adequate. The evaluation in this area was reasonable. Offerors were required to demonstrate their capacity to acquire qualified staff and/or expert consultants within a very short period of time, i.e., on an as-needed basis. While Abt's FPR reiterated its agreement with its consultants and the commitment of each to provide the equivalent of [deleted]A percent of an FTE, it did not address whether or how these consultants, employed elsewhere, would be able to make themselves available on an as-needed basis. A commitment to provide a consultant for a certain percentage of an FTE does not equate with availability on short notice. The agency thus reasonably identified this as a weakness in Abt's proposal under criterion 3. Id. at 4. The evaluation of NORC's proposal under criteria 2 and 3 was unobjectionable. The agency explains that it was aware of the depth and quality of NORC's experience in conducting [deleted] studies and considered it in its evaluation. In this regard, during discussions, the agency specifically asked NORC to provide additional information to support its ability to provide [deleted] survey of the requisite magnitude. In response to this and related questions, NORC submitted some 12 pages of information, plus resumes that included descriptions of its [deleted] extensive experience with other projects while working for different companies, including Abt. Contracting Officer's Statement at 23; NORC FPR at 78-90. Although the weakness was still reflected in the FPR evaluation, the agency added as a strength that NORC [deleted] had experience in surveys much larger than the NIS--including the [deleted]--and other CDC surveys as large as the NIS. AR, Tab 8. The agency also noted as a strength under criterion 3 that, apart from the NIS, [deleted] is one of the largest ongoing [deleted] studies in the U.S. NORC Evaluation at 4. In view of the fact that evaluation criteria 2 and 3 encompassed more than just [deleted] experience, and that NORC's proposal received less than perfect consensus scores under these criteria--reflecting NORC's limited [deleted] experience--there is no basis to conclude that the evaluation did not accurately reflect NORC's actual experience; thus, there is no basis to conclude that the agency's evaluation of NORC's proposal was flawed. Abt asserts that the agency did not conduct a reasonable analysis of NORC's proposed costs. In this regard, it asserts that DCAA's cost realism review was inadequate because it did not discuss whether NORC's proposed cost elements were realistic for the work to be performed, reflected a clear understanding of the requirements, and were consistent with the offeror's unique methods of performance and materials. Abt also notes that the agency failed to calculate NORC's probable cost. When an agency evaluates proposals for the award of a cost-reimbursement contract, an offeror's proposed estimated costs are not considered controlling because, regardless of the costs proposed, the government is bound to pay the contractor its actual allowable costs. Federal Acquisition Regulation SS 15.305(a)(1), 15.404-1(d). Our review of an agency's cost realism analysis is limited to whether the analysis is reasonably based and is not arbitrary. Systems Integration & Research, Inc., et al., B-279759.2 et al., Feb. 16, 1999, 99-1 CPD PA 54 at 7-8. In this regard, an agency is not required to conduct an in-depth cost analysis or to verify each and every item in assessing cost realism; rather, the evaluation requires the exercise of informed judgment by the contracting agency, which is in the best position to make a realism determination. ITT Indus., Inc., B-294389 et al., Oct. 20, 2004, 2004 CPD PA 222 at 14. The cost realism analysis here was reasonable. The RFP defined cost realism as "the offeror's ability to project costs which are reasonable and indicate that the offeror understands the nature and extent of the work to be performed." RFP SA M.1. To this end, in evaluating NORC's initially proposed costs (approximately $[deleted]), the agency questioned various costs, including the firm's [deleted] costs for the first year, [deleted], and overall high [deleted]. Summary of Negotiations at 8-10. NORC responded to each concern in its revised proposal, and lowered its overall cost by some $[deleted]. DCAA reviewed NORC's revised cost proposal, including its direct labor rates, labor escalation rates, fringe benefit rates, general and administrative (G&A) rate, as well as NORC's rates overall, for realism and possible understatement. DCAA Cost Report at 1. DCAA discussed each of these areas in detailed notes and concluded that, with limited exceptions (not challenged by Abt), all rates were consistent with NORC's current rates and were not understated.[6] Id.A atA 3-6. Although the agency was concerned that NORC had significantly lowered its proposed cost, it considered NORC's explanation for doing so to be reasonable. Specifically, NORC explained that the downward revision was based on the agency's discussion comments indicating [deleted] initial price; agency concerns about the staff mix; and agency requests for elimination of costs for innovative ideas. Award Determination at 2. In addition, NORC reduced its [deleted] costs by adjusting its [deleted]; reduced its [deleted]; and reduced the [deleted]. Id. While Abt asserts that the agency failed to determine NORC's probable costs after reviewing these justifications, it does not identify any specific adjustments it believes were required. In addition, the agency considered its history with NORC under other CDC contracts that had been subjected to cost realism analyses and DCAA audits. The agency notes that, after more than a decade of experience with CDC-performed evaluations of NORC business proposals, it has not found "material questioned cost." CDC Letter, Jan. 13, 2005. Based on its and DCAA's review of NORC's costs and its own history with the firm under other cost contracts, the agency concluded that there was no reason to believe that NORC's costs were not achievable, or that the firm did not understand the work to be performed. Contracting Officer's Statement at 27. The contracting officer thus concluded that NORC was able to satisfactorily perform under the contract at its revised cost amount. Award Determination atA 2. Abt notes that DCAA did not review NORC's ODCs--which accounted for approximately [deleted] percent of its proposed costs--for realism. Again, however, an agency is not required to conduct an in-depth cost analysis or to verify each and every item in conducting a cost realism analysis. ITT Indus., Inc., supra. The agency notes that, while DCAA did not review those costs, the realism analysis covered the other [deleted] percent of the proposal. The agency further explains that, based on its prior history with NORC's cost-reimbursement contracts, it did not believe that this omission necessitated an adjustment to its analysis. CDC Letter, Jan. 13, 2005. Abt has not identified any aspect of NORC's ODCs that were understated. In fact, it has not suggested any adjustment that should have been made to any aspect of NORC's proposed cost despite its access to the firm's initial and revised cost proposals. Under these circumstances, despite the fact that no adjustments were made to NORC's proposed cost, we find no basis for objecting to the agency's cost realism evaluation. See PAE GmbH Planning and Constr., B-250470, Jan.A 29, 1993, 93-1 CPD PA 81 at 6-7 (fact that no cost adjustments resulted from realism evaluation provides no basis for objection where cost evaluation was otherwise reasonable). The protest is denied. Anthony H. Gamboa General Counsel ------------------------ [1] A data year is 18 months long--1 calendar year, January 1 through December 31, plus 6 months of the following year. [2] Abt raises numerous issues in its protest. We have considered them all and find that they have no merit or did not prejudice the protester. This decision addresses the more significant of Abt's arguments. [3] That is, due to the averaging of scores in the consensus process, 2 raw points, when added to the other two evaluators' totals, would be divided by 4, resulting in a net increase of one-half point. [4] We note, moreover, that it is extremely doubtful that the source selection would change even if Abt received the maximum score of 10 under criterion 3; even though this would increase its overall consensus score to 94 points, slightly above NORC's, there is no reason to believe that this minor scoring advantage would be found to offset NORC's significant cost advantage. [5] In a related argument, Abt asserts that the agency did not adequately document the evaluation of its proposal. This assertion is without merit. The record contained individual and consensus score sheets, along with comprehensive lists of Abt's strengths and weaknesses under each evaluation criterion. This documentation is sufficient to allow us to review the bases for the agency's evaluation conclusions. [6] DCAA identified one [deleted] rate as being [deleted] than the one NORC had included under a generic title. Abt does not assert that any specific adjustment should be made to NORC's cost proposal to account for this. Our own review discloses no reason to adjust for this cost; the [deleted] was listed elsewhere in NORC's revised cost proposal with the appropriate [deleted] rate.