BNUMBER: B-278839
DATE: March 20, 1998
TITLE: I.S. Grupe, Inc., B-278839, March 20, 1998
**********************************************************************
Matter of:I.S. Grupe, Inc.
File: B-278839
Date:March 20, 1998
Peter B. Schipma for the protester.
Michael Colvin, Department of Health & Human Services, for the agency.
Linda S. Lebowitz, Esq., and Michael R. Golden, Esq., Office of the
General Counsel, GAO, participated in the preparation of the decision.
DIGEST
Agency's decision not to fund the protester's proposal for a Phase II
effort under a Small Business Innovation Research program procurement
was not objectionable where the record supported the agency's
conclusion that the protester's proposal was technically unacceptable.
DECISION
I.S. Grupe, Inc. protests the decision by the Agency for Health Care
Policy and Research, Department of Health & Human Services (HHS), not
to award it a Phase II contract under request for proposals (RFP) No.
AHCPR-97-0021 for a project the firm proposed under the agency's Small
Business Innovation Research (SBIR) program. The protester contends
that the agency improperly determined that its proposal, entitled
"Internet Multimedia Cancer Patient Education System," was technically
unacceptable and would not continue to be funded.
We deny the protest.
The RFP was issued on July 10, 1997, under the SBIR program. This
program was established under the Small Business Innovation
Development Act of 1982, 15 U.S.C. sec. 638 (1994), which requires
certain federal agencies, including HHS, to establish SBIR programs.
Under these programs, the agencies reserve a statutory percentage of
their research and development budgets for award to small business
concerns for research or research and development through a
three-phase process. The purpose of the work performed under Phase I
is to determine the scientific or technical merit and feasibility of
ideas submitted under the SBIR program; under Phase II (awarded on the
basis of Phase I results), to identify the potential for yielding a
product or process of continuing interest to the agency; and under
Phase III (involving private capital), to pursue commercial
applications of the research or development. The funding vehicle for
this HHS SBIR program in both Phase I and Phase II was contracts,
rather than cooperative agreements or grants.
The RFP provided the following five Phase II evaluation factors and
respective weights: (1) the degree to which the Phase I objectives
were met and feasibility demonstrated--25 percent; (2) the
scientific/technical merit of the proposed Phase II research,
including the adequacy of the objectives for addressing the
problem/opportunity--35 percent; (3) the qualifications of the
principal investigator, supporting staff, and consultants--20 percent;
(4) the potential of the proposed research for technological
innovation--15 percent; and (5) the adequacy and suitability of the
facilities and research environment--5 percent. Under the RFP, cost
reimbursement contracts would be awarded to some or all offerors
submitting technically acceptable proposals.
The three firms, including the protester, which previously had been
awarded Phase I contracts, submitted Phase II proposals in response to
this RFP. The basis for the protester's proposal was the completion
of an interactive multimedia educational system for cancer patients
which would allow an individual patient to determine the materials
relevant to his/her medical problem and treatment. Proposals were
evaluated by a panel of six individuals in accordance with the
referenced evaluation scheme. Of relevance to this protest, one of
the evaluators (whom the agency designated as evaluator No. 5) was a
medical doctor and the chief executive officer of a media firm
specializing in medical information for consumers. Each panel member
initially assigned points to each offeror's proposal and provided
narratives of the strengths and weaknesses in these proposals. The
evaluators then met as a group, discussed each proposal, and modified
individual scores and narratives in light of these discussions. The
average of the individual scores for an offeror's proposal became the
consensus score for that proposal.
The evaluators concluded that the protester's proposal, which was
accompanied by a videotape showing "not the prototype product, but
only a view of the prototype product," and which would not allow the
viewer to actually interact with the educational system depicted, was
technically unacceptable (consensus score--55.3 points out of a
possible 100). The agency ultimately awarded Phase II contracts to
the other two offerors, whose proposals were determined technically
acceptable (consensus scores--84.7 points and 79.2 points).
The protester challenges the evaluation of its proposal, primarily
expressing disagreement with the evaluators' conclusions concerning
the technical merit of the proposal. The protester first complains
about the evaluators' viewing of the videotape, which accompanied the
firm's Phase II written proposal.[1] The videotape contained a
partial demonstration of the operational prototype developed under the
protester's Phase I contract and incorporated a sampling of the
multimedia objects (text, audio, images, animation, and video)
generated during that project. The protester's complaint, however, is
without merit because the protester, in fact, encouraged the agency to
distribute copies of the videotape to the evaluators for viewing
during the evaluation process. Specifically, the protester stated in
the cover letter to its proposal that
[w]e have also enclosed . . . copies of a videotape which you may
want to distribute with the copies of the proposal. We
understand and accept that such distribution is at your sole
option and that the proposal in text form represents [the firm's]
response to your RFP. . . . Since we are working with a
multimedia presentation, it is often easier to show things than
to describe them in words, which is the reason for including the
videotape. The tape shows some selected multimedia segments from
the prototype as examples to support the proposal text. Only a
commonplace VCR is needed to view the videotape, rather than a
computer system. We believe that the videotape is a useful
adjunct to the proposal, and we hope that you elect to distribute
copies of the tape to the reviewers as well as copies of the
text. [Emphasis added.]
In addition, in the proposal itself, the protester requested that the
evaluators "please view the videotape sent with the Phase II proposal
to experience the real Phase I result." Simply stated, to accommodate
the protester's requests, the evaluators reasonably viewed the
videotape in conjunction with the evaluation of the protester's Phase
II written proposal.
The protester next complains that the evaluators, particularly
evaluator No. 5, improperly downgraded the overall technical merit of
its proposal. The protester maintains that evaluator No. 5 not only
assigned an unreasonably low number of points to its proposal based in
large measure on his views of the content and quality of the
videotape, but also caused the other evaluators to lower their initial
scores assigned to the firm's proposal, with the cumulative effect
that the overall consensus score for its proposal was improperly
skewed.
Because of the experimental and creative nature of an SBIR
procurement, which is not based on design or performance
specifications for existing equipment, but rather emphasizes
scientific and technical innovation and has as its objective the
development of new technology, the contracting agency is given
substantial discretion in determining which proposals it will fund.
Quantum Magnetics, Inc., B-257968, Nov. 30, 1994, 94-2 CPD para. 215 at 4.
In light of the agency's discretion, we limit our review of awards
under SBIR procurements to determining whether the agency violated any
applicable regulations or solicitation provisions, or acted in bad
faith. Virginia Accelerators Corp., B-271066, May 20, 1996, 97-2 CPD para.
13 at 2; Bostan Research, Inc., B-274331, Dec. 3, 1996, 96-2 CPD para. 209
at 2. Here, the agency's conclusion that the protester's proposal was
technically unacceptable is supported by the record.
The record shows that evaluator No. 5 was a medical doctor and the
chief executive officer of a media firm specializing in medical
information for consumers. As such, he used his particular expertise
in two areas--medicine and media--as the basis for evaluating the
merits of the protester's technical proposal. While this evaluator
believed that the theme of the protester's proposal--providing timely
medical and treatment information to cancer patients--was the
proposal's greatest strength, he nevertheless found weaknesses in the
underlying details of the proposal. Evaluator No. 5 assigned 18 out
of a possible 100 points to the protester's proposal, and provided
narratives of the strengths and weaknesses in the firm's proposal.
For example, evaluator No. 5 noted as a weakness that after viewing
the videotape, he was concerned that the protester did not demonstrate
a sensitivity to the needs of health care consumers--the target
audience--from the perspective of language used. This evaluator
referenced a tenth grade literacy level as the goal mentioned by the
protester for this project, and noted that the language appeared to be
at a pre-medical literacy level. The protester objects to these
characterizations contending that if this evaluator had read the
firm's Phase II written proposal, he would have seen the statement
that a "sixth grade reading level is a reasonable comprehension target
level for typical patients." However, in the Phase I Final Report, a
required section of the protester's Phase II proposal, the protester
included a section entitled "Operational Reviews." In that section,
the protester stated that "[a]lmost all the consultants who viewed the
operational prototype system [i.e., the videotape] were concerned
about the intended target audience for the material and the
organization of the presentation. . . . [S]ome reviewers found
inconsistency with regard to the educational level assumed for the
target audience." More specifically, the protester referenced the
statement of a doctor/consultant for the project who stated that
"'portions of the text seem comprehensible by a Grade 10 or higher
audience while other portions seem to be very much like a medical
school text.'" In light of the statement of the protester's own
doctor/consultant, we cannot conclude that evaluator No. 5 was
unreasonable in his assessment that the videotape appeared to be
directed to a more educated audience.
Evaluator No. 5 also listed as a weakness that in preparing the
videotape, the protester neglected to utilize the skills of those who
clearly could have contributed to making the interactive multimedia
educational system a higher quality product, for example, medical
illustrators and instructional designers. The protester did its media
work in-house in order to save money. The protester stated in its
proposal that "[w]e did animation, both two-dimensional and
three-dimensional internally. . . without the services of a graphic
artist. . . . We did audio internally. . . . We imported graphics, and
we both imported and internally created full motion video. Production
values were high throughout, yet no expensive studio facilities were
used." However, given the medical and media background of evaluator
No. 5, we believe he could reasonably downgrade the protester's
proposal for failing to utilize the skills of various media and other
professionals who could have created a more professional, interactive
multimedia project to educate cancer patients. The evaluator stated
that "in his eyes," the protester's failure to use media professionals
"sever[ely] compromis[ed] the integrity and value of the effort . . .
."
Accordingly, we believe the record supports evaluator No. 5's
downgrading of the protester's proposal because of weaknesses in the
proposal details for completing an interactive multimedia cancer
education system. Technical evaluators have considerable latitude in
assigning ratings which reflect their subjective judgments of a
proposal's relative merits. MiTech, Inc., B-275078, Jan. 23, 1997,
97-1 CPD para. 208 at 5. A protester's mere disagreement with the
particular point scores awarded to its proposal does not render the
evaluation unreasonable. Id.
We also point out that the other evaluators expressed concerns similar
to those raised by evaluator No. 5, noting, for example, that the
protester used very advanced and medical language; the protester
assumed a very high literacy level for its audience; and the protester
failed to utilize the professional services of a graphic artist, an
audio/visual firm, and an instructional designer or health educator.
In addition, the other evaluators believed the protester did not
clearly address in its proposal the target audience, that is, cancer
patients, as the firm failed to spend sufficient time testing
materials with this audience. Since the project was intended to
educate cancer patients, the evaluators believed the protester had to
focus on what a patient's needs for cancer education were, for
example, by addressing treatment alternatives as experienced by actual
patients. One of the evaluators even noted that a patient educator
was needed as a key person for this project. Finally, with respect to
the videotape, the evaluators commented that the information presented
was unfocused and uneven, and the doctor who did most of the videotape
presentation sounded like a "talking head."
In sum, while evaluator No. 5, as well as the other evaluators,
believed that the concept proposed by the protester to educate
consumers with respect to cancer treatments through the use of
interactive multimedia techniques was an overall strength of the
firm's proposal, the evaluators nevertheless determined there were
many weaknesses in the firm's proposed approach, as evidenced by its
written proposal and accompanying videotape. While the protester
expresses disagreement with the weaknesses noted by the evaluators,
the protester has failed to show that these weaknesses were not
supported by the record.
Finally, to the extent the protester argues that evaluator No. 5
unfairly influenced and persuaded the other evaluators to lower the
scores initially assigned to the protester's proposal, thereby
resulting in a lower overall consensus score assigned to the firm's
proposal, we point out that agency evaluators may discuss their
individual evaluations with each other in order to reach a valid
consensus score since such discussions generally operate to correct
mistakes or misperceptions that may have occurred in the initial
evaluation. Wastren, Inc., B-276093, May 12, 1997, 97-1 CPD para. 175 at
3 n.2. A consensus score need not be the score the majority of the
evaluators initially awarded; a score may reasonably be determined
after discussions among the evaluators. The overriding concern in
these matters is whether the final scores assigned accurately reflect
the relative merits of the proposals. Id. In this case, based upon
the record, including the protester's written Phase II proposal and
the videotape, we have no basis to question the agency's decision not
to continue to fund the protester's project because the proposal
submissions were reasonably determined technically unacceptable.
The protest is denied.
Comptroller General
of the United States
1. Our Office viewed a copy of the videotape, which lasted
approximately 36 minutes.