Contract Management: Benefits of the DOD Mentor-Protege Program
Are Not Conclusive (19-JUL-01, GAO-01-767).
To boost the participation of small disadvantaged businesses as
subcontractors and suppliers under the Department of Defense
(DOD) contracts, Congress authorized the Pilot Mentor Protege
Program. The program provides incentives for major defense
contractors (mentors) to assist small disadvantaged businesses
(proteges) in strengthening their ability to compete for work.
However, DOD has been criticized in the past for not establishing
compelling evidence about the program's overall effectiveness. To
ensure that the program is focused on a results-oriented approach
to assessing program performance, Congress asked DOD to obtain
information that could be used to assess the program's
effectiveness. This report reviews (1) the relationship between
the results of the Mentor-Protege Program and the statutory goal
of awarding five percent of the total dollar amount contracted by
DOD and subcontracted by DOD prime contractors to small
disadvantaged businesses, (2) whether the Mentor-Protege Program
enhanced the business competitiveness, financial independence,
and business development of protege firms, and (3) whether
program funds had been used as an effective incentive for mentor
firms to participate in the program. GAO found that DOD lacks
data integral to assessing the success of the Mentor-Protege
Program. DOD does not have sufficient information to determine
the relationship between the program and the goal of awarding
five percent of the total dollar amount contracted to small
disadvantaged businesses. Although DOD has consistently achieved
this goal since 1992, the overall contribution of the
Mentor-Protege Program in achieving this goal is unknown. DOD
also does not have sufficient information to assess whether the
Mentor-Protege Program enhanced the business competitiveness,
financial independence, and business development of protege
firms. Although there are small disadvantaged business program
participants that have realized increased success, data is not
available to attribute this success to the program. Program funds
have been used to encourage major defense contractors to provide
developmental assistance for small disadvantaged businesses.
However, data is not available to determine whether program funds
are needed to continue to encourage major defense contractors to
establish business relationships with small disadvantaged
businesses.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-767
ACCNO: A01416
TITLE: Contract Management: Benefits of the DOD Mentor-Protege
Program Are Not Conclusive
DATE: 07/19/2001
SUBJECT: Subcontractors
Performance measures
Program evaluation
Department of Defense contractors
Small disadvantaged business contractors
DOD Pilot Mentor-Protege Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-767
GAO
July 2001 CONTRACT MANAGEMENT
Benefits of the DOD Mentor- Prot�g� Program Are Not Conclusive
GAO- 01- 767
Report to Congressional Requesters
United States General Accounting Office
Page i GAO- 01- 767 Contract Management Letter 1
Results in Brief 2 Background 3 DOD Exceeds 5- Percent Goal, but Program?s
Contribution Cannot
Be Fully Determined 4 No Explicit Link Between Mentoring and Prot�g�s?
Competitiveness, Financial Independence, or Business Development 5 Effect of
Program Funding on Defense Contractor Participation Is
Unknown 6 Conclusions 6 Matter for Congressional Consideration 7
Recommendations 7 Agency Comments and Our Evaluation 7 Scope and Methodology
8
Appendix I Performance Measurements Reported in the September 1999 Semi-
Annual Reports 10
Appendix II Comments From the Department of Defense 12
Appendix III GAO Contact and Staff Acknowledgments 15
Tables
Table 1: Distribution of Prot�g�s and the Value of Contracts Received 10
Table 2: Distribution of Prot�g�s and Change in Revenue 11 Table 3:
Distribution of Prot�g�s and Change in the Number of
Employees 11
Figure
Figure 1: Percentage of Total Dollar Value of DOD Contract Awards and
Subcontracts Awarded to Small Disadvantaged Businesses 4 Contents
Page 1 GAO- 01- 767 Contract Management
July 19, 2001 The Honorable Carl Levin Chairman The Honorable John Warner
Ranking Minority Member Committee on Armed Services United States Senate
The Honorable Bob Stump Chairman The Honorable Ike Skelton Ranking Minority
Member Committee on Armed Services House of Representatives
To boost the participation of small disadvantaged businesses as
subcontractors and suppliers under Department of Defense (DOD) contracts,
the Congress authorized 1 the Pilot Mentor- Prot�g� Program. The program
provides incentives for major defense contractors (mentors) to assist small
disadvantaged businesses (prot�g�s) in strengthening their ability to
compete for work (for example, by providing computer training or help in the
areas of proposal writing and contract administration). However, DOD has
been criticized in the past for not establishing compelling evidence about
the program?s overall effectiveness.
To ensure that the program is focused on a results- oriented approach to
assessing program performance, the Congress asked DOD to obtain information
that could be used to assess the program?s effectiveness. The Congress also
mandated us to study the program?s effectiveness. 2 In responding to this
mandate, we
reviewed the relationship between the results of the Mentor- Prot�g� Program
and the statutory goal 3 of awarding 5 percent of the total
1 Section 831 of the National Defense Authorization Act for Fiscal Year 1991
(P. L. 101- 510, 10 U. S. C. 2302 Note). 2 Section 811 of the National
Defense Authorization Act for Fiscal Year 2000 (P. L. 106- 65).
3 10 U. S. C. 2323.
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 01- 767 Contract Management
dollar amount contracted by DOD and subcontracted by DOD prime contractors
to small disadvantaged businesses;
assessed whether the Mentor- Prot�g� Program enhanced the ?business
competitiveness, financial independence, and business development of prot�g�
firms?; and
evaluated whether program funds had been used as an effective incentive for
mentor firms to participate in the program.
In performing our work, we relied primarily on data the DOD Mentor- Prot�g�
Program Office collected. Program Office officials said that, in response to
our last review, 4 that data they collected would be reliable and sufficient
enough to evaluate program success. A complete statement of our methodology
begins on page 8 of this report.
DOD lacks data integral to assessing the success of the Mentor- Prot�g�
Program. Specifically, DOD does not have sufficient information to
explicitly
Determine the relationship between the program and the goal of awarding 5
percent of the total dollar amount contracted to small disadvantaged
businesses. Although DOD has consistently achieved this goal since 1992, the
overall contribution of the Mentor- Prot�g� Program in achieving this goal
is unknown.
Assess whether the Mentor- Prot�g� Program enhanced the business
competitiveness, financial independence, and business development of prot�g�
firms. Although there are small disadvantaged business program participants
that have realized increased success, data is not available to attribute
this success to the program.
Program funds have been used to encourage major defense contractors to
provide developmental assistance for small disadvantaged businesses.
However, data is not available to determine whether program funds are needed
to continue to encourage major defense contractors to establish business
relationships with small disadvantaged businesses.
4 Defense Contracting: Sufficient, Reliable Information on DOD?s Mentor-
Prot�g� Program Is Unavailable (GAO/ NSIAD- 98- 92, Mar. 30, 1998). Results
in Brief
Page 3 GAO- 01- 767 Contract Management
This report recommends that DOD better demonstrate the benefits of the
Mentor- Prot�g� Program. In commenting on a draft of this report, DOD
partially concurred with our recommendations as discussed on page 7. DOD?s
written comments are in appendix II.
In 1986, congressional concern about the low participation of small
disadvantaged businesses within DOD?s procurement system resulted in section
1207 of the National Defense Authorization Act for Fiscal Year 1987 (P. L.
99- 661, later codified as amended at 10 U. S. C. 2323). This section
established a goal of awarding 5 percent of the total dollar amount
contracted by DOD and subcontracted by DOD prime contractors to small
disadvantaged businesses. The 5- percent goal was not met in the years
immediately following its establishment because, according to large DOD
prime contractors, there were not enough qualified small disadvantaged
businesses available as DOD subcontractors. Subsequently, the National
Defense Authorization Act for Fiscal Year 1991 established the Pilot Mentor-
Prot�g� Program. The purpose of the program is to provide incentives for
major DOD contractors (mentors) to furnish small disadvantaged businesses
(prot�g�s) with assistance designed to enhance their capabilities and
increase their participation as subcontractors and suppliers under DOD
contracts, other federal government contracts, and commercial contracts.
Under the Mentor- Prot�g� Program, prot�g�s are to benefit from
developmental assistance provided by mentors in such areas as (1)
infrastructure development, including organizational, financial, and
personnel management; proposal writing; contract administration; and overall
general business development; and (2) technology transfer such as, training
in the areas of production, quality control, manufacturing, engineering,
computer hardware and software, and assistance in obtaining production and
accounting certifications needed to work on large DOD contracts. Mentors are
to benefit from a strengthened cadre of subcontractors and DOD is to benefit
from a resultant robust and competitive supplier base. The metrics DOD uses
to measure program performance are presented in appendix I.
Since the Pilot Mentor- Prot�g� Program was authorized, it has been
continuously extended and, with the Fiscal Year 2000 National Defense
Authorization Act, DOD is now authorized to approve new agreements through
September 30, 2002. Program performance is authorized through September 30,
2005. Background
Page 4 GAO- 01- 767 Contract Management
Through fiscal year 2000, the Congress has appropriated $287 million for the
program. Funding covers program operational costs and reimburses mentors for
providing developmental assistance. Mentor- prot�g� agreements are now
limited to a 3- year period, with the expectation that the developmental
assistance provided will allow prot�g�s to grow their businesses.
Figure 1 shows that DOD has achieved or exceeded its 5- percent goal of
awarding prime contracts and subcontracts to small disadvantaged businesses
since 1992. However, DOD has not been able to attribute this success to the
program.
Figure 1: Percentage of Total Dollar Value of DOD Contract Awards and
Subcontracts Awarded to Small Disadvantaged Businesses
While data contained in figure 1 indicates success in bringing small
disadvantaged businesses into DOD contract work, the contribution of the
Mentor- Prot�g� Program to this achievement is not fully known, in part,
because DOD does not fully track the contributions of the 360 former
prot�g�s in meeting the goal. The requirement to track the progress of
former prot�g�s was established by the Fiscal Year 2000 National Defense
Authorization Act, for those agreements entered into on or after October 1,
1999. Because agreements can last 3 years, post- agreement reporting may not
start until October 1, 2002. DOD Exceeds
5- Percent Goal, but Program?s Contribution Cannot Be Fully Determined
Page 5 GAO- 01- 767 Contract Management
Data is not available to establish a cause- and- effect relationship between
mentoring provided and prot�g�s? competitiveness, financial independence, or
business development. DOD officials contend it is difficult to establish a
cause- and- effect relationship between mentoring provided and prot�g�
success. DOD officials said prot�g�s that have successfully received mentor-
provided assistance might have graduated from the program before receiving
contracts that can be attributed to the program.
A reliable way to determine program impact is to compare the success of a
group of participants to the success of an equivalent group of non-
participants. Implicit in such a comparison is that differences found would
not have occurred without program participation. Although impact studies are
sometimes difficult and expensive, they are the most reliable way to
determine if a program, such as the Mentor- Prot�g� Program, is producing
real results and answer the question, ?Is the program making a difference??
DOD tries to measure program effectiveness by requiring mentors to report on
such performance metrics as prot�g� revenue and employee growth. Our
analysis of that data shows that prot�g� revenue increased by $456 million
and the number of prot�g� employees grew by 2,909 personnel. 5 However, the
revenue and employee growth cannot be explicitly linked to the Mentor-
Prot�g� Program. The revenue and employee changes include all changes, not
just those that may have resulted from the Mentor- Prot�g� Program. The same
holds true for the performance metrics captured by DOD?s data collection
instrument. Appendix I contains additional details on the DOD performance
metrics.
Other data problems make it further difficult to determine the success of
the program. Specifically,
Our review of the 1999 and 2000 semi- annual reports 6 submitted by
mentors revealed that many of the reports are incomplete. For example, about
41 percent of the reports have no evidence of prot�g� concurrence or
nonconcurrence, which is expected to provide
5 This growth was measured from the time a prot�g� entered the program
through September 30, 1999. 6 Mentor firms are required by DOD to report,
semiannually, on the progress made under active mentor- prot�g� agreements
for the periods ending March 31 st and September 30 th . No Explicit Link
Between Mentoring and Prot�g�s? Competitiveness, Financial Independence, or
Business Development
Page 6 GAO- 01- 767 Contract Management
corroborating support of the data submitted by mentor firms. Further, as of
January 2001, less than half of the September 2000 semi- annual reports had
been received.
To meet prot�g� business needs, mentors and prot�g�s are to jointly
fashion business development plans, known as mentor- prot�g� agreements.
These agreements outline the developmental assistance mentors are to provide
to their prot�g�s. DOD officials said they rely on the Defense Contract
Management Agency to verify that agreements have been met. We found that
over 22 percent of the Defense Contract Management Agency reviews of the
agreements had no indication of whether the mentor- prot�g� agreements were
being met.
Program funds have been used to encourage major defense contractors to
provide developmental assistance for small disadvantaged businesses.
However, sufficient data is not available to determine whether program funds
are needed to continue to encourage major defense contractors to establish
business relationships with small disadvantaged businesses.
A study completed in December 2000 for DOD by the Logistics Management
Institute (LMI) indicated that 71 percent of the 93 active mentors, at the
time the survey was administered, would withdraw from the Mentor- Prot�g�
Program if funding for reimbursable agreements were eliminated. LMI
recommended that DOD continue to fund reimbursable agreements. It also
recommended that DOD explore other incentives, such as assigning additional
points during the source selection evaluation process to those contractors
who choose to participate in the Mentor- Prot�g� Program, and through the
use of award fees and procurement set- asides to encourage continued mentor
assistance to prot�g�s. In exploring the need for continued funding, LMI
asked program beneficiaries whether additional incentives are needed.
Direct surveys of a program?s beneficiaries are not the best way to analyze
a program because beneficiaries would not be inclined to report that they
did not need government funds after receiving them. Accordingly, it is
unknown how the absence of funding would affect overall provisions of
developmental assistance by mentors or discourage them from establishing
business relationships with small disadvantaged businesses.
There are small disadvantaged businesses that have participated in DOD?s
Mentor- Prot�g� Program and are receiving contract and subcontract awards,
increasing their revenues by hundreds of millions of dollars. At Effect of
Program
Funding on Defense Contractor Participation Is Unknown
Conclusions
Page 7 GAO- 01- 767 Contract Management
the same time, DOD is meeting its 5- percent contracting goal with these
businesses. Nevertheless, DOD has not been able to demonstrate the success
of the program- first because it does not have complete data on program
results from mentors and second because it has not performed an impact study
to compare the success of a group of participants to the success of an
equivalent group of non- participants. Implicit in such a comparison is that
differences found would not have occurred without program participation.
Although impact studies are sometimes difficult and expensive, they can be a
reliable way to determine whether a program, such as the Mentor- Prot�g�
Program, is making a difference.
Before extending the Pilot Mentor- Prot�g� Program, the Congress may wish to
consider directing DOD to conduct a more conclusive assessment of the
Program?s impact and its contributions.
We recommend that DOD (1) gather more complete information by asking
prot�g�s how the Mentor- Prot�g� Program has resulted in additional prot�g�
contracting and subcontracting, and (2) determine if mentor- prot�g�
agreements have been met. We also recommend that DOD not accept incomplete
mentor- reported data.
In commenting on a draft of this report, DOD contends that the Mentor-
Prot�g� Program is successful. However, DOD recognizes the need to
strengthen its data collection and performance reviews to enable more
complete and consistent reporting. DOD partially concurred with our
recommendation that it (1) gather more complete information by asking
prot�g�s how the Mentor- Prot�g� Program has resulted in additional prot�g�
contracting and subcontracting, and (2) determine if mentor- prot�g�
agreements have been met. DOD agreed to establish a team to review the
current semi- annual reporting mechanism to determine and implement
improvements to the data collection instrument and the performance review
process. DOD believes that asking prot�g�s how the program has resulted in
additional contracting and subcontracting would be an imposition to
prot�g�s. We disagree and believe that DOD should ask prot�g�s if and how
the program has resulted in contracting and subcontracting opportunities
within the context of the existing statutory reporting requirement on
progress made in employment, revenues, and participation in DOD contracts.
Further, the gathering of such information is consistent with the
advancement of the statutory purpose of the program, namely, to increase the
participation of prot�g�s as Matter for
Congressional Consideration
Recommendations Agency Comments and Our Evaluation
Page 8 GAO- 01- 767 Contract Management
subcontractors and suppliers under DOD contracts and other contracts and
subcontracts.
DOD concurred with our recommendation that it not accept incomplete mentor-
reported data. DOD?s written comments are in appendix II.
Our congressional mandate required that we study the relationship between
the results of the Mentor- Prot�g� Program and the objectives established in
10 U. S. C. 2323, which established a goal of awarding 5 percent of the
total dollar amount contracted by DOD and subcontracted by DOD prime
contractors to small disadvantaged businesses. However, sufficient
information is not available to establish a relationship between the Mentor-
Prot�g� Program and its effect on the 5- percent goal.
The congressional mandate also required that we determine whether that, as a
result of the program, prot�g�s are more competitive, financially
independent, and what the correlation of program funding is to prot�g�
business development. In response to our last review, 7 DOD officials wrote
that they were developing a program database that would be reliable and
sufficient enough to evaluate program success. We attempted to answer our
mandate by using this database. We found, however, that the DOD database
does not have sufficient information to answer these mandated requirements.
To evaluate the effectiveness of the incentives provided to mentor firms to
participate in the program, we interviewed program officials and reviewed a
Mentor- Prot�g� Program contractor study, completed in December 2000, which
examined (1) whether it would be feasible to operate the MentorProt�g�
Program without reimbursement and (2) whether alternative incentives would
motivate mentors to participate without reimbursement.
We also were asked to evaluate how program funds had been used as incentives
for mentor firms to participate in the program. Because we could not
correlate funding and prot�g�s? business development, we did not further
report on the manner in which funds were obligated and on the average amount
spent on individual agreements.
7 Defense Contracting: Sufficient, Reliable Information on DOD?s Mentor-
Prot�g� Program Is Unavailable (GAO/ NSIAD- 98- 92, Mar. 30, 1998). Scope
and
Methodology
Page 9 GAO- 01- 767 Contract Management
During our review, we held discussions with (1) the President of the
National Association of Small Disadvantaged Businesses, which consists of
about 300 member firms; (2) current and former Mentor- Prot�g� Program
participants, consisting of four mentors and eight prot�g�s; (3) Mentor-
Prot�g� Program Office officials; and (4) the Mentor- Prot�g� Program Office
supporting contractor responsible for maintaining the Mentor- Prot�g�
Program database. In addition, we reviewed MentorProt�g� Program Office
documentation and minutes of a Mentor- Prot�g� Program Office sponsored
mentor meeting. We also held discussions with officials from the Small
Business Administration.
We performed this phase of work between December 2000 and April 2001 in
accordance with generally accepted government auditing standards.
We will send copies to the Chairmen and the Ranking Minority Members of
other appropriate congressional committees; the Secretary of Defense; and
the Director, Office of Management and Budget; and other interested parties.
We will also make copies available to others on request.
Please contact me at (202) 512- 4841 if you or your staff have any questions
concerning this report. Additional contact and key contributors to this
report are acknowledged in appendix III.
David E. Cooper Director Acquisition and Sourcing Management
Appendix I: Performance Measurements Reported in the September 1999 Semi-
Annual Reports
Page 10 GAO- 01- 767 Contract Management
The Department of Defense (DOD) requires mentor firms to report semiannually
on the progress made under active mentor- prot�g� agreements for the periods
ending March 31st and September 30th. The September semi- annual report is
to summarize progress made for the entire fiscal year from the start of each
mentor- prot�g� agreement. The metrics listed in the tables below were
developed from the data in the September 1999 semi- annual reports 1
because, as of January 2001, less than half of the September 2000 semi-
annual reports had been received and, therefore, were not available for our
review.
The semi- annual report is not designed to establish a cause- and- effect
relationship between mentoring provided and reported success. A prot�g� may
realize additional revenue because of mentor- related business
opportunities. However, the semi- annual report does not isolate
mentorrelated revenue growth from growth that is unrelated to the program.
The same holds true for the other metrics that the semi- annual report
captures and that are presented below, such as employment growth and
contracts awarded.
Table 1: Distribution of Prot�g�s and the Value of Contracts Received
Dollars in millions Distribution of prot�g�s Value of awarded
contracts a
3 $519.5 10 242.6 16 155.1 13 52.7 54 53.2 76 0 a Contracts cannot be
explicitly linked to the Mentor- Prot�g� Program. The amounts include all
prot�g� contracts from DOD, not just those that may have resulted from the
Mentor- Prot�g� Program.
1 We reviewed 186 semi- annual reports dated September 30, 1999. However, 14
of these reports were incomplete and were not used in this appendix.
Appendix I: Performance Measurements
Reported in the September 1999 Semi- Annual Reports
Appendix I: Performance Measurements Reported in the September 1999 Semi-
Annual Reports
Page 11 GAO- 01- 767 Contract Management
Table 2: Distribution of Prot�g�s and Change in Revenue Dollars in millions
Distribution of prot�g�s a Change in
revenue b
12 $240.2 17 113.4 25 71.1 19 24.4 54 25.1 6 0 22 (28.3) a Semi- annual
reports submitted on 17 prot�g�s either did not list the prot�g�s? revenue
at the start of the agreement or the prot�g�s? revenue at the time of the
semi- annual report. Therefore, these reports were not included in this
analysis. b These revenue changes cannot be explicitly linked to the Mentor-
Prot�g� Program. The revenue
increase includes all prot�g�s? revenue increases, not just those that may
have resulted from the Mentor- Prot�g� Program.
Table 3: Distribution of Prot�g�s and Change in the Number of Employees
Distribution of prot�g�s a Change in number of employees b
12 1,750 16 868 24 501 15 173 44 190 17 0 35 (573) a Semi- annual reports
submitted on nine prot�g�s either did not list the number of employees at
the start of the agreement or the number of employees at the time of the
semi- annual report. Therefore, these reports were not included in this
analysis. b These employee changes cannot be explicitly linked to the
Mentor- Prot�g� Program. The employee changes include all prot�g�s? employee
changes, not just those that may have resulted from the Mentor- Prot�g�
Program.
Appendix II: Comments From the Department of Defense Page 12 GAO- 01- 767
Contract Management
Appendix II: Comments From the Department of Defense
Appendix II: Comments From the Department of Defense Page 13 GAO- 01- 767
Contract Management
Appendix II: Comments From the Department of Defense Page 14 GAO- 01- 767
Contract Management
Appendix III: GAO Contact and Staff Acknowledgments
Page 15 GAO- 01- 767 Contract Management
Ralph Dawn (202) 512- 4544 In addition to the name above, Catherine
Baltzell, Daniel Hauser, Russ Reiter, and Adam Vodraska made key
contributions to this report. Appendix III: GAO Contact and Staff
Acknowledgments GAO Contact Acknowledgments
(120029)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***