Contract Management: Proteges Value DOD's Mentor-Protege Program, but Annual Reporting to Congress Needs Improvement

Contract Management: Proteges Value DOD's Mentor-Protege Program,
but Annual Reporting to Congress Needs Improvement (31-JAN-07,
GAO-07-151).

Congress authorized the Mentor-Protege Program to boost the
participation of small disadvantaged businesses as subcontractors
and suppliers under Department of Defense (DOD) and other
contracts. The program provides incentives to major defense
contractors (mentors) to help small disadvantaged businesses
(proteges) strengthen their ability to compete for contracts. GAO
administered a Web-based survey to determine whether former
proteges believe the program enhanced their business development;
examined the accuracy of the Mentor-Protege Program Office's
annual reporting to Congress; determined whether DOD reported on
the progress of former proteges and their contributions to small
business goals; and, identified how program funds have been
obligated and used.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-151
ACCNO: A65432
TITLE: Contract Management: Proteges Value DOD's Mentor-Protege
Program, but Annual Reporting to Congress Needs Improvement
DATE: 01/31/2007
SUBJECT: Accountability
Contract administration
Contracts
Defense procurement
Department of Defense contractors
Federal funds
Mentoring
Program evaluation
Small business
Small disadvantaged business contractors
Surveys
Program goals or objectives
DOD Pilot Mentor-Protege Program

******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-151

* [1]Results in Brief
* [2]Background
* [3]Most Former Proteges Value Mentor-Protege Program
* [4]Accuracy of Statutorily Required Annual Reporting Is Questio
* [5]Requirement to Report Trends in Progress Made by Former Prot
* [6]Mentor-Protege Program Funds Reimburse Mentors, Proteges, an
* [7]Conclusions
* [8]Recommendations for Executive Action
* [9]Agency Comments and Our Evaluation
* [10]Mentor's Support Provided to Proteges
* [11]Level of Satisfaction, Perceived Value, and Willingness to R
* [12]Developmental Assistance Providers
* [13]North American Industry Classification System Codes
* [14]GAO's Mission
* [15]Obtaining Copies of GAO Reports and Testimony

* [16]Order by Mail or Phone

* [17]To Report Fraud, Waste, and Abuse in Federal Programs
* [18]Congressional Relations
* [19]Public Affairs

Report to Congressional Committees

United States Government Accountability Office

GAO

January 2007

CONTRACT MANAGEMENT

Proteges Value DOD's Mentor-Protege Program, but Annual Reporting to
Congress Needs Improvement

GAO-07-151

Contents

Letter 1

Results in Brief 2
Background 4
Most Former Proteges Value Mentor-Protege Program 6
Accuracy of Statutorily Required Annual Reporting Is Questionable 8
Requirement to Report Trends in Progress Made by Former Proteges Has Not
Been Met 9
Mentor-Protege Program Funds Reimburse Mentors, Proteges, and Cover
Program Operational Costs 11
Conclusions 12
Recommendations for Executive Action 12
Agency Comments and Our Evaluation 12
Appendix I Scope and Methodology 14
Appendix II Results Attributable to Mentoring 16
Mentor's Support Provided to Proteges 22
Level of Satisfaction, Perceived Value, and Willingness to Recommend the
Program to Others 24
Appendix III Mentoring Completion, Adequacy, and Helpfulness 27
Appendix IV Composition of Responding Protege Firms 30
Developmental Assistance Providers 31
North American Industry Classification System Codes 31
Appendix V Comments from the Department of Defense 32

Tables

Table 1: GAO Analysis of DCMA Reviews of Fiscal Year 2003 and 2004
Semi-Annual Reports 9
Table 2: Mentor-Protege Program Fiscal Year 2004, 2005, and 2006
Appropriations and Obligations as of October 12, 2006 (Dollars in
Millions) 11
Table 3: Socioeconomic Categories Reported by Proteges 30
Table 4: Distribution of Institutions Providing Development Assistance to
Proteges 31

Figures

Figure 1: Protege Attribution of Outcomes to Program Participation 8
Figure 2: Extent of Increase in Contracts Directly Attributable to the
Mentor-Protege Program 16
Figure 3: Extent of Increase in Contract Dollar Value Directly
Attributable to the Mentor-Protege Program 17
Figure 4: Extent of Overall Increase in Revenue Directly Attributable to
the Mentor-Protege Program 18
Figure 5: Extent of Increase in Employees Directly Attributable to the
Mentor-Protege Program 19
Figure 6: Extent of Increase in Certification Directly Attributable to the
Mentor-Protege Program 20
Figure 7: Extent of Enhancement of Overall Capabilities Directly
Attributable to the Mentor-Protege Program 21
Figure 8: Business Management/Corporate Infrastructure Support 22
Figure 9: Engineering or Technical Support 23
Figure 10: Level of Satisfaction with Participation in the Mentor-Protege
Program 24
Figure 11: Response to Whether the Mentor-Protege Program Provided a
Valuable Experience 25
Figure 12: Response to Whether Proteges Would Recommend the Mentor-Protege
Program to Other Eligible Firms 26
Figure 13: Mentors' Ability to Provide Anticipated Mentoring 28
Figure 14: Mentor Helpfulness with Business Development and Engineering or
Technical Expertise 29

Abbreviations

CMMI Capability Maturity Model Integration
DCMA Defense Contract Management Agency
DOD Department of Defense
NAICS North American Industry Classification System
SDB Small Disadvantaged Business

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

January 31, 2007

The Honorable Carl Levin
Chairman
The Honorable John Warner
Ranking Minority Member
Committee on Armed Services
United States Senate

The Honorable Ike Skeleton
Chairman
The Honorable Duncan L. Hunter
Ranking Minority Member
Committee on Armed Services
House of Representatives

Each year, the Department of Defense (DOD) obligates hundreds of billions
of dollars in contracts for goods and services. To boost the participation
of small disadvantaged businesses as subcontractors and suppliers under
DOD and other federal and commercial contracts, Congress authorized a
pilot Mentor-Protege Program. The program, which was authorized in 1990^1
and has been continually renewed, provides incentives for major defense
contractors (mentors) to help small disadvantaged businesses (proteges)
strengthen their ability to compete for defense and other federal
contracts as well as commercial contracts.

To ensure that the program is focused on a results-oriented approach to
assessing program performance, Congress has required DOD to report
annually on trends in the progress made in employment, revenues, and
participation in DOD contracts of protege firms and former protege
firms.^2 Congress also required GAO to study the program's
effectiveness.^3 In responding to this mandate, we

^1National Defense Authorization Act for Fiscal Year 1991, Pub. L. No.
101-510, S 831 (1990).

^2National Defense Authorization Act for Fiscal Year 2000, Pub. L. No.
106-65, S 811(d) (1999).

^3Ronald W. Reagan National Defense Authorization Act for Fiscal Year
2005, Pub. L. No. 108-375, S 841 (2004).

o determined whether former proteges believe the Mentor-Protege
Program enhanced their business development,

o assessed the accuracy of the Mentor-Protege Program Office's
annual reporting to Congress,

o determined if the Mentor-Protege Program Office reports on the
progress of former proteges and how their progress helped DOD
contribute to the statutory objective^4 of awarding 5 percent of
its contracting dollars to small disadvantaged businesses, and

o identified how Mentor-Protege Program funds have been obligated
and used.

In performing our work, we administered a Web-based survey to proteges
that completed or left the program during fiscal years 2004 and 2005,
obtaining a 63 percent response rate. We reviewed mentor-prepared
semi-annual reports, designed to quantify the progress made under active
mentor-protege agreements, and we examined the Defense Contract Management
Agency's (DCMA) audit of those reports. In addition, we interviewed
proteges, mentors, and Mentor-Protege Program Office officials. We also
obtained funding data from the Mentor-Protege Program Office and reviewed
the criteria established to ensure that costs incurred were reasonable. A
complete statement of our methodology is in appendix I. We conducted our
review from March 2006 to October 2006, in accordance with generally
accepted government auditing standards.

Results in Brief

Most of the 48 former proteges that responded to our survey reported that
the Mentor-Protege Program was a valuable experience and helped enhance
their business development. About 87 percent of responding proteges
reported that support from their mentors helped their business development
and corporate infrastructure. About 84 percent stated that mentor support
enhanced their engineering or technical expertise; one-fourth directly
linked their participation in the program to their ability to meet
International Organization for Standardization^5 quality standards; and
over one-fifth responded they had received additional training and
certifications, such as manufacturing process and control training or
Capability Maturity Model Integration^6 certifications. Proteges also
reported quantifiable business growth. For example, proteges collectively
attributed 95 new contracts and about $131 million in increased revenue to
their participation in the program. Further, about 63 percent reported an
increase in employees. Despite the overall value proteges attributed to
the program, about one-quarter reported that the program had no impact on
gaining new contracts or on increasing revenues.

^4National Defense Authorization Act for Fiscal Year 1987, Pub. L. No.
99-661, S 1207 (1986), codified at 10 U.S.C. S 2323.

^5The International Organization for Standardization is the world's
largest developer of standards. Suppliers meeting these standards have
wide acceptance within their vendor industry.

The accuracy of DOD's annual reports to the Congress is questionable,
primarily because the data are not validated before the reports are
submitted to Congress. Specifically, each March DOD submits its annual
report to Congress, which is based on mentor-prepared reports on protege
progress, but DCMA does not complete its validation of the mentor-prepared
reports until the following September. In addition, many of the
validations are incomplete, and some mentor-prepared reports are not
validated at all. In fiscal year 2004, for example, 40 of the 183 active
agreements were not reviewed at all, and 30 percent of the mentor-prepared
reports did not have proteges' signatures and/or concurrence, as required
to help corroborate data accuracy. If the data are flawed, the report
could overstate or understate the success of the program. Yet, DOD does
not know the extent of the validation problem or the overall effectiveness
of the program because it does not summarize the audits in a meaningful
comprehensive report to assess what the findings of the DOD audits mean,
how the DOD findings affect the program's operation, or identify any
corrective actions that need to be taken to make the program more
effective.

Until April 2006, DOD did not attempt to report on the trends of the
progress made by proteges after they completed the program, as required by
the Fiscal Year 2000 National Defense Authorization Act. Although
Mentor-Protege Program agreements signed since February 2000 required
proteges to report their progress for 2 years after they complete the
program, they were not doing so. According to proteges we interviewed,
they either were unaware or had forgotten that they had agreed to this
reporting requirement. Program office officials, when asked if they sent
letters to former proteges as a reminder to comply with the 2-year
reporting requirement, said the first such letter was sent in April 2006,
after GAO asked to see the progress reports. Even if the 2-year
post-agreement reporting requirement had been met, we could not determine
the contribution of the program to the statutory objective of awarding 5
percent of DOD's contracting dollars to small disadvantaged businesses. To
make this determination would require knowing the progress of those
proteges completing the program since the program's inception; this
information is unavailable.

^6A Capability Maturity Model Integration (CMMI)^(R) certification
confirms that a firm knows the value of establishing effective processes.
It is based on the premise that even a quality workforce cannot perform
its best when a firm's processes are not understood or not operating
effectively. CMMI can be used to guide process improvement across a
project, a division, or an entire organization. CMMI helps integrate
traditionally separate organizational functions, set process improvement
goals and priorities, provide guidance for quality processes, and provide
a point of reference for appraising current processes.

Over the past 3 years, the majority of the Mentor-Protege Program's funds,
which totaled about $70 million, were obligated to mentors. The mentors
were reimbursed for the developmental assistance they provided to
proteges, either directly or indirectly, by subcontracting to Historically
Black Colleges and Universities, minority institutions, and certain
government-sponsored groups--procurement technical assistance centers and
small business development centers. Funds were also obligated to reimburse
proteges for their incidental costs, such as travel, and to cover the
Mentor-Protege Program Office's operational costs. DCMA reviews determined
that the reimbursements received by mentors during fiscal years 2003 and
2004 were reasonable.

We are making recommendations to improve the accuracy and completeness of
the Mentor-Protege annual reports to Congress, which should help to better
assist in evaluating the program. In written comments on a draft of this
report, DOD concurred with our findings and recommendations, indicating
that implementation of our recommendations is already underway. DOD stated
that these actions are expected to fully address GAO's recommendations and
are scheduled to be completed by the end of fiscal year 2007.

Background

Congress established, as an objective for DOD, the goal of awarding 5
percent of the amount contracted by DOD and subcontracted by DOD's prime
contractors to small disadvantaged businesses. The 5 percent goal was not
met in the years immediately following its establishment because,
according to large DOD prime contractors, there were not enough qualified
small disadvantaged businesses available as DOD subcontractors.

Subsequently, the National Defense Authorization Act for Fiscal Year 1991
required the Secretary of Defense to establish a pilot Mentor-Protege
Program. The purpose of the program is to provide incentives for major DOD
contractors (mentors) to furnish small disadvantaged businesses
(proteges^7) with assistance designed to enhance their capabilities and
increase their participation as subcontractors and suppliers under DOD
contracts, other federal government contracts, and commercial contracts.
Since the Pilot Mentor-Protege Program was authorized, it has been
continuously extended. With the passage of the Fiscal Year 2005 National
Defense Authorization Act, DOD is now authorized to approve new agreements
through September 30, 2010, and program performance is authorized through
September 30, 2013.

DOD's Office of Small Business Programs manages the Mentor-Protege
Program. Each year, as required by the fiscal year 2000 authorizing
statute,^8 DOD submits to Congress an annual report on the trends in the
progress made in employment, revenues, and participation in DOD contracts
by protege firms. To satisfy the requirements of the fiscal year 2000
authorizing statute, DOD's implementing regulations require (1) mentors to
report on the progress made under active mentor-protege agreements,
semi-annually, throughout the term of an agreement, and (2) proteges to
report progress in each of the 2 years following the completion of their
agreements. In addition, DCMA is to conduct annual performance reviews to
verify whether the mentors and proteges accurately reported progress and
to determine that all costs reimbursed to mentors were reasonably incurred
in accordance with the statute and applicable regulations.

From the program's inception through fiscal year 2006, Congress has
appropriated $437.7 million to the program, and DOD has approved 910
mentor-protege agreements. Funding covers program operational costs and
reimburses mentors for providing developmental assistance. Mentor-protege
agreements are usually limited to a 3-year period. At the end of July
2006, there were 149 active mentor-protege agreements.

^7Eligible proteges now consist of small disadvantage businesses, 8(a)
certified firms, Women-Owned Small Businesses, firms that employ severely
disabled persons, service-disabled veteran-owned businesses, historically
underutilized business zone small business concerns, Indian tribe-owned,
and Native Hawaiian Organization-owned.

^8National Defense Authorization Act for Fiscal Year 2000, Pub. L. No.
106-65, S 811 (1999).

This report is one of a series we have issued on the Mentor-Protege
Program. We reported on the Mentor-Protege Program in 1992,^9 1994,^10
1998,^11 and 2001.^12 In our first report, one of our recommendations was
that DOD develop and implement adequate internal controls in the
application and approval process and in the oversight of protege
development. In our subsequent reports, we found DOD did not have
sufficient data to adequately assess the program's performance in meeting
its objectives. Among other things, we recommended that DOD gather more
complete information on program performance by strengthening performance
reviews and seeking protege feedback on the extent to which the program
had resulted in additional contracting and subcontracting opportunities.
DOD concurred with our recommendations to strengthen its data collection
and performance reviews, but stated that asking proteges how the program
has resulted in additional contracting and subcontracting would be an
imposition to proteges. We disagreed because, within the context of
statutory reporting requirements, DOD should remind proteges to report on
how the program had resulted in contracting and subcontracting
opportunities.

Most Former Proteges Value Mentor-Protege Program

To determine whether former proteges believe the Mentor-Protege Program
enhanced their business development, we administered a Web-based survey
and received responses from 48 of the 76 proteges that completed or left
the program during fiscal years 2004 and 2005. Ninety-three percent of
proteges reported the Mentor-Protege Program enhanced, at least to some
degree, their firms' overall capabilities. About 87 percent reported that
support from their mentors helped their businesses' development, and about
84 percent stated that mentor support helped their engineering or
technical expertise. As an example of specific development support, about
71 percent of proteges reported receiving help with their quality
management programs, with one-fourth directly linking the assistance they
received to their ability to meet International Organization for
Standardization quality standards, which increased their business
competitiveness. Over one-fifth responded that they had received
additional training and certifications, such as manufacturing process and
control training or Capability Maturity Model Integration certifications.
Finally, nearly 98 percent of the proteges reported that they would
recommend the program to other small businesses eligible to participate in
the program.

^9GAO, Defense Contracting: Interim Report on Mentor-Protege Program for
Small Disadvantaged Firms, [20]GAO/NSIAD-92-135 (Washington, D.C.: Mar.
30, 1992).

^10GAO, Defense Contracting: Implementation of the Pilot Mentor-Protege
Program, [21]GAO/NSIAD-94-101 (Washington, D.C.: Feb. 1, 1994).

^11GAO, Defense Contracting: Sufficient, Reliable Information on DOD's
Mentor-Protege Program is Unavailable, [22]GAO/NSIAD-98-92 (Washington,
D.C.: Mar. 30, 1998).

^12GAO, Contract Management: Benefits of the DOD Mentor-Protege Program
are Not Conclusive, [23]GAO-01-767 (Washington, D.C.: July 19, 2001).

In terms of the level of satisfaction, our survey found about 71 percent
of proteges were at least generally satisfied with their experience, with
reasons ranging from enhanced capabilities and heightened exposure in the
marketplace, to quantifiable business growth. Nearly 80 percent of
proteges reported that their mentors provided the anticipated type and
level of mentoring. The overall results also indicate that the majority of
responding proteges directly attributed some degree of business growth to
their participation in the program. Measures of this growth include the
number and dollar value of contracts received, overall revenues, new
employee hiring, and new certifications. For example, proteges
collectively attributed 95 new contracts and about $131 million in
increased revenues, to their participation in the program.

About 15 percent of the proteges reported dissatisfaction with their
participation and about 21 percent responded that they did not receive the
level of mentoring that they had anticipated. Reasons for dissatisfaction
included a lack of mentor commitment to the program, mentor failure to
meet the objectives of mentor-protege agreements, and costs to the
proteges that exceeded the return from participation. Furthermore, about
28 percent of proteges reported that their program participation did not
produce increases in their contracts and nearly one-quarter reported no
impact on revenues.

Figure 1 illustrates the extent to which proteges attributed the
enhancement of their overall capabilities as well as increases in
contracts, dollar value of contracts, certifications, revenues, and
additional employees to the Mentor-Protege Program.

Figure 1: Protege Attribution of Outcomes to Program Participation

Note: The percents are rounded and do not include responses of no basis to
judge and not applicable.

Appendixes II through IV provide detailed information on the results of
our survey.

Accuracy of Statutorily Required Annual Reporting Is Questionable

The accuracy of statutorily required annual reporting is questionable
because DOD's annual performance reviews of mentor-protege agreements,
designed to verify the accuracy of progress made by proteges, are not
completed until after DOD sends its annual report to Congress.
Specifically, each March DOD submits its annual report to Congress. This
report is based on mentor-prepared reports on protege progress, but DOD
does not complete its validation of the mentor-prepared reports until the
following September. In addition, not all of the mentor-protege agreements
are reviewed; many of the reviews are incomplete; and many reports were
not signed and/or concurred to by proteges. The protege's signature and
concurrence are required to provide corroborating support of the data
submitted by mentors. Further, the overall message of the performance
reviews is unknown because the reviews, which consist of over a hundred
individual audits each year, are not summarized into a single report to
assess what the findings of its audits mean, how these findings affect the
program's operation, and to identify any corrective actions that need to
be taken to make the program more effective.

The Mentor-Protege Program Fiscal Year 2005 Annual Report to Congress,
dated March 31, 2006, shows the trends in the progress of protege
employment, revenues, and participation in DOD contracts during fiscal
year 2005. The information in this report and previous annual reports sent
to Congress was derived from mentor-prepared semi-annual reports, which
had not been reviewed by DCMA. For example, the DCMA annual performance
reviews of the fiscal year 2005 mentor-prepared semi-annual reports were
not completed until September 2006, 6 months after the Mentor-Protege
Program Office reported to Congress.

We analyzed the results of DCMA's review of the fiscal years 2003 and 2004
reports, which are shown in table 1. In sum, not all of the mentor-protege
agreements were reviewed, many reports were not signed and/or concurred to
by proteges, and many of the reviews were incomplete.

Table 1: GAO Analysis of DCMA Reviews of Fiscal Year 2003 and 2004
Semi-Annual Reports

Areas of Analysis FY 2003 FY 2004
Active agreements 189 183
Active agreements with reports reviewed 125 143
Active agreements not reviewed 64 (34%) 40 (22%)
Reports without protege signature and/or indication of 34 (27%) 43 (30%)
concurrence or non-concurrence
Reports reviewed without verification of all the metrics 28 (22%) 21 (15%)

Source: GAO analysis of DCMA reviews of fiscal year 2003 and 2004
semi-annual reports

Requirement to Report Trends in Progress Made by Former Proteges Has Not Been
Met

Although the Fiscal Year 2000 National Defense Authorization Act
established the requirement for DOD to report on the trends in the
progress of former proteges for agreements entered into on or after
October 1, 1999, the requirement has not been met. Pursuant to the Act,
proteges are to report their progress in employment, revenues, and
participation in DOD contracts in each of the 2 years following the
completion of their agreements. The Mentor-Protege Program agreement
application template was revised in February 2000 to require proteges to
sign a statement indicating their willingness to comply with the program's
reporting requirements, including the 2-year post-agreement requirement.
The Act also required DCMA to determine whether proteges are accurately
reporting data during the 2-year post-agreement period. Accordingly,
DCMA's implementing guidance states that if the data are either not
provided or are not verifiable, the DCMA report should provide the reason
why.

Despite the Act and DCMA's guidance, the 2-year post-agreement-reporting
requirement has not been met, and DCMA did not explain in its annual
performance reviews that protege annual reporting data had not been
provided.

The proteges we interviewed told us that after their agreement period,
they either forgot that they had agreed to the reporting requirement or
were not aware of the requirement from the beginning. DCMA officials
acknowledged that the reporting shortfall was the result of DCMA not
following its own policy. Finally, when we asked Mentor-Protege Program
office officials whether they sent letters to former proteges as a
reminder to comply with the 2-year reporting requirement, they said that
the first such letter was sent on April 17, 2006, after we first asked
about compliance with the 2-year reporting requirement. Program Office
officials advised us, in October 2006, that 48 percent of the proteges
they attempted to contact^13 responded to the reminder letter.

Part of our requirement to review the Mentor-Protege Program was to
determine if the Mentor-Protege Program office reports on the progress of
former proteges and how their progress helps DOD contribute to the
statutory objective of awarding 5 percent of its contracting dollars to
small disadvantaged businesses. Even if the 2-year post-agreement
reporting requirement had been met, we could not determine the
contribution of the program to the statutory objective of awarding 5
percent of DOD's contracting dollars to small disadvantaged businesses. To
make this determination would require knowing the progress of those
proteges completing the program since the program's inception; this
information is not available.

^13Eight protege companies had either gone out of business or had been
purchased by a larger company, precluding the program office from
contacting them.

Mentor-Protege Program Funds Reimburse Mentors, Proteges, and Cover Program
Operational Costs

Over the past 3 years, the majority of the Mentor-Protege Program's funds
were obligated to mentors.^14 The mentors were reimbursed for the
developmental assistance they provided to proteges, either directly or
indirectly, by subcontracting to Historically Black Colleges and
Universities, minority institutions, and certain government-sponsored
groups--procurement technical assistance centers and small business
development centers. Funds were also obligated to reimburse proteges for
their incidental costs, such as travel, and to cover the Mentor-Protege
Program Office's operational costs. The services are responsible for
managing reimbursable mentor-protege agreements and do so by either
modifying existing contracts or awarding new ones.

Table 2: Mentor-Protege Program Fiscal Year 2004, 2005, and 2006
Appropriations and Obligations as of October 12, 2006 (Dollars in
Millions)

Obligated
funds
Appropriation Mentors
Received by subcontracted Incidental MPP Office
Fiscal Program Un-Obligated to Protege Operational
year Office^a Mentors Funds HBCU^b Fees Costs Totals MI^c PTAC^d SBDC^e
2004 $25.006 $18.155 $1.357 $1.206 $.190 0 $.730 $3.362 $25.000 $.006
2005 $19.325 $10.837 $1.859 $.934 $.169 0 $.481 $3.787 $18.068 $1.257
2006 $25.543 $15.250 $.251 $.479 $.042 $.078 $.149 $1.936 $18.185 $7.358

Source: Mentor-Protege Program Office

Note: Totals may not add due to rounding.

aFunds reflect the effect of DOD reprogramming actions.

bHBCU--Historically Black Colleges and Universities are accredited
institutions of higher education with the principal mission of educating
African Americans.

cMI--Minority institutions are accredited institutions of higher education
whose enrollment of a single minority or a combination of minorities
(American Indian; Alaskan Native; African American; not of Hispanic
origin; Hispanic, including persons of Mexican, Puerto Rican, Cuban, and
Central or South American origin; Pacific Islander, and/or other ethnic
group underrepresented in science and engineering), exceeds 50 percent of
the enrollment or 25 percent of the enrollment if of Hispanic origin.

dPTAC--Procurement technical assistance centers provide assistance in
marketing products and services to the federal, state, and local
governments. The Defense Logistics Agency administers the DOD procurement
technical assistance program.

eSBDC--Small business development centers are administered by the SBA to
assist small business owners by providing a wide variety of information
and guidance in central and easily accessible branch locations.

^14Mentor-Protege appropriations are procurement funds, which can be
obligated over a 3 year period.

As part of its performance reviews of Mentor-Protege agreements, DCMA is
responsible for determining whether the costs reimbursed to mentor firms
were reasonable. For the agreements reviewed by DCMA for fiscal years 2003
and 2004, the costs claimed by mentors were determined to be reasonable.
DCMA had not completed its review of the fiscal years 2005 and 2006
agreements at the time of our review.

Conclusions

The Mentor-Protege Program has clearly been of value to the majority of
proteges that responded to our survey. However, DOD is not measuring the
impact of the program on participating and former protege firms, as
Congress intended. Until DOD does so, by obtaining and validating the data
Congress requested to measure success, neither DOD nor Congress will have
the information it needs to oversee the program.

Recommendations for Executive Action

To better evaluate the success of the DOD Mentor-Protege Program and to
improve annual reporting to Congress, we recommend that the Secretary of
Defense

o direct DCMA to ensure that all Mentor-Protege agreements are
audited in accordance with DOD regulations and DCMA guidance and
that these audits are summarized into a single report so that the
Mentor-Protege Program Office and Congress have reliable data on
whether the program is accomplishing its goals and to determine
whether any corrective action is necessary, and

o direct DOD's Office of Small Business to submit its annual
report to Congress only after the data in the report have been
validated by DCMA and require that the annual report contain
information on the progress of proteges for 2 years following
their completion of the Mentor-Protege Program.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD concurred with our
findings and recommendations, stating that the implementation of a new
DCMA guidebook and revised reporting processes will address the
recommendations. DOD expects to complete the revised reporting processes
by the end of fiscal year 2007. DOD also noted in its response that many
of the recommendations regarding reporting requirements were already being
addressed through restructuring and procedural changes that have been
underway within DCMA. This is not an accurate depiction of DOD's actions.
It was as a result of our work that the Small Business Office became aware
of the reporting deficiencies. Only then did the Small Business Office
identify actions to address these deficiencies.

We are sending copies of this report to the Secretary of Defense, the
Director of the Office of Management and Budget, and interested
congressional committees. We will also make copies available at no charge
on the GAO Web site at http://www.gao.gov .

If you have questions about this report or need additional information,
please contact me at (202) 512-6986 or [email protected] .
Contact points for our Office of Congressional Relations and Public
Affairs may be found on the last page of this report. Key contributors to
this report were James Fuquay, Assistant Director; Greg Campbell, Daniel
Hauser, Stewart Kaufman, Sean Merrill, Charlie Shivers, and Karen
Thornton.

Ann Calvaresi-Barr
Director
Acquisition and Sourcing Management

Appendix I: Scope and Methodology

In the absence of data to determine whether former proteges believe the
Mentor-Protege Program enhanced their business development and translated
into growing their businesses, we designed and administered a Web-based
survey. We pretested the survey with two protege firms to determine
whether (1) the questions were clear, (2) the terms used were precise, (3)
the questions were unbiased, and (4) the survey did not place an undue
burden on the respondents. Based on the pretests, we made changes to our
final survey. In coordination with the Mentor-Protege Program Office, we
identified a universe of 79 proteges that had either completed or
terminated Mentor-Protege agreements during fiscal years 2004 and 2005. We
subsequently determined that three proteges would not be included in our
survey because two had active agreements and one had no contact
information. We selected former protege firms to reduce the potential for
bias responses that could result from proteges with active agreements. Of
the 76 eligible proteges that were sent the survey, 48 responded,
representing a response rate of 63 percent. The survey consisted of
open-ended and close-ended questions to characterize protege firms'
demographics as well as obtain information on their program participation
experiences, the adequacy of mentoring, the results attributable to
mentoring, and proteges' overall assessment of the program. We did not
independently verify the data provided by the proteges. We supplemented
the survey by interviewing officials from five protege firms with active
or recently completed agreements. These interviews were conducted to
determine if current proteges had opinions of the program that were
inconsistent with our survey findings. We also interviewed officials from
four mentor firms to obtain their views on the value of their agreements
with proteges.

To assess the accuracy of the Mentor-Protege Program Office's annual
reporting to Congress, we reviewed (1) the program's statutory reporting
requirements, (2) DCMA mentor-protege reporting regulations, (3) DCMA
reviews of the fiscal year 2003 and 2004 mentor-prepared semi-annual
reports, and (4) the fiscal years 2003, 2004, 2005, and 2006
Mentor-Protege Program Office's annual reports. We also held discussions
with officials from the program office, DCMA, and the Army, Navy, and Air
Force.

To determine whether the Mentor-Protege Program Office reports on the
progress of former proteges, we reviewed its reports to Congress and held
discussions with program office officials. We also interviewed proteges
and mentors, asking them if they were aware of the proteges'
post-agreement reporting requirement.

Finally, we obtained DOD funding data for the Mentor-Protege Program and
reviewed the criteria established to ensure that program costs incurred
were reasonable. We also reviewed the DCMA audits of those agreements that
were reviewed by DCMA for fiscal year 2003 and fiscal year 2004.

Appendix II: Results Attributable to Mentoring

We asked protege firms to respond to questions related to the business
growth and whether that growth was attributable to their participation in
the Mentor-Protege Program. The questions addressed increases in the
number of contracts, revenue increases, expanding numbers of employees,
and additional business certifications. The overall results indicate that
the majority of responding proteges directly attributed some degree of
business growth to their participation in the program for all of the
topics discussed above.

The following bar graphs illustrate how proteges attributed increases in
contracts, dollar value of contracts, revenues, employees, certifications,
and enhancement of overall capabilities to their participation in the
Mentor-Protege Program. The graphs represent the responses of all 48
proteges responding to our survey. The narrative accompanying each graph
represents the responses of the subset of the proteges that elected to
attribute specific values to each metric.

Figure 2: Extent of Increase in Contracts Directly Attributable to the
Mentor-Protege Program

The actual number of increased contracts reported ranged from 1 to 20
contracts, with an average number of contracts of nearly 2, a median of 2,
and a mode of 1 contract, among 26 proteges that reported specific
contract number increases.

Figure 3: Extent of Increase in Contract Dollar Value Directly
Attributable to the Mentor-Protege Program

The actual increase in contract dollar value reported ranged from $100,000
to $20 million, with an average dollar value over $750,000, a median
increase of $530,000, and a mode increase of $500,000, among 18 proteges
that reported specific increases in contract dollar value.

Figure 4: Extent of Overall Increase in Revenue Directly Attributable to
the Mentor-Protege Program

The overall increase in revenue reported ranged from $25,000 to $75
million, with an average revenue increase of over $5.2 million, a median
increase of $1 million, and a mode increase of $1.5 million, among 25
proteges that reported specific increases in overall revenue.

Figure 5: Extent of Increase in Employees Directly Attributable to the
Mentor-Protege Program

The extent of increase in employees reported ranged from 1 to 500
employees, with an average increase of 44 employees, a median increase of
17 employees, and a mode increase of 1 employee, among 23 proteges that
reported specific increases in employees.

Figure 6: Extent of Increase in Certifications Directly Attributable to
the Mentor-Protege Program

The extent of increase in the number of certifications reported ranged
from 1 to 60 certifications, with an average increase of 5 certifications,
a median increase of 1 certification, and a mode increase of 1
certification, among 27 proteges that reported specific increases in
certifications.

Figure 7: Extent of Enhancement of Overall Capabilities Directly
Attributable to the Mentor-Protege Program

The vast majority of proteges reported that some degree of enhancement of
overall capabilities was attributable to their participation in the
program, with 15 reporting the program provided enhancements to a great or
very great extent.

Mentor's Support Provided to Proteges

The figures below portray the types of support that proteges received
through participation in the Mentor-Protege Program. Figure 8 shows the
different types of business management and corporate infrastructure
support and the number of proteges reporting support for each type. Figure
9 shows the different types of engineering or technical support and the
number of proteges receiving support for each type.

Figure 8: Business Management/Corporate Infrastructure Support

Figure 9: Engineering or Technical Support

Level of Satisfaction, Perceived Value, and Willingness to Recommend the Program
to Others

To determine former proteges' overall assessment of the Mentor-Protege
Program, we asked (1) how satisfied or dissatisfied participants were with
the program; (2) whether the Mentor-Protege experience was valuable
(regardless of whether participants realized an increase in contracts
and/or revenues); and (3) if they would recommend participation in the DOD
Mentor-Protege Program to other eligible small firms. The following bar
graphs present how the proteges responded to these questions.

Figure 10: Level of Satisfaction with Participation in the Mentor-Protege
Program

Figure 11: Response to Whether the Mentor-Protege Program Provided a
Valuable Experience

Figure 12: Response to Whether Proteges Would Recommend the Mentor-Protege
Program to Other Eligible Firms

Appendix III: Mentoring Completion, Adequacy, and Helpfulness

Of the 48 proteges that responded to the survey, 42, or about 88 percent,
completed the agreement they signed with their mentors. For the six
proteges that terminated their agreement before it was scheduled to end,
two reported their agreements were terminated early because their
companies no longer qualified for the program and four reported that they
chose to end their participation in the program. For the two agreements
that ended because of a change in their eligibility, one reported that it
grew out of its small business status, and the other reported that it was
acquired by a publicly traded large business. For the four proteges that
chose to terminate their agreements, the reported reasons for their
decision included high cost both in resources and money, the program's
paperwork burden, and a lack of commitment from their mentor.

Of the proteges that responded to questions regarding their mentors'
ability to provide the anticipated type and level of mentoring, about 80
percent reported that their mentors provided the anticipated type and
level or mentoring. Figure 13 shows the specific reporting results.

Figure 13: Mentors' Ability to Provide Anticipated Mentoring

Note: "Not Sure" responses are not included.

To address the adequacy of mentoring specifically related to business
development and engineering or technical expertise, we asked proteges to
provide responses on their program experiences with both. Their responses
are depicted in Figure 14. Most reported the mentoring they received in
both areas was helpful to some degree. However, 6 proteges, or nearly 13
percent, reported that the business development mentoring they received
was not helpful, and 7 proteges, almost 16 percent, reported that
mentoring related to engineering or technical expertise was not helpful.

Figure 14: Mentor Helpfulness with Business Development and Engineering or
Technical Expertise

Note: Numbers include respondents providing actual responses, but do not
include "Not Applicable" or "Not Sure" responses.

Appendix IV: Composition of Responding Protege Firms

Through analysis of the responses from the 48 protege firms that completed
our survey, we were able to identify basic demographic information for our
respondents. For example, over 70 percent of proteges reported having
small and disadvantaged business status, and just over 35 percent reported
being women-owned small businesses. Table 3 depicts the socioeconomic
categories reported by proteges.

Table 3: Socioeconomic Categories Reported by Proteges

Percentage Reporting Socioeconomic
Reported Socioeconomic Category Category
Small disadvantaged business (SDB) 70.8%
SDB owned and controlled by an Indian 2.1%
Tribe
Woman-owned small business 35.4%
HUBZone small business 10.4%
Service-disabled veteran-owned small 4.2%
business
Other^a 10.4%

Source: GAO analysis of survey responses.

^aOther responses were specified by respondents and included
minority-owned business, veteran-owned business, former 8(a) business,
small business, and recently acquired by large business.

Notes: No Proteges reported being SDB owned and controlled by a Native
Hawaiian Organization or an Entity Employing the Severely Disabled.

Percents shown add to more than 100 percent due to rounding and because
respondents could answer with more than one response.

The year that each responding protege firm was established ranged from
1957 to 2001, with about 85 percent having been established within the
last 25 years and 60 percent opening for business between 1990 and 2001.
In terms of business size, as many as 900 employees were reported by
proteges, with an average of 166 employees and a median of 76 employees.
For gross revenue, nearly 45 percent reported receiving more than $10
million in 2005 and about 85 percent earned at least $1 million. Proteges
also reported that their percentage of current business with DOD averaged
almost 54 percent. The median was even higher, at 65 percent of their
total. Additionally, about 48 percent of the protege firms reported that
they are currently working as a subcontractor for their former mentor
firm.

Developmental Assistance Providers

We also asked proteges whether they received any of their developmental
assistance from Historically Black Colleges and Universities, minority
institutions, procurement technical assistance centers, and small business
development centers. The results are shown in table 4.

Table 4: Distribution of Institutions Providing Development Assistance to
Proteges

Percentage of respondents receiving
Development assistance institution assistance from provider^a
Historically Black Colleges or 47.8%
Universities
Minority institutions 10.0%
Procurement technical assistance 19.5%
centers
Small business development centers 23.8%

Source: GAO analysis of survey responses.

^aPercentages adjusted to only reflect respondents that were sure of their
assistance providers.

Note: Percents shown add to more than 100 percent due to rounding and
because respondents could answer with more than one response.

North American Industry Classification System Codes

We asked proteges for their North American Industry Classification System
(NAICS) codes. In total, 39 of 48 survey respondents provided 47 different
NAICS codes that represent their businesses. It is not unusual for a firm
to have more than one NAICS code. Below are the code descriptions reported
that had at least four responding proteges. The number beside each code
description is the number of proteges reporting the code description.

o Engineering services--14
o Computer systems design services--11
o Custom computer programming services--10
o Other computer related services--10
o Facilities support services--9
o Research and development in the physical, engineering, and life
sciences--8
o Computer facilities management services--7
o Other management consulting services--5
o Administrative management and general management consulting
services--4
o Remediation services--4
o Environmental consulting services--4

Appendix V: Comments from the Department of Defense

(120530)

GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or (202)
512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

www.gao.gov/cgi-bin/getrpt?GAO-07-151 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Anne Calvaresi-Barr at (202) 512-4841
[email protected].

Highlights of [33]GAO-07-151 , a report to congressional committees

January 2007

CONTRACT MANAGEMENT

Proteges Value DOD's Mentor-Protege Program, but Annual Reporting to
Congress Needs Improvement

[34]What GAO Recommends

GAO recommends that the Secretary of Defense take a number of actions to
better evaluate the performance of DOD's Mentor-Protege Program and to
improve annual reporting on the program to the Congress.

DOD concurred with GAO's findings and recommendations.

Most of the 48 former proteges that responded to GAO's questionnaire
reported that the Mentor-Protege Program was a valuable experience and
enhanced business development. Ninety-three percent of responding proteges
reported the Mentor-Protege Program enhanced, at least to some degree,
their firms' overall capabilities. While proteges also attributed
increases in contracts and revenues as a result of their participation in
the program, about one-quarter reported that the program had no impact on
gaining new contracts or on increasing revenues.

Protege Attribution of Outcomes to Program Participation

Although DOD's annual reports to the Congress indicate the program has
increased business opportunities, the accuracy of these reports is
questionable, primarily because the data are not validated before the
reports are submitted to Congress. Specifically, each March DOD submits
its annual report to Congress, based on mentor-prepared reports on protege
progress, but the Defense Contract Management Agency does not complete its
validation of the mentor-prepared reports until the following September.
In addition, many of the validations are incomplete or not done at all.

GAO could not measure the contribution of the Mentor-Protege Program to
the statutory objective of awarding 5 percent of DOD's contracting dollars
to small disadvantaged businesses. To do this, the progress of those
proteges completing the program since the program's inception would need
to be identified. This data is not available.

References

Visible links
20. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-92-135
21. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-94-101
22. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-98-92
23. http://www.gao.gov/cgi-bin/getrpt?GAO-01-767
33. http://www.gao.gov/cgi-bin/getrpt?GAO-07-151
*** End of document. ***