Manufacturing Extension Programs: Manufacturers' Views About Delivery and
Impact of Services (Letter Report, 03/14/96, GAO/GGD-96-75).
Pursuant to a congressional request, GAO surveyed manufacturers' views
regarding the impact of manufacturing extension programs (MEP), focusing
on: (1) factors contributing to the positive impact of overall business
performance reported by the majority of survey respondents; (2) whether
companies' expectations were met regarding MEP impact on specific
business performance indicators; and (3) whether companies thought that
MEP actually demonstrated attributes they valued most.
GAO found that: (1) the results of its survey could not be applied to
all MEP participants or all MEP service categories; (2) companies that
supplemented MEP assistance with their own resources, implemented more
MEP recommendations, were small and relatively new, and did not pay fees
for MEP assistance were more likely to view the MEP program positively;
(3) the source of MEP funding did not influence companies' views of the
assistance's impact; (4) National Institute of Standards and Technology
(NIST) officials believed that NIST support improved MEP programs'
efficiency and effectiveness and made MEP services more widely
available; (5) about two-thirds to three-quarters of the companies that
expected MEP assistance to enhance specific business indicators believed
that the results met or exceeded their expectations; and (6) over 90
percent of the companies rated staff expertise, timely assistance, and
reasonably priced MEP service fees and project proposals as important
MEP features and most were satisfied with their specific MEP programs in
these areas.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GGD-96-75
TITLE: Manufacturing Extension Programs: Manufacturers' Views
About Delivery and Impact of Services
DATE: 03/14/96
SUBJECT: Small business assistance
Research and development
Manufacturing industry
Technical assistance
Strategic planning
Productivity
Systems conversions
Surveys
IDENTIFIER: NIST Manufacturing Extension Partnership
DOD Technology Reinvestment Project
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairwoman Subcommittee on Technology Committee on
Science
House of Representatives
March 1996
MANUFACTURING EXTENSION PROGRAMS -
MANUFACTURERS' VIEWS ABOUT
DELIVERY AND IMPACT OF SERVICES
GAO/GGD-96-75
Manufacturing Extension Programs
(280133)
Abbreviations
=============================================================== ABBREV
MEP - Manufacturing extension programs
NIST - National Institute of Standards and Technology
TRP - Technology Reinvestment Project
Letter
=============================================================== LETTER
B-261875
March 14, 1996
The Honorable Constance A. Morella
Chairwoman
Subcommittee on Technology
Committee on Science
House of Representatives
Dear Chairwoman Morella:
Manufacturing extension programs (MEP) offer manufacturers assistance
in modernizing or upgrading their operations, often with state and
federal funding. The National Institute of Standards and Technology
(NIST) manages federal MEP funding through its Manufacturing
Extension Partnership Program, also known as MEP. In this report,
MEP collectively refer to all state, federal, and university
manufacturing extension programs.
In the current climate of federal budget reductions, Congress is
reevaluating its funding of MEP through NIST. One issue Congress is
considering is whether MEP services have helped improve companies'
business performance. This report responds to your request that we
obtain manufacturers' views regarding the impact of MEP services on
their business performance and the factors that affected the impact
and delivery of MEP services.
To identify the impact of MEP services on manufacturers' business
performance, we conducted a national survey of manufacturers who had
received substantive services from MEP in 1993.\1 The survey asked
manufacturers to assess the impact of MEP assistance on various
aspects of their business performance. In our August 1995 briefing
report to you,\2 we summarized the overall impact of MEP assistance
on the business performance of manufacturers we surveyed, and
presented the views of a number of companies that had not used MEP
services. We reported that about 73 percent (or 389) of 535
respondents indicated that they believed MEP assistance had
positively affected their overall business performance. About 15
percent (or 82) reported they believed MEP assistance had not
affected their overall business performance.\3
In addition, most respondents reported that MEP assistance had
positively affected their use of technology in the workplace (about
63 percent), the quality of their product (about 61 percent), and
their customers' satisfaction (about 56 percent).\4
This report presents our conclusions from further analysis of our
national survey results. These results cannot be generalized to all
manufacturers that used MEP.
Our objectives for this report were to (1) identify factors that may
have contributed to the positive impact on overall business
performance reported by the majority of survey respondents; (2)
determine whether companies' expectations were met regarding the
impact of MEP assistance on specific business performance indicators,
such as manufacturing time frames and labor productivity; and (3)
determine whether companies thought that MEP actually demonstrated
attributes they valued most, such as MEP staff expertise, timely
assistance, and reasonably priced fees. We did not verify either
positive or negative effects of MEP assistance reported by
manufacturers, and we did not evaluate the operations or management
of specific federal or state programs.
--------------------
\1 We sent questionnaires to 766 manufacturers that had completed at
least 40 hours of assistance from one of 57 MEP, in one or more of
four service categories, in 1993. Thirteen of these MEP received
NIST funding in fiscal year 1994. These 13 MEP accounted for 36
percent of the 551 total respondents to our survey. See app. II for
details on our scope and methodology.
\2 Manufacturing Extension Programs: Manufacturers' Views of
Services (GAO/GGD-95-216BR, Aug. 7, 1995).
\3 In addition, approximately 8 percent (or 41) said it was too early
to tell the effect, and another 4 percent (or 22) said they had no
basis to estimate the effect. One respondent reported a generally
negative effect on business performance.
\4 About 2 percent, or fewer, respondents reported a negative impact
on any specific business performance indicator.
BACKGROUND
------------------------------------------------------------ Letter :1
The primary mission of MEP is to give "hands-on" technical assistance
to small- and medium-sized manufacturers\5 trying to improve their
operations through the use of appropriate technologies. MEP engage
in a variety of activities to assist small- and medium-sized
manufacturers, often in partnership with other business assistance
providers such as Small Business Development Centers, community
colleges, and federal laboratories. MEP offer a wide range of
business services, including helping companies (1) solve individual
manufacturing problems, (2) obtain training for their workers, (3)
create marketing plans, and (4) upgrade their equipment and
computers. MEP assistance focuses on small- and medium-sized
manufacturers because research by the National Research Council and
others has indicated that these companies lack the resources
necessary to improve their manufacturing performance.
MEP funding typically comes from a variety of sources, which may
include federal and state government agencies, universities, private
industry, and fees. Between fiscal years 1988 and 1994 Congress
appropriated a total of $141.7 million (in 1994 dollars) to MEP
through NIST.\6 Fiscal year 1995 appropriations were $104 million.
State or local agencies are to provide matching funds for NIST grants
to individual MEP. A 1995 Battelle Memorial Institute report\7
estimated that states collectively spent about $57.7 million
specifically on MEP in fiscal year 1994. That same fiscal year,
federal MEP spending was $66 million. We were not able to determine
the amount of MEP funding from other sources of support, including
universities, private industry, and users' fees.
--------------------
\5 The Small Business Administration generally defines a small
business as having fewer than 500 employees. Some experts have
further divided small manufacturers into small firms with fewer than
100 employees and medium-sized firms with from 100 to 499 employees.
This report collectively refers to firms with fewer than 500
employees as small- and medium-sized manufacturers.
\6 NIST has allocated MEP funds from its budget as well as from the
Technology Reinvestment Project (TRP) under the Advanced Research
Projects Agency. Manufacturing Extension Programs (GAO/GGD-95-124R,
Mar. 24, 1995) lists NIST and TRP MEP funding for fiscal years 1994
and 1995.
\7 Partnerships: A Compendium of State and Federal Cooperative
Technology Programs, ed. C. M. Coburn (Columbus, OH: Battelle
Memorial Institute, 1995).
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
The responses to our survey showed that the level of companies'
involvement with MEP assistance had an important influence on
companies' assessment of the outcome of this assistance.
Specifically, the companies that supplemented MEP assistance with
their own resources, such as additional financial investments, were
more likely to report that it affected their business performance
positively. We also found that company size was a significant
factor. The relatively small companies (those with fiscal year 1994
gross sales of less than $1 million), and the relatively new
companies (those started since 1985), were most likely to report that
their overall business performance was boosted by MEP assistance.
MEP funded by NIST had the same likelihood as other MEP of receiving
positive assessments of their impact on the companies' overall
business performance.
Most of the companies that expected MEP assistance to help improve
specific areas of their business performance reported that the
results "met" or "exceeded" their expectations. About 75 percent of
the companies that received equipment modernization and plant layout
assistance reported that their expectations were met or exceeded for
improvements to production time frames. Among the companies that
received product design and development assistance, large numbers
reported an increase in the number of completed product development
projects (77 percent) and improved research quality (71 percent) that
matched or surpassed their expectations. Of the companies that
received quality improvement assistance, large proportions
experienced fulfilled expectations regarding increased sales to new
(69 percent) and repeat (74 percent) customers. However, not all of
the companies reported that their expectations were met. Between 23
and 39 percent of the companies reported that their expectations were
not met for improvements to specific business performance indicators.
In general, the companies we surveyed reported that MEP demonstrated
attributes that were most important to them. Over 90 percent of the
companies rated staff expertise, timely assistance, and reasonably
priced MEP service fees and project proposals as important attributes
for any MEP to exhibit. Most respondents also said they were
satisfied with the staff expertise (88 percent) and timeliness of
assistance (83 percent) provided by the specific program they had
used. Also, many respondents were satisfied with the fees (80
percent) and project proposal costs (81 percent) of the specific
program they had used.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
This report analyzes data from questionnaires we sent to 766
manufacturers that had completed at least 40 hours of MEP assistance
in 1993 in one or more of four service categories.\8
We obtained the names of these manufacturers from the directors of 57
MEP in 34 states. A total of 551 manufacturers (72 percent)
completed and returned the questionnaire. We also interviewed eight
manufacturers who had received MEP services and were given tours of
their manufacturing facilities in Maryland, Georgia, North Carolina,
and South Carolina. Appendix II provides more details on our scope
and methodology.
In assessing the impact of MEP services on their companies' overall
business performance, 13 percent of survey respondents reported an
extremely positive impact, 59 percent reported a generally positive
impact, and 15 percent reported no impact. (Less than 1 percent of
respondents (0.2 percent) said the assistance had had a negative
impact.) We analyzed the likelihood of the companies reporting that
the impact of MEP assistance on their overall business performance
was extremely positive, somewhat positive, or not positive, depending
on various company and program characteristics identified through the
survey. We analyzed how the reported impact of MEP assistance
related to the companies' reported age, 1994 gross sales, and number
of permanent employees. Also, we analyzed the reported impact of MEP
assistance in relation to the companies' activities associated with
the assistance--whether they made financial investments, spent staff
time, implemented recommendations, and paid for the assistance. In
addition, we analyzed how the reported impact of MEP assistance
varied according to whether programs received NIST funds.
We used logistic regression techniques to determine which factors
were statistically significant in predicting the reported impact of
MEP assistance on companies' overall business performance. The
strength of these particular statistical techniques is that they
allowed us to estimate the individual influence of each factor on the
reported impact, both before and after the influences of all other
relevant factors identified in the survey were controlled. Appendix
III provides more detailed information on our methodology, the models
tested, and the results obtained.
We used simple frequency distributions to determine whether the
companies' expectations were met regarding the impact of MEP
assistance on specific business performance indicators and to analyze
whether MEP demonstrated the attributes most valued by the companies.
Results from our work cannot be generalized to all companies that
used MEP because our questionnaire covered only companies that had
completed at least 40 hours of MEP assistance. In addition, our
results do not apply to all MEP services because we limited our
analysis to four MEP service categories.
Since we did not evaluate the operations or management of specific
federal programs, we did not obtain agency comments on this report.
However, on February 12, 1996, we discussed a draft of this report
with NIST officials, including the Director of the NIST Manufacturing
Extension Partnership Program. He agreed with the technical accuracy
of the report and offered minor clarifications, which we incorporated
into the report where appropriate.
We did our work primarily in Los Angeles, New York, San Francisco,
and Washington, D.C., from February 1995 to January 1996 in
accordance with generally accepted government auditing standards.
--------------------
\8 The four service categories were (1) quality improvement, (2)
equipment modernization and plant layout, (3) product design and
development, and (4) environmental or energy.
FACTORS THAT INFLUENCED THE
IMPACT OF MEP ASSISTANCE ON
COMPANIES' OVERALL BUSINESS
PERFORMANCE
------------------------------------------------------------ Letter :4
We analyzed several factors related to company and program funding
characteristics to determine whether they influenced the companies'
assessment of the impact of MEP assistance on their overall business
performance. We found that several company characteristics--relating
to company level of involvement with MEP assistance, and company size
and age--influenced the companies' assessment of the impact of MEP
assistance on their overall business performance. However, the
program funding characteristic we examined--whether the program
received NIST funds--did not influence the companies' assessment of
the impact of MEP assistance.
We found that the level of companies' involvement played an important
role in determining the outcome of MEP assistance. The manufacturers
that had made financial investments in their company as a result of
MEP assistance were 2.5 times more likely than those that did not to
report an "extremely positive" impact on their overall business
performance, as opposed to a "generally positive" impact. They also
were 5.6 times as likely to report a generally positive impact as
opposed to a "neutral" or "negative" impact. Companies whose staff
spent relatively more time in activities related to MEP assistance
were 1.7 times more likely to report an extremely positive impact of
MEP assistance on their overall business performance, as opposed to a
generally positive impact.\9
Furthermore, the relatively small companies, which research has
indicated are most in need of modernization assistance, were most
likely to report that their overall business performance was improved
by MEP assistance. According to the National Research Council,
small- and medium-sized manufacturers generally lack the expertise,
time, money, and information necessary to improve their manufacturing
performance.\10 We found through our survey that the companies whose
fiscal year 1994 gross sales were less than $1 million were 3.1 times
more likely to assess the impact of MEP assistance as extremely
positive, as opposed to generally positive. Likewise, the companies
started since 1985 were 2.0 times as likely as the older companies to
report an extremely positive effect of MEP assistance on their
overall business performance, as opposed to a generally positive
effect.
Our visits to manufacturers provided examples of how MEP assistance
benefited growing companies. A furniture manufacturer said his
company needed MEP assistance to make fewer mistakes in the growth
process. This manufacturer said he used MEP experts to help identify
and correct environmental and worker safety hazards, so the facility
would comply with federal workplace standards. At a company that
makes molded plastics, the company president said that the company
needed MEP assistance to guide its rapid growth. MEP helped this
company with strategic management, planning, worker training, and
quality improvement.
Our survey revealed no significant differences in how the companies
viewed the impact on their overall business performance of MEP that
did and did not receive NIST funds. MEP funding typically comes from
a variety of sources, which may include federal and state government
agencies, universities, industries, and fees.\11 The combination of
funding sources varies across programs, but our analysis revealed no
significant distinction in how the companies assessed the impact of
MEP that did and did not receive NIST funds. Specifically, MEP that
received NIST funds were equally as likely as other MEP to have their
impact on business performance rated positively by the companies. In
commenting on our analysis, NIST officials said that, given the
manufacturers' positive responses to our survey, they expected no
difference in how the manufacturers viewed the impact of MEP that did
and did not receive NIST funds. Moreover, they said that the
function of NIST funding is to help MEP serve more clients, with a
wider variety of services. Also, they said that they believed NIST
support improves programs' efficiency and effectiveness, which are
dimensions of MEP that our survey did not address.
--------------------
\9 Our analysis also indicated that the percentage of recommendations
implemented by companies may have influenced their rating of the
impact of MEP assistance. When we considered the effect of this
factor without controlling for the influence of other factors, we
found that firms which implemented relatively more recommendations
than others were 5.7 times more likely to assess the impact of MEP
assistance extremely positively, rather than generally positively,
and 5.2 times more likely to assess the impact generally positively,
rather than not positively. However, we were not able to analyze the
effect of implementing recommendations in our analysis that
controlled for the influence of other factors because there were too
few responses to perform this analysis. Only 70 percent of the
companies surveyed received recommendations and provided information
on the percentage of recommendations implemented.
\10 Learning to Change: Opportunities to Improve the Performance of
Smaller Manufacturers, National Research Council (Washington, D.C.:
National Academy Press, 1993).
\11 Whether the companies paid for MEP assistance did affect their
rating of its impact. The companies that paid fees for MEP
assistance were half as likely as those that paid no fees to credit
the assistance for having an extremely positive impact, as opposed to
generally positive impact, on their business performance.
IMPROVEMENTS TO SPECIFIC
BUSINESS PERFORMANCE INDICATORS
MET MOST COMPANIES'
EXPECTATIONS
------------------------------------------------------------ Letter :5
As part of our analysis, we compared what the companies said they
expected from MEP assistance to the results they reported.\12 We
found that most of the companies (between 61 and 77 percent) reported
that MEP assistance met or exceeded their expectations for
improvements to specific business performance indicators, such as
manufacturing time frames, the quality of market research, and sales
to new and repeat customers. However, between 23 and 39 percent of
the companies reported that their expectations were not met for
improvements to these indicators.\13
Our survey results indicate that equipment modernization and plant
layout assistance improved manufacturing time frames for most of the
companies expecting these improvements (see fig. 1). In particular,
the survey results indicate that equipment modernization and plant
layout assistance met or exceeded the expectations of a substantial
number of the companies for reducing cycle times--the times required
by machines or work stations to fully complete their sequence of
operations (77 percent)--and setup time--the time it takes to prepare
equipment for changes to production (76 percent). In addition, the
assistance met a large number of the companies' expectations for
improvements to worker output (76 percent). However, about 30
percent of companies we surveyed that received equipment
modernization and plant layout assistance reported that they did not
have their expectations met for reductions to manufacturing lead
time, the time it took them to process an order, from start to
finish, after design approval.
Figure 1: Equipment
Modernization and Plant Layout
Assistance Indicators
(See figure in printed
edition.)
Note: Companies that had expectations for no improvement, or were
unable to report the actual effect of the assistance, are not
included.
Source: GAO questionnaire.
Several companies commented on how MEP assistance affected their
efforts to improve plant layout and modernize equipment. One
manufacturer that we visited said the company was able to solve
problems with congestion and redundant product movement on the plant
floor after implementing MEP plant layout recommendations. The
company was rewarded with faster production and lower costs. Another
manufacturer responding to our survey commented that, by modernizing
equipment and improving plant layout, the company was better able to
meet its delivery schedules and, thus, satisfy its customers' needs.
Most of the companies that received product design and development
assistance reported in our survey that they achieved anticipated
improvements to quality (see fig. 2). In particular, large
proportions of these companies reported fewer incomplete product
development projects (77 percent) and improved quality of market
research (71 percent). Most of the comments we received regarding
product design and development assistance were positive. For
example, one respondent commented that the assistance it received
made it possible for the company to develop a process that it could
not have developed on its own. However, not all companies shared
such views. One respondent wrote that it took too much management
time to work with MEP consultants, and he felt that the company had
educated the consultants, and not vice versa.
Figure 2: Product Design and
Development Assistance
Indicators
(See figure in printed
edition.)
Note: Companies that had expectations for no improvement, or were
unable to report the actual effect of the assistance, are not
included.
Source: GAO questionnaire.
Our survey also indicated that most companies' expectations for
reduced product design and development time frames were satisfied.
Seventy percent of the companies reported they received anticipated
reductions in the time needed to get new products to market. One
survey respondent commented about the importance of MEP assistance in
getting a new product to market, noting that the assistance helped
the company to overcome equipment problems, which freed the company
to market new machine technology. Despite positive assessments such
as these, our survey results show that product design and development
assistance met fewer of the companies' expectations for reducing
costs of product development (66 percent) and increasing access to
new customers (65 percent), compared to other business performance
indicators.
Between 61 and 74 percent of the companies we surveyed that expected
quality improvement assistance to bolster specific business
performance indicators were satisfied with the results they received
(see fig. 3). A substantial percentage of the companies had their
expectations fulfilled regarding increased sales to repeat customers
(74 percent) and new customers (69 percent). However, our results
indicate that, for 39 percent of the companies, quality improvement
assistance did not meet expectations for reducing rework and scrap
levels.
Figure 3: Quality Improvement
Assistance Indicators
(See figure in printed
edition.)
Note: Companies that had expectations for no improvement, or were
unable to report the actual effect of the assistance, are not
included.
Source: GAO questionnaire.
Customer satisfaction was an important goal of the companies seeking
quality improvement assistance. Ninety-four percent of the companies
we surveyed regarding quality improvement assistance said they sought
the assistance in order to enhance their competitive position in the
marketplace. In interviews, several manufacturers told us that they
undertook quality improvement initiatives in order to retain and
attract customers. They said that an increasing number of customers
had high expectations for the quality of products. For example, at a
foundry we visited, the company president said that many customers of
foundry products were reducing the number of suppliers and were
working on a closer, more long-term basis with the remaining
suppliers. He said that this new customer-supplier relationship put
more emphasis on quality than ever before and that it was extremely
important to guarantee quality in order to retain customers. Another
survey respondent said that the company was "forced" to comply with a
quality assurance program by its customers, even though customers
rejected virtually none of its products.
--------------------
\12 In this section, we do not report the results of our analysis of
the environmental or energy assessment survey because we received too
few responses to conduct the analysis.
\13 Our analysis also shows that most companies (an average of 87
percent) that expected the assistance they received to have no effect
on a business indicator had their expectations met. However, for
some of these companies, MEP assistance had unanticipated positive
effects. For example, 23 percent of companies that did not expect
equipment and plant layout assistance to change production time
frames found that the assistance actually helped reduce them.
MEP HAD FEATURES COMPANIES
VALUED
------------------------------------------------------------ Letter :6
Most of the companies that responded to our survey were satisfied
with the service delivery features of the program they used. The
companies ranked MEP staff expertise, timeliness, and affordability
as the features most important to them. A majority of the companies
(80 percent or more) also responded that they were satisfied that
their program demonstrated each of the service delivery features they
deemed important (see fig. 4).
Figure 4: Companies'
Satisfaction with MEP Service
Delivery Features
(See figure in printed
edition.)
Note: Ninety percent or more of survey respondents said that each of
these service delivery features was important to them.
Source: GAO questionnaire.
About 93 percent of the companies responding to our survey rated
staff expertise as an important attribute for MEP in general, and 88
percent of respondents said they were satisfied with the expertise of
the staff at the specific program they had used. In our visits to
manufacturers, they cited several examples of how MEP staff expertise
benefited their company.
A manufacturer of heavy agricultural equipment said its three staff
engineers were fully occupied solving day-to-day manufacturing
problems, with no time to address the "big picture." The company
used MEP experts to support company efforts to develop
innovations to keep the company moving forward.
A manufacturer of souvenir and collectible items was considering
investing in over $600,000 worth of advanced production
equipment. The manufacturer told us that MEP located a
consultant who had the expertise to provide the company with an
independent opinion about whether the equipment under
consideration was appropriate for the company's needs.
A hosiery mill had installed advanced knitting machines but
continual machinery breakdowns had cut productivity by 70
percent. A senior company official told us that MEP brought
experts in training, engineering, and human resources to help
the company reverse this decline and benefit from the machinery
upgrade.
Most respondents looking for timely and affordable assistance said
they found it through MEP.\14 About 92 percent of the survey
respondents rated timely assistance as an important MEP attribute,
and 83 percent said they were satisfied with the timeliness of the
assistance provided by the program they had used. Ninety-one percent
of respondents rated reasonably priced MEP service fees and project
proposals as important MEP attributes, and most were satisfied with
the price of fees and proposals costs at their own program. Eighty
percent of respondents who paid fees were satisfied that the fees
were reasonable, and about 81 percent of respondents were satisfied
that their program had project proposal costs within their financial
means. Three hundred twenty-eight survey respondents (60 percent)
paid a fee for MEP assistance. Of those, 58 percent said that the
value added or worth of the assistance was worth more than what they
paid for it, 27 percent said the assistance was worth about what they
paid, and 11 percent said the assistance was worth less than the fee
they had paid.\15
--------------------
\14 The percentages presented in this section include only the
respondents who had an opinion.
\15 Four percent of respondents said they had no basis to judge the
worth of MEP assistance.
---------------------------------------------------------- Letter :6.1
As agreed with you, unless you announce the contents of this report
earlier, we plan no further distribution until 14 days after the date
of this letter. At that time, we will send copies to the Director of
NIST, the Secretary of Commerce, and the Chairmen and Ranking
Minority Members of congressional committees that have
responsibilities related to these issues. Copies also will be made
available to others upon request.
The major contributors to this report are listed in appendix IV.
Please contact me at (202) 512-8984 if you have any questions
concerning this report.
Sincerely yours,
JayEtta Z. Hecker, Associate Director
International Relations and Trade Issues
(See figure in printed edition.)Appendix I
QUESTIONNAIRE WITH AGGREGATE
RESPONSES
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
OBJECTIVES, SCOPE, AND METHODOLOGY
========================================================== Appendix II
At the request of Chairwoman Constance A. Morella of the
Subcommittee on Technology, House Committee on Science, we obtained
manufacturers' views regarding the impact of manufacturing extension
programs' (MEP) services on their business performance and the
factors that affected the impact of MEP services.
In August 1995, we reported\1 that most manufacturers responding to
our questionnaire believed MEP assistance had positively affected
their overall business performance. Our objectives for this report
were to analyze (1) the factors that may have contributed to the
positive impact of MEP assistance on companies' overall business
performance; (2) the question of whether companies' expectations were
met regarding the impact of MEP assistance on specific business
performance indicators, such as manufacturing time frames and labor
productivity; and (3) the issue of whether MEP actually demonstrated
attributes that companies indicated they valued most, such as MEP
staff expertise, timely assistance, and reasonably priced fees. We
did not verify either positive or negative impacts reported by
manufacturers.
To identify manufacturers that had used MEP services to survey
regarding the services' impact on their business performance and the
factors that had affected the services' impact, we (1) developed
criteria for the type of MEP our study would include, (2) located all
MEP that fit our criteria, and (3) asked these MEP for their
cooperation in supplying names of clients that met our survey
criteria (described in the following paragraphs).
Since the term "MEP" could include a variety of programs and
organizations, we consulted MEP literature and MEP experts to develop
a set of criteria to use in identifying programs to include in our
study. For the purpose of our study, we considered programs to be
relevant if their primary function was to provide direct technical
assistance to individual manufacturers, using program staff or
supervised consultants. We defined "technical assistance" as one or
more of the following activities:
providing access to and encouraging the use of innovative and/or
off-the-shelf manufacturing technologies and processes;
disseminating scientific, engineering, technical, and management
information about manufacturing;
providing access to industry-related expertise and capability in
university research departments; and
transferring advanced manufacturing (i.e., cutting edge)
technologies and techniques to companies.
Our definition excluded business assistance programs such as the
Small Business Administration's Small Business Development Centers;
business incubators;\2 financial assistance, funding, and grant
programs; joint research ventures with universities and/or federal
laboratories; on-line technical data base services; and industry
networks.
We located 80 MEP that met our criteria for inclusion and had been
established before 1994.\3 We used reports from the National
Governor's Association, the Northeast-Midwest Institute,\4 and the
Battelle Memorial Institute in Ohio that contained references to
existing MEP as the basis for identifying programs that would
possibly fit our criteria. We confirmed and updated information in
these reports by conducting structured telephone interviews with all
programs that we believed matched our criteria. We interviewed
officials from a total of 114 programs in 40 states. Eighty of them
met our criteria for inclusion and had been established before
January 1994.
Fifty-seven\5 of the 80 MEP that qualified for our study supplied us
with the names of clients that met our survey criteria. Thirteen of
these MEP received NIST funding for fiscal year 1994, accounting for
36 percent of survey respondents.\6 In an effort to determine if the
qualified programs that provided client information differed from the
qualified programs that did not, we compared the two sets of programs
on the basis of program age, total funding, federal funding, and type
of administration. The results of the comparisons indicated that
there were no significant differences between MEP that did and did
not provide client data.
We asked the 57 participating MEP to select from their records all
manufacturers that met specific criteria that we developed in
consultation with MEP officials and MEP evaluation experts. The
client had to meet the following criteria:
It had to be a manufacturing facility, which means that its
products had to belong to one or more of the manufacturing
categories in the Department of Commerce's Standard Industrial
Classification codes.\7 Our survey excluded nonmanufacturing
facilities, such as service providers or farmers.
It had to have received at least 40 hours of MEP assistance\8
in 1993. Thus, when the facility received our survey in early 1995,
at least 1 year would have elapsed since the MEP assistance ended.
MEP evaluation experts have told us that 1 year would have been
sufficient time for facilities to be able to gauge the value of the
assistance and its impact on their business performance. Experts
also have told us that 40 hours would have been enough assistance to
have had a potential effect on a manufacturer's business performance.
It had to have completed assistance in one or more of the four
categories defined in the following paragraph. In cases in
which a manufacturer completed more than one type of assistance,
we asked the MEP official to choose the primary assistance
provided to the manufacturer (i.e., the assistance requiring the
most MEP time and/or resources).
We did not verify the client information MEP provided against the
programs' records.
The assistance categories we included in our survey involved the
following:
Quality improvement. Technical assistance in planning, developing,
and implementing a quality system to help a manufacturer attain
higher quality standards.
Equipment modernization and plant layout. The evaluation and
analysis of plant layout and equipment to determine the most
efficient means of manufacturing or assembly through reorganization
of the process flow through the facility, and/or upgrading,
reconfiguring, or replacing manufacturing equipment.
Product design and development. Services to support the creation,
enhancement, or marketing of a manufacturer's product.
Environmental or energy assessment. Assessment of hazardous
materials, discharge, waste products, energy use, and other
environmental effects within a manufacturing operation.
We chose these four assistance categories because they share
important characteristics. They are types of assistance that MEP
typically offer clients, so our survey potentially could include
clients from most MEP. Also, the four types of assistance are
defined in a similar way by most MEP, according to MEP officials.
Other MEP services (such as worker training and strategic business
planning) may vary considerably from one program to another.
Finally, we selected types of assistance that were directed at
clients' manufacturing operations. MEP clients receiving
operations-related assistance were able to tell us (1) how they
expected the assistance would affect their operations and/or
performance and (2) whether or not these expectations were met.
Other types of MEP assistance--examples are material engineering,
electronic data exchange, and computer upgrading--have effects on
manufacturers' operations that are less visible and less easily
measured. As a result, manufacturers may have difficulty determining
the expected and actual impact of these types of services on their
business operations and performance.
We designed four questionnaires, each focusing on one assistance
category. In designing our survey questions, we obtained input from
National Institute of Standards and Technology (NIST) and MEP
officials, MEP evaluation experts, and managers at manufacturing
facilities. We also reviewed client surveys that MEP used.
Each questionnaire contained identical questions to obtain background
information about the respondent and to get respondents' views on the
impact of MEP services on their business performance and the factors
affecting the impact of MEP services. However, the four surveys also
had unique questions asking about the expected and actual outcomes of
the assistance, because each type of assistance focuses on a
different aspect of manufacturers' operations. We tailored these
questions to ask about the kind of impacts that reasonably could be
expected to result from the particular kind of assistance received.
As part of our survey development, we tested all four surveys with
manufacturers who had received MEP assistance in Texas, Iowa, New
York, and Kansas. We chose those states in order to cover diverse
areas of the country where MEP are located. We also interviewed
eight manufacturers who had received MEP services and were given
tours of their manufacturing facilities in Maryland, Georgia, North
Carolina, and South Carolina. We visited these southern states
because MEP directors had agreed to arrange for us to meet selected
clients. We asked the manufacturers about their experiences with MEP
services and the impact of those services on their business
performance.
Our final surveys initially were mailed to a total of 843
manufacturers from February 1995 through March 1995. Follow-up
mailings were made through May 1995. Each manufacturer was sent one
survey, based on MEP information on the primary type of service the
manufacturer had received.
The primary reason manufacturers did not respond to our survey was
their inability to recall MEP assistance they had received. We wrote
letters asking the nonrespondents why they did not return our survey.
We received responses from 60 companies out of 274 nonrespondents.
About 33 percent told us that no one at their facility could recall
the assistance received in 1993 and/or that we had addressed the
survey to a person who no longer worked at the facility. On the
basis of this information, in addition to other information provided
by our nonrespondents, we reduced our survey population from 843 to
766.
We obtained an overall response rate of 72 percent across all four
surveys. Response rates varied from a low of 63 percent for the
environmental/energy survey to a high of 76 percent for the quality
improvement survey.
Our analysis of the companies that did and did not respond to our
survey found nothing to indicate that our results would have been
different if the nonrespondents had completed our questionnaire. The
respondents and nonrespondents were similarly distributed across
different geographic locations and different MEP.
Since we did not evaluate the operations or management of specific
federal programs, we did not obtain agency comments on this report.
However, on February 12, 1996, we discussed a draft of this report
with NIST officials, including the Director of the NIST Manufacturing
Extension Partnership Program. He agreed with the technical accuracy
of the report and offered minor clarifications, which we incorporated
into the report where appropriate.
We did our work primarily in Los Angeles, New York, San Francisco,
and Washington, D.C., from February 1995 to January 1996 in
accordance with generally accepted government auditing standards.
--------------------
\1 GAO/GGD-95-216BR.
\2 Incubator facilities provide office and lab space for start-up
companies at below-market rates. Shared support services such as
clerical, reception, and data processing often are made available, as
well.
\3 Since our survey focused on manufacturers receiving MEP services
in 1993 (for reasons explained in the text) we limited our study to
MEP that were operating before 1994.
\4 The Northeast-Midwest Institute provides information and analysis
to Members of Congress and the public related to economic development
issues affecting the Northeast-Midwest region.
\5 Of the remaining 23 MEP, 7 were willing to provide client
information but did not have any clients meeting all of our survey
criteria. Ten declined our request because of concerns over client
confidentiality, three never responded to our request, and three
others did not participate for other reasons.
\6 According to NIST officials, 5 of the 13 MEP received NIST funds
in fiscal years 1993 and 1994. The other eight MEP were first
awarded NIST funding in fiscal year 1994.
\7 The Standard Industrial Classification is the statistical
classification standard underlying all establishment-based federal
economic statistics classified by industry. The classification
covers the entire field of economic activities and defines industries
in accordance with the composition and structure of the economy.
\8 The 40 hours need not have been consecutive. Assistance may have
been provided by MEP staff or by consultants affiliated with MEP. In
cases involving consultants, MEP should have performed a case
management role.
TECHNICAL APPENDIX: LOGLINEAR AND
LOGISTIC METHODOLOGIES AND
ANALYSIS RESULTS
========================================================= Appendix III
We used logistic regression techniques to determine which factors
were statistically significant in predicting the reported impact of
MEP assistance on companies' overall business performance. We began
our analysis by considering nine factors that may have affected how
the manufacturers we surveyed assessed the impact of MEP assistance
on their overall business performance. The factors included the
following characteristics of those manufacturers: (1) the number of
permanent employees as of January 1, 1995, (2) the number of hours
company staff devoted to MEP assistance, (3) the year the company
started operating, (4) the company's gross sales in fiscal year 1994,
(5) whether the company paid any fees for MEP assistance, (6) whether
the company made any financial investments as a result of the
assistance, (7) whether the assistance included recommendations, and
(8) the percentage of MEP recommendations the company implemented.
We also considered whether the company used a program that had
received NIST funds. These factors all are listed in the first
column of table III.1.
Table III.1
Odds Ratios from Logistic Regression
Analysis
Multi- Multi-
Categories Bivari variat Bivari variat
Factor contrasted ate e ate e
---------------------------- ------------------ ------ ------ ------ ------
Number of 0 = 100 or more; 2.2* \a 0.8 \a
permanent employees 1 = 20 -99;
2 = less than 20
Company staff hours devoted 0 = less than 1.3 1.7* 2.2* 2.0*
to MEP assistance 100;
1 = 100 -250;
2 = more than 250
Year the company 0 = before 1985; 2.5* 2.0* 0.7 0.8
started operating 1 = since 1985
Gross annual sales for 0 = over $1 4.0* 3.1* 1.0 1.4
fiscal year 1994 million;
1 = under $1
million
Whether the company 0 = no; 0.5* 0.5* 1.0 0.8
paid any fees for MEP 1 = yes
assistance
Whether the company made 0 = no; 2.8* 2.5* 7.0* 5.6*
financial investments 1 = yes
Whether the assistance 0 = no; 1.9\b 1.6 1.8* 1.3
included recommendations 1 = yes
Percentage of MEP 0 = few or none; 5.7* \c 5.2* \c
recommendations 1 = some;
the company implemented 2 = all or almost
all
Whether the company used MEP 0 = yes; 1.1 1.3 0.9 1.1
that 1 = no
received NIST funds
--------------------------------------------------------------------------------
Note: Asterisk indicates odds ratios that are statistically
significant at the 0.05 level.
\a Number of permanent employees was dropped from the multivariate
analysis because of its strong association with gross sales. Each of
these two indicators of company size were significantly related to
assessments when the other indicator was ignored. However, when we
controlled for gross annual sales, the effect of number of permanent
employees was not statistically significant.
\b Significant at the 0.06 level of confidence.
\c The percentage of recommendations implemented was dropped from the
multivariate analysis because there were too few responses to perform
the analysis. Only 70 percent of the companies received
recommendations and provided information on the percentage of
recommendations implemented.
Some of these factors had many categories. We used loglinear methods
to determine which of those categories differed with respect to
companies' assessment of the overall impact of MEP. We combined the
categories that were not significantly different from one another.
The categories which ultimately were contrasted with one another are
given in the second column of table III.1. For the purpose of our
analysis, the factors were used as the independent variables.
We used simple bivariate logistic regression models to estimate the
individual influence of each factor on the reported impact of MEP
assistance, without controlling for the influence of all other
relevant factors identified in the survey. We estimated which of the
nine factors, as categorized in Table III.1, were related to (1) the
odds on the overall impact of MEP being assessed as extremely
positive versus generally positive and (2) the odds on the overall
impact of MEP being assessed as generally positive versus negative or
neutral.\1 Our bivariate estimates are given as odds ratios in the
third and fifth columns of table III.1.
As can be seen in that table, seven of the nine factors had a
significant relationship\2 to the likelihood that companies assessed
the impact of MEP assistance as extremely positive, as opposed to
generally positive. In addition, four of the nine factors were
significantly related to the odds of companies assessing the impact
of MEP assistance as generally positive, as opposed to neutral or
negative.
The bivariate odds ratios have a straightforward interpretation. The
odds ratio gives an estimate of how each factor, as categorized\3 in
column 2 of Table III.1, affected companies' assessment of MEP
assistance. For example, the companies with 20-99 employees were
more than twice as likely as the companies with 100 or more employees
to assess the impact of MEP as extremely positive as opposed to
generally positive. Likewise, the companies with less than 20
employees were more than twice as likely as the companies with 20 to
99 employees to assess the impact of MEP as extremely positive, as
opposed to generally positive. Similar interpretations can be given
to the other odds ratios in the table.\4 The bivariate odds ratios
are estimates that do not take into account the effects of other
variables.
We also undertook multivariate analysis of the data. Multivariate
analysis also estimated the individual effect of each factor on the
reported impact of MEP assistance, but it controlled for the
influence of all other relevant factors. It is necessary to control
for the influence of multiple factors because some factors are
associated with others, making it impossible to isolate their
individual effect on the dependent variable. Our multivariate
analysis did not include two factors used in the bivariate analysis:
the number of permanent employees and the percentage of
recommendations companies had implemented.\5
The odds ratios in the fourth and sixth columns of table III.1
provide the results of multivariate analysis. Odds ratios that are
marked by an asterisk represent statistically significant effects.
Five factors had significant effects on the odds of whether programs
were assessed extremely positively as opposed to generally
positively: (1) the number of company staff hours devoted to the
assistance, (2) when the company started operating, (3) the company's
1994 fiscal year gross sales, (4) whether the company paid any fees
for the assistance, and (5) whether the company made any financial
investments as a result of the assistance. Only two factors had
significant effects on whether assessments were generally positive as
opposed to neutral or negative: (1) the number of company staff
hours devoted to the assistance and (2) whether the company made any
financial investments as a result of the assistance.
Many of the significant effects from the multivariate analysis are
quite sizable. For example, the companies that made financial
investments were 2.5 times as likely as those that had not made
financial investments to assess the impact of MEP assistance as
extremely positive, as opposed to generally positive. The companies
that made financial investments also were 5.6 times as likely as
companies that had not made financial investments to assess the
impact of MEP assistance as generally positive, as opposed to neutral
or negative. Other odds ratios can be similarly interpreted.\6
Our letter report features the results of the multivariate analysis.
The multivariate estimates may differ from the bivariate estimates
because the multivariate analysis controlled for the effects of all
other factors when estimating the influence of one factor. Bivariate
analysis estimates the influence of one factor without controlling
for the effects of other factors. In general, the multivariate and
bivariate estimates for each factor are similar, with two exceptions.
The first exception is company staff hours devoted to MEP assistance.
In the bivariate analysis, this factor was unrelated to whether
companies assessed the impact of MEP assistance as extremely positive
versus generally positive. However, multivariate results indicate
that company staff hours were significantly related to companies'
assessment of the impact of assistance as extremely positive, as
opposed to generally positive. We believe that the significance
varies because of a relationship between company size and the number
of company staff hours spent on MEP assistance. In particular,
larger companies devoted more staff hours to the program. In order
to accurately assess the independent influence of company staff
hours, we needed to control for company size. Our multivariate model
controls for company size by including the variable that measures
gross sales. Therefore, the multivariate model provides a more
accurate assessment of the impact of company staff hours, independent
of company size.
The second exception was the factor measuring whether MEP assistance
included recommendations. In our bivariate analysis, this variable
was significantly related to both extremely positive and generally
positive assessments. However, its significance disappeared in our
multivariate analysis. Companies receiving recommendations were more
likely to devote more staff hours to the program and to make
financial investments as a result of MEP assistance. Therefore, when
the multivariate analysis controlled for company staff hours spent on
the assistance and financial investments made as a result of the
assistance, the effect of recommendations was rendered insignificant.
--------------------
\1 Of the 472 companies that provided us with information on the
overall impact of MEP on their business performance, 71 (15 percent)
assessed the impact as extremely positive, 318 (67 percent) assessed
the impact as generally positive, and 83 (18 percent) assessed the
impact as negative or neutral. Using this data, the overall odds on
MEP being assessed as extremely positive versus generally positive
were 71/318 = 0.22. That is, 22 companies viewed MEP extremely
positively for every 100 that viewed MEP as having a generally
positive impact. The overall odds on the program being assessed as
generally positive versus negative or neutral were 318/83 = 3.81.
This implies that 381 companies assessed MEP as having a generally
positive impact for every 100 that viewed the overall impact of MEP
as neutral or negative.
\2 For six factors, the confidence level was 0.05. One additional
factor--whether the assistance included recommendations--was
significant at the 0.06 level of confidence.
\3 Our bivariate and multivariate analysis directly contrasted the
factor categories. Where a factor had two categories, we compared
the category coded 1 with the category coded 0. For the factors with
three categories, we scored the categories linearly with codes of
0,1, and 2.
\4 The size of an effect is indicated by the odds ratio. A factor
with an estimated odds ratio of 1.0 indicates that the factor
categories being contrasted have equal likelihood of influencing
companies' assessment of the impact of MEP assistance. An odds ratio
of 0.5 indicates that one category is one half as likely as the other
category to result in a positive assessment by companies; an odds
ratio of 2.0 indicates that one category is twice as likely as the
other category to result in a positive assessment by companies.
\5 The number of permanent employees was dropped from the
multivariate analysis because of its strong association with gross
sales. While each of these two indicators of company size were
significantly related to assessments when the other was ignored, when
we controlled for gross sales, the effect of number of permanent
employees became insignificant. Also, we omitted from our analysis
the percentage of MEP recommendations the company implemented. A
substantial percentage of companies (30 percent) had received no
recommendations.
\6 Like the bivariate analysis, the categories compared in the
multivariate analysis had a linear relationship to one another.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix IV
GENERAL GOVERNMENT DIVISION,
WASHINGTON, D.C.
Susan S. Westin, Assistant Director
Douglas Sloane, Supervisory Social Science Analyst
Stuart Kaufman, Senior Social Science Analyst
Barry L. Reed, Senior Social Science Analyst
Rona Mendelsohn, Senior Evaluator (Communications Analyst)
LOS ANGELES REGIONAL OFFICE
Patrick F. Gormley, Assistant Director
Amy L. Finkelstein, Evaluator-in-Charge
Edward Laughlin, Senior Evaluator
*** End of document. ***