Performance Measurement: Efforts to Evaluate the Advanced Technology
Program (Letter Report, 05/15/95, GAO/RCED-95-68).

The Advanced Technology Program, which is run by the National Institute
of Standards and Technology (NIST), seeks to provide support on a
cost-sharing basis to research and development projects in industry.
These projects are intended to stimulate economic growth and improve the
competitiveness of U.S. industry. Funding for the program has risen from
$68 million in fiscal year 1993 to $431 million in fiscal year 1995,
more than doubling each year.  The President has set a goal of $750
million in funding for the program by 1997. The agency has reported
short-term results that it claims show the program is making an impact.
This report (1) analyzes these short-term results and (2) reviews NIST's
plans for evaluating the program in the future.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  RCED-95-68
     TITLE:  Performance Measurement: Efforts to Evaluate the Advanced 
             Technology Program
      DATE:  05/15/95
   SUBJECT:  Research and development
             Research programs
             Program evaluation
             Scientific research
             Research program management
             Business assistance
             Strategic planning
             Economic analysis
IDENTIFIER:  NIST Advanced Technology Program
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Ranking Minority Member, Committee on Science, House of
Representatives

May 1995

PERFORMANCE MEASUREMENT - EFFORTS
TO EVALUATE THE ADVANCED
TECHNOLOGY PROGRAM

GAO/RCED-95-68

Evaluating the Advanced Technology Program

(307719)


Abbreviations
=============================================================== ABBREV

  ATP - Advanced Technology Program
  CBO - Congressional Budget Office
  GAO - General Accounting Office
  NIST - National Institute of Standards and Technology
  R&D - research and development

Letter
=============================================================== LETTER


B-259591

May 15, 1995

The Honorable George E.  Brown, Jr.
Ranking Minority Member
Committee on Science
House of Representatives

Dear Mr.  Brown: 

This report responds to your request concerning the Advanced
Technology Program (ATP), which is administered by the National
Institute of Standards and Technology (NIST) within the Department of
Commerce.  ATP's purpose is to provide support on a cost-sharing
basis to research and development (R&D) projects in industry.  These
projects are intended to have a significant potential for stimulating
economic growth and improving the competitiveness of U.S.  industry. 
Funding for ATP has grown from $68 million in fiscal year 1993 to
$431 million in fiscal year 1995, more than doubling each year.  The
President has set a goal for the program's funding to reach $750
million by 1997. 

In light of these significant budget increases, the Congress is
interested in ATP's impact.  Although NIST recognizes that it is too
early to measure ATP's long-term economic effects, the agency has
reported short-term results that, it says, indicate the program is
making an impact.  As agreed with your office, we (1) analyzed these
short-term results and (2) reviewed NIST's plans for evaluating ATP
in the future, as reported in the NIST document entitled Setting
Priorities and Measuring Results at the National Institute of
Standards and Technology, dated January 31, 1994.  In addition, we
are presenting information on other ATP evaluation efforts that NIST
has planned or under way. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

Evaluating the Advanced Technology Program poses many challenges. 
For example, while funded projects are intended to be technical
successes and to have a commercial impact, several years can elapse
between the end of technical work and the realization of such an
impact.  NIST has, however, identified six short-term results in the
Setting Priorities document that it believes demonstrate the program
is making an impact.  While all six have limitations, our analysis
shows that four are overstated or lack adequate support.  For
example, NIST projected ATP's impact from one joint venture to the
entire industry of approximately 800 companies. 

NIST also summarized its plans for evaluating ATP in Setting
Priorities.  This summary includes indicators, such as the number of
technical milestones completed and the number of joint ventures
formed, that we do not believe reflect the long-term economic success
of the program.  Setting Priorities does, however, provide
descriptive information that may be useful to NIST officials in
managing the program. 

According to NIST officials, NIST has other evaluation efforts
planned or under way besides those included in Setting Priorities. 
These efforts include engaging the advice and services of the
nation's leading economists in impact assessment and evaluation. 
NIST has also put in place an extensive data collection system to
support ATP's evaluation.  The results of some of these evaluation
efforts may not be known for some time. 


   BACKGROUND
------------------------------------------------------------ Letter :2

ATP was established by the Omnibus Trade and Competitiveness Act of
1988 (P.L.  100-418).  The program is intended to assist U.S. 
businesses in creating and applying the generic technology and
research results necessary to (1) commercialize significant new
scientific discoveries and technologies rapidly and (2) refine
manufacturing technologies.  Funding for ATP is awarded through
announced competitions.  Single companies that receive awards are
reimbursed for the direct costs of their proposed research but must
pay for all overhead costs.  Joint ventures, which consist of two or
more companies, are reimbursed for both their direct and overhead
costs but must provide more than 50 percent of the total funding for
their project.  ATP supports high-risk projects that have the
potential for eventual substantial widespread commercial application. 
Since the first competition in fiscal year 1990, NIST has funded 177
ATP projects.  As of April 1995, 12 projects had been completed. 

NIST summarized the results of its initial evaluation efforts and
future plans in Setting Priorities, which received wide distribution. 
NIST distributed 3,800 copies of this document to the Congress,
administration officials, and industry.  NIST officials also
submitted the document to the Congress during the fiscal year 1995
appropriations hearings. 

Evaluating ATP poses many challenges.  For example, ATP research
projects are intended not only to be technical successes but also to
have commercial results.  The linkage between technical work and
commercial results may not always be direct and may be subject to
interpretation.  Also, several years can elapse between the end of
technical work and the realization of commercial results. 


   ATP'S RESULTS REPORTED IN
   SETTING PRIORITIES
------------------------------------------------------------ Letter :3

NIST cites six examples of ATP's short-term results in Setting
Priorities.  While all six have limitations, four overstate ATP's
success or lack adequate support.  Specifically, NIST reported that
as a result of ATP

  -- total U.S.  research on advanced technologies for printed wiring
     boards has quadrupled,

  -- participants have pursued research they otherwise could not have
     pursued,

  -- participants have forged new relationships with companies and
     government or academic laboratories, and

  -- the number of joint R&D ventures in private industry has
     increased. 


      ATP'S IMPACT ON PRINTED
      WIRING BOARD INDUSTRY
      OVERSTATED
---------------------------------------------------------- Letter :3.1

NIST overstated ATP's impact on the printed wiring board industry. 
On the basis of ATP's impact on one five-member joint venture, NIST
concluded that total U.S.  R&D in the U.S.  printed wiring board
industry had quadrupled.  NIST reported that "total U.S.  R&D work on
advanced technologies for printed wiring boards essential to all
modern electronic devices more than quadrupled as a result of the
ATP." According to NIST officials, this statement is based on a
third-party review that resulted in a report entitled Advanced
Technology Program:  Economic Study of the Printed Wiring Board Joint
Venture After Two Years.  This study was designed to measure the
impact, after 2 years, of a 5-year ATP-supported project undertaken
by a five-company joint venture and does not assess ATP's impact on
the entire U.S.  printed wiring board industry.  Specifically, the
statement is based on the study's finding that

     "of the 29 major project areas under investigation [in the joint
     venture's research project], the participants reported that on
     average only 6.5 projects would have been started in the absence
     of the ATP award.  In fact, a number of critical projects would
     not have been attempted in the absence of the joint venture."

According to an industry association representative, however, there
are approximately 800 merchant manufacturers in the printed wiring
board industry, many of which are active in R&D.\1 Discussions with
the industry association indicate that the industry spent at least
$26.5 million on R&D in 1992.  In comparison, the ATP-supported joint
venture spends $5.7 million per year, on average, on printed wiring
board research. 

We believe that NIST's conclusion that total U.S.  R&D on printed
wiring boards quadrupled as a result of ATP was an overstatement
because the third-party study was limited to a single five-member
joint venture in an industry that contains over 800 merchant
manufacturers, many of which engage in R&D.  The evidence presented
by NIST supports statements only about the five-member joint venture,
not about the entire U.S.  industry. 


--------------------
\1 The industry association is the Institute for Interconnecting and
Packaging Electronic Circuits (IPC).  Merchant manufacturers make
printed wiring boards and sell them to companies. 


      CLAIMS OF INCREASED
      HIGH-RISK RESEARCH AND NEW
      RELATIONSHIPS NOT ADEQUATELY
      SUPPORTED BY SURVEY DATA
---------------------------------------------------------- Letter :3.2

In reporting the most important effects of ATP on the basis of a
survey of early award recipients, NIST was selective in its use of
the survey data.  NIST reported that the most important effect cited
by award recipients was "the ability to pursue promising lines of
research that they otherwise could not have followed." According to
NIST officials, this statement is based on responses to the survey's
question "What would you say is the single most important effect that
the ATP award has had on your organization thus far?" Fifteen out of
28 responses in the study were categorized as saying "the ability to
afford and engage in this kind of high-risk, long-term research."\2

However, responses to another question in the same survey provided
conflicting information.  When asked "In the absence of this ATP
award, would your organization have pursued the development of this
technology?" nearly as many respondents--14 out of 26--responded
affirmatively.  Four respondents said they definitely would have; 10
said they probably would have; 7 said they probably would not have;
and 5 said they definitely would not have.  In response to a
subsequent question, the 14 respondents said they would have pursued
the development of the technology at a different level of effort. 
Thus, although 15 participants believed that ATP enabled them to
pursue this kind of high-risk, long-term research, 14 participants in
the same study believed that they would at least probably have
pursued the technology even without the ATP award, although at a
different level of effort. 

On the basis of the same survey, NIST reported that the second most
important effect cited by early ATP participants was "forging new
relationships between companies, and between companies and government
or academic labs." However, several discrepancies exist.  First, the
survey results do not refer to a second most important effect, nor
were the participants asked to identify a second most important
effect.  NIST officials said this statement was supported by the
second most frequent response to the question about the "single most
important effect" of the ATP award.  However, the second most
important effect cannot be inferred from the second most frequent
response because the frequency of the response to this question does
not say anything about the relative importance that individual
respondents would have ascribed to this effect.  The survey would
have to have asked the participants specifically what they believed
the second most important effect of ATP was in order to make that
determination. 

Another discrepancy is that the second most frequent responses were
categorized as "the benefits that can flow from industry-industry
collaboration," but made no mention of "government or academic labs,"
as NIST reported.  To support its conclusion that the second most
important effect was "forging new relationships between companies,
and between companies and government or academic labs" NIST officials
said that they had also included information from responses to
another survey question.  The other question asked participants to
rate a list of potential effects of ATP.  According to NIST
officials, the item rated second highest on the list of potential
effects for this question was the basis for NIST's statement about
the "second most important effect." This item was "enhanced the
technology infrastructure by strengthening linkages between sectors
(industry-government, industry-university) and/or within sectors
(industry-industry)."

However, NIST did not base its "most important effect" on the same
question.  Since NIST based its "second most important effect" on
this question (respondents were asked to rate a list of potential
effects of ATP), in order to be consistent, the highest rated
response to the same question should have been the "most important
effect." However, the highest rated item on this list is
"collaboration and strategic alliances." This conflicts with the
responses mentioned previously, which said that the "single most
important effect" was "the ability to afford and engage in this kind
of high-risk, long-term research." This inconsistent methodology
casts doubt on NIST's reporting of ATP's most important effects. 


--------------------
\2 Twenty-six organizations participated in the study.  However, two
participants provided two answers to this question about ATP's
"single most important effect." Two of these answers are included in
the 15 responses classified as "the ability to afford and engage in
this kind of high-risk, long-term research."


      RELATIONSHIP BETWEEN ATP AND
      INCREASE IN NUMBER OF JOINT
      VENTURES NOT ADEQUATELY
      SUPPORTED
---------------------------------------------------------- Letter :3.3

NIST's conclusion that ATP has increased the number of joint R&D
ventures in private industry is not adequately supported.  NIST
states that ATP was responsible for an increase in U.S.  joint
ventures, despite a variety of possible causes.  According to NIST,

     "Profiles also suggest that the ATP has led--as desired--to an
     increase in joint R&D ventures in private industry.  In the
     first four competitions, approximately 125 joint ventures
     involving over 800 organizations were formed to apply to the
     ATP."

However, the only support NIST gave us for this statement is the fact
that 125 joint ventures submitted proposals to ATP.  Although the
number of joint R&D ventures has increased, there are several reasons
to question a direct relationship between this increase and ATP.  The
number of joint R&D ventures has steadily increased since 1986--years
before NIST made its first ATP award.  Some explanations for the
causes of this trend and for the formation of joint ventures are
unrelated to ATP.  For example, the National Science Foundation cites
the passage of the National Cooperative Research Act of 1984 as one
reason for this growth.  The Foundation explains that this act
encourages research collaboration among industry competitors by
better defining joint R&D ventures and protecting them from antitrust
suits by limiting potential liability.  We believe that NIST's
conclusion about the causal relationship between ATP and an increase
in the number of joint ventures lacks adequate support. 


   NIST'S FUTURE PLANS FOR
   EVALUATING ATP IN SETTING
   PRIORITIES
------------------------------------------------------------ Letter :4

The evaluation plan, as presented in Setting Priorities, includes
several measures that NIST expects will indicate the long-term
economic success of ATP projects.  However, some of these measures
may not indicate the economic success of ATP. 

One of the measures that NIST believes will indicate the long-term
economic success of ATP projects is "straightforward tracking of
technical milestones."\3

However, achieving technical milestones may not be a valid indicator
of the economic success of ATP projects because technical advancement
does not always lead to economic success.  For example, earlier
versions of the ATP evaluation plan pointed to one ATP project that
was achieving all of its technical milestones as evidence of the
project's likely success in stimulating economic growth.  The lead
company involved in this joint venture, however, went bankrupt before
the project was completed.  Although the other company in the joint
venture has stated its intention to continue the joint venture's
commercialization plan, the lead company's bankruptcy reduces the
likelihood of future economic effects being realized from this ATP
project. 

Tracking the completion of technical milestones for ATP projects
provides helpful information to ATP managers who need to know whether
this vital step in the commercialization process is being achieved. 
However, using this information as an indicator of "long-term
economic success" may create the false expectation that technical
success will result in commercial success. 

"Increased collaborations and strategic alliances [between
companies]" is another measure that NIST expects to indicate
long-term economic success.  However, the number of collaborations
and strategic alliances may not indicate ATP's economic success.  A
joint venture is one form of collaboration or strategic alliance that
can occur between companies.  As the previously cited example of the
bankrupt company and its collaboration shows, the use of this measure
to indicate "long-term economic success" may create the false
expectation that collaboration will lead to commercial success. 

The ATP evaluation plan summarized in Setting Priorities shows that
NIST intends to continue providing descriptive data on the program
and its operations.  Two of the five major components of the
evaluation plan focus on obtaining this type of information and are
descriptive in nature:  (1) assessing ATP's operational activities
and (2) profiling applicants, recipients, technologies, and projects. 
This information will include descriptive data about the program's
operations, participants, and monitoring activities as integral
parts. 

This information is helpful to ATP officials in managing the program. 
Collection of these data, however, does not provide the Congress with
information about the program's impact and economic success. 


--------------------
\3 Technical milestones are significant points in the course of a
research project.  They consist of individual research tasks with
estimated completion dates that are part of the project's overall
timetable.  Technical milestones might consist of the estimated
completion dates of experiments or tests in the project. 


   ADDITIONAL ATP EVALUATION
   EFFORTS
------------------------------------------------------------ Letter :5

According to NIST officials, program evaluation has been a part of
ATP from its beginning, and the development of a long-term evaluation
strategy is an ongoing NIST process.  NIST says that at this point in
ATP's history, its approach to evaluating ATP is to lay the
groundwork to provide metrics for the program's results at the
earliest possible time. 

NIST's evaluation efforts include engaging the advice and services of
the nation's leading economists in impact assessment and evaluation. 
NIST has also put in place an extensive data collection system to
support its ATP evaluation efforts.  In addition, NIST says it is
conducting microeconomic case studies and supporting the development
and use of economic models for projecting outcomes of ATP. 


   CONCLUSIONS
------------------------------------------------------------ Letter :6

It is too early to determine ATP's long-term economic impact;
therefore, there has not been a complete assessment of ATP. 
Evaluating ATP will be challenging.  For example, ATP research
projects are intended not only to be technically successful but also
to have a commercial impact.  The linkage between technical work and
commercial results may not always be direct and may be subject to
interpretation. 

NIST has reported short-term results in Setting Priorities that, it
says, indicate that the program is making an impact.  However, our
analysis indicates that these results are overstated or lack adequate
support.  Thus, judgments about the economic success of ATP should
not be based solely on the information in Setting Priorities.  In
addition, some of the indicators contained in Setting Priorities that
NIST proposes to track for future evaluations of ATP, namely
technical milestones and the number of collaborations and strategic
alliances, may create false expectations of the program's economic
success.  Neither of these indicators necessarily reflects the
long-term economic success of the program.  According to NIST, other
efforts are under way that will support studies of the program's
long-term outcomes as soon as such studies are feasible. 


   AGENCY COMMENTS AND OUR
   EVALUATION
------------------------------------------------------------ Letter :7

We provided a draft of this report to the Department of Commerce for
written comments.  These comments, along with our detailed responses,
are provided in appendix I.  In addition, at the Secretary of
Commerce's request, we met with the Under Secretary for Technology
and the Director of NIST to discuss the draft report in more detail. 
Specifically, NIST made the following observations about our draft
report: 

  -- It overestimated the amount of "advanced" R&D in the printed
     wiring board industry by citing industry figures that include
     R&D that is not "advanced." However, we point out in our
     response that the broader industry figure is appropriate to use
     for comparison purposes because the ATP project also includes
     R&D that is not "advanced."

  -- It introduced a negative bias to our conclusions by including
     only partial responses to a survey question.  We have added
     language to the final report reflecting the additional
     information contained in responses to this survey question. 
     However, this does not change our analysis or conclusions
     because the information still suggests that as many as half of
     the ATP projects would have been undertaken even without ATP
     support, although at a different level of funding.  Moreover, a
     Congressional Budget Office study reached a conclusion similar
     to ours on the basis of the same data. 

  -- It overlooked evidence and made it appear that NIST's
     conclusions about ATP's impact on forging relationships between
     companies and government or academic labs lacked support.  Our
     review of this evidence is included in this report and shows
     that NIST's conclusions are based on an inconsistent methodology
     used in analyzing the evidence. 

  -- It overlooked evidence supporting NIST's statement that ATP has
     led to an increase in the number of joint ventures.  However,
     the evidence provided by NIST still does not demonstrate that
     ATP has caused an increase in the number of joint ventures for
     primarily two reasons.  First, the National Cooperative Research
     Act was influencing the number of joint ventures over the same
     time period.  Second, NIST has no evidence that shows why joint
     ventures that applied to ATP formed in the first place.  NIST
     currently has a study under way to determine that information. 
     The fact that the joint ventures registered with the Federal
     Trade Commission or the Department of Justice when applying to
     ATP is irrelevant because joint ventures are not required to
     register with these agencies when they form. 

NIST's comments on our draft report also include important
qualifications that help dispel false expectations about the
indicators of long-term economic success in Setting Priorities.  Had
these qualifications appeared in Setting Priorities, one would have
been less likely to arrive at false conclusions about the program's
impact. 

Our draft report contained a proposed recommendation that the
Secretary of Commerce direct NIST officials to develop an evaluation
strategy that includes measures of the program's outcomes.  In
commenting on our draft report, NIST said that it intends to continue
to refine the ATP evaluation plan through the use of microeconomic
case studies and economic models for projecting ATP's outcomes. 
Therefore, we have withdrawn that proposed recommendation. 


   SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :8

In conducting our analysis, we interviewed the NIST senior economist
responsible for evaluating ATP and examined Setting Priorities.  The
NIST economist said that this document summarizes the evaluations of
the program conducted to date, as well as the plan for evaluating ATP
in the future.  We investigated all statements about ATP in this
document by reviewing the supporting studies and data to determine
their consistency with NIST's reported statements.  We analyzed the
ATP evaluation plan's "indicators of future economic success" but
were unable to analyze the usefulness of those indicators that were
too general for understanding the effects of ATP.  For example,
indicators that include terms such as "technological infrastructure"
and "enabling technologies" do not clearly identify what they measure
or how they are related to the economic success of ATP.  In addition,
NIST supports some of its statements about ATP's effects with
references to two NIST-supported studies.  Although we examined these
studies, we did not evaluate them for their validity.  NIST's
evaluation of ATP is an ongoing process.  When we had nearly
completed our work, NIST provided us with a copy of NIST Industrial
Impacts:  A Sampling of Successful Partnerships, which contains
anecdotes about ATP awards.  We did not evaluate this document.  We
also consulted economists, the R&D evaluation literature, and a trade
association representative.  We conducted our review from January
1994 to April 1995 in accordance with generally accepted government
auditing standards. 


---------------------------------------------------------- Letter :8.1

As arranged with your office, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 30 days after the date of this letter.  At that time, we will
send copies to the Secretary of Commerce; the Director, NIST; the
Director, ATP; the Inspector General, Department of Commerce; the
Director, Office of Management and Budget; and other interested
parties.  We will make copies available to others on request. 

Please contact me at (202) 512-3841 if you or your staff have any
questions.  Major contributors to this report are listed in appendix
II. 

Sincerely yours,

Victor S.  Rezendes
Director, Energy and
 Science Issues




(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
COMMERCE
============================================================== Letter 



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


The following are GAO's comments on the Department of Commerce's
letter dated January 27, 1995. 


   GAO'S COMMENTS
------------------------------------------------------------ Letter :9

1.  Our work focused specifically on the information provided by the
National Institute of Standards and Technology (NIST) in its document
entitled Setting Priorities and Measuring Results at the National
Institute of Standards and Technology.  We examined (1) the
short-term results that NIST says indicate the impact of the Advanced
Technology Program (ATP) and (2) the measures that NIST expects will
indicate ATP's long-term economic success.  We did not address the
progress made by NIST in implementing its evaluation plan.  After
reviewing additional support provided by NIST, we maintain that our
original assessment of NIST's conclusions about short-term results is
valid.  None of the information provided in the comments refutes our
original conclusions.  We have included additional information about
NIST's ongoing evaluation efforts beyond those cited in Setting
Priorities.  We also maintain that the indicators of long-term
economic success included in Setting Priorities may create false
expectations.  Although NIST's comments on our draft report included
important qualifications that help dispel false expectations (e.g.,
"accomplishment of technical milestones is a necessary, but
insufficient, condition for the achievement of long-run economic
success"), Setting Priorities did not include any of these
qualifications. 

2.  The draft report said "ATP evaluations would better assist the
Congress in making budget decisions if the evaluations focused more
on outcomes, which reflect the impacts of the program, than on
outputs, which describe the activities of the program." This
statement was not intended to be a broad conclusion about the plan. 
The report now discusses this topic within the context of the
descriptive information that ATP collects.  As the report notes, this
information does not necessarily provide the Congress with
information about the program's impact and economic success. 

3.  We continue to maintain that "tracking technical milestones" and
"increased industrial collaborations and strategic alliances," when
presented as indicators of long-term economic success, may create
false expectations.  Presenting additional information, such as NIST
provides in its comments, would help avoid creating false
expectations. 

4.  Since we focused our work specifically on the statements about
ATP in Setting Priorities, we reviewed the studies and data
supporting these statements to determine their consistency.  We did
not ignore any information provided that was relevant to this work. 
In addition, we have added information about NIST's evaluation
efforts that does not appear in Setting Priorities. 

5.  We maintain that our analysis and estimates are appropriate and
accurate.  The details are provided in the body of the final report
and in comments 13 and 14.  Moreover, the evidence NIST provided--a
report on a single five-member joint venture--cannot be extrapolated
to the entire U.S.  printed wiring board industry. 

6.  See comment 15. 

7.  We included all of the relevant information in our analysis and
maintain that NIST's statements lack support and are based on a
selective use of data.  See comment 19. 

8.  Comment 23 summarizes our rationale for questioning NIST's
statement about ATP's impact on the formation of joint ventures. 

9.  We have withdrawn the proposed recommendation in light of
additional information about plans to refine the ATP evaluation plan. 

10.  We agree that our draft report did not lay out the full scope of
the ATP evaluation plan.  That was not our intent.  Our work focused
on the information contained in Setting Priorities, which states on
page 15 that "A number of measurable short-term effects are expected
to provide indicators of long-term economic success.  In addition to
straightforward tracking of technical milestones, these indicators
include:  .  .  .  increased industrial collaborations and strategic
alliances; .  .  ."

11.  We have not changed our position.  Our 1993 report did not
evaluate NIST's ATP evaluation strategy but did present a
NIST-provided summary of the ATP evaluation strategy.  In addition,
the current report contains similar language concerning the barriers
and challenges facing NIST in evaluating ATP.  For example, our
report states, "Evaluating ATP poses many challenges.  For example,
ATP research projects are intended not only to be technical successes
but also to have commercial results.  The linkage between technical
work and commercial results may not always be direct and may be
subject to interpretation.  Also, several years can elapse between
the end of technical work and the realization of commercial results."

12.  According to NIST, the statement about ATP's impact on the
printed wiring board industry is based on a study that is limited to
a single five-member joint venture.  We maintain that it is an
overstatement to project the impact of this joint venture to the
entire industry of over 800 manufacturers.  For purposes of
comparison, in the draft report we estimated spending for research
and development (R&D) for the entire printed wiring board industry. 
We still maintain that these estimates are the correct figures to use
for such purposes and that the figures further reinforce our
conclusion. 

13.  Our analysis is based on the industry's overall expenditures for
R&D for the following reason.  As NIST comments, only a portion of
the industry's R&D is focused on advanced technology.  But similarly,
only a portion of the joint venture's R&D is focused on advanced
technology, and the larger balance is devoted to incremental
improvements in existing technology. 

14.  NIST's suggested analysis still does not demonstrate that ATP
has quadrupled total U.S.  R&D work on advanced technologies for
printed wiring boards.  Taking into account the Department of
Energy's contribution of $5.2 million, or $1.04 million annually, the
total annual spending on R&D by the ATP-supported joint venture is
that amount plus the joint venture's original annual expenditure of
$5.7 million, for a total of $6.74 million.  Since less than half of
that total is spent for R&D on advanced technology, $3.37 million, or
a half, is a high estimate of the amount spent annually by the joint
venture on advanced technology R&D.  NIST's claim remains overstated
because the joint venture's annual expenditure of $3.37 million still
does not quadruple the industry's expenditure of $2.65 million per
year for advanced technology research on printed wiring boards. 

15.  Language has been added to the final report reflecting the
information provided by responses to this question.  This
information, however, does not change our analysis or our
conclusions.  Our conclusions are reinforced by a Congressional
Budget Office (CBO) report, which arrives at a similar conclusion
about this evidence.  According to the CBO report, "One privately
funded study of the 11 projects supported by the first [ATP]
competition in 1990 suggests that as many as half of them would
probably have been undertaken even without ATP support, although at a
lower level of funding."\4

16.  Language has been added to the body of the final report
reflecting the information provided by these responses.  This
information does not change our conclusions. 

17.  We included all of the relevant information in our analysis, and
our conclusions remain unchanged.  See comment 19. 

18.  At the exit conference we said we understood NIST's logic and
rationale for making the statement.  We did not say we accepted it as
reasonable. 

19.  As stated in our final report, this is an inconsistent use of
survey data.  NIST's synthesis of the results of two different
questions, one open-ended and one closed-ended, does not adequately
support NIST's statements, nor does the information provided by
another question in the survey (section L.). 

20.  We included all of the relevant information in our analysis, and
our conclusions remain unchanged.  We agree with NIST's comment on
the National Cooperative Research Act and feel that NIST should have
included references such as this in Setting Priorities to avoid
overstating any potential effects of ATP on the formation of joint
ventures.  As we pointed out, the effects of the National Cooperative
Research Act make it difficult to determine the effects of ATP on the
number of joint ventures during ATP's first four competitions. 

21.  Joint R&D ventures exist throughout industry and are not
required to register with the Department of Justice or the Federal
Trade Commission.  Therefore, the joint ventures may have been formed
before applying to ATP and may never have applied to the Department
of Justice or the Federal Trade Commission. 

22.  None of this information shows that ATP caused 125 joint
ventures to form.  As shown in comment 21, registration with the
Department of Justice or the Federal Trade Commission does not mean a
joint venture is new and did not exist before the time of
registration. 

23.  As noted in comments 20, 21, and 22, the current evidence
supporting ATP's impact on the formation of joint ventures is
inconclusive and anecdotal.  Moreover, the NIST statement says that
"approximately 125 joint ventures," i.e., all of the joint ventures
that sent in a proposal to ATP, were formed because of ATP.  We look
forward to the completion of NIST's new survey, which is under way,
for more definitive information on "whether or not the ATP had any
influence on [ATP award recipients'] decision to collaborate."


--------------------
\4 Reducing the Deficit:  Spending and Revenue Options, CBO (Feb. 
1995). 


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II

RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C. 

Bernice Steinhardt, Associate Director
Robin M.  Nazzaro, Assistant Director
Alice G.  Feldesman, Supervisory Social Science Analyst
Andrew J.  Vogelsang, Evaluator-in-Charge
George Warholic, Senior Evaluator
Patton Stephens, Evaluator

OFFICE OF THE CHIEF ECONOMIST

Loren Yager, Assistant Director


*** End of document. ***