Pipeline Safety: Systematic Process Needed to Evaluate Outcomes  
of Research and Development Program (30-JUN-03, GAO-03-746).	 
                                                                 
From 1998 through 2002, a total of 1,770 pipeline accidents	 
occurred, resulting in 100 fatalities and $621 million in	 
property damage. The Office of Pipeline Safety (OPS) within the  
Department of Transportation operates a research and development 
(R&D) program aimed at advancing the most promising technologies 
for ensuring the safe operation of pipelines. In fiscal year	 
2003, OPS received $8.7 million for its R&D program, a sevenfold 
increase since fiscal year 1998. In response to a directive from 
the House Committee on Appropriations, GAO (1) assessed OPS's	 
distribution of funding among various areas of R&D and the	 
alignment of this funding with its mission and goals, (2)	 
surveyed experts to obtain their views on R&D priorities, and (3)
determined how OPS evaluates R&D outcomes.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-746 					        
    ACCNO:   A07104						        
  TITLE:     Pipeline Safety: Systematic Process Needed to Evaluate   
Outcomes of Research and Development Program			 
     DATE:   06/30/2003 
  SUBJECT:   Accident prevention				 
	     Internal controls					 
	     Pipeline operations				 
	     Strategic planning 				 
	     Research and development				 
	     Safety regulation					 
	     Safety standards					 
	     Appropriated funds 				 
	     Funds management					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-746

                                       A

Report to Congressional Subcommittees

June 2003 PIPELINE SAFETY Systematic Process Needed to Evaluate Outcomes
of Research and Development Program

GAO- 03- 746

Letter 1 Results in Brief 2 Background 5 OPS*s R& D Funding Is Aligned
with Its Mission and Pipeline Safety

Goals 10 Experts Generally Support OPS*s R& D Priorities 19 OPS Lacks a
Systematic Process for Evaluating R& D Outcomes 24 Conclusions 31
Recommendations for Executive Action 32 Agency Comments 32

Appendixes

Appendix I: Experts* Views on R& D Priorities and OPS*s R& D Funding, by
Type of R& D 34

Appendix II: Scope and Methodology 38

Appendix III: GAO Contacts and Staff Acknowledgments 42 GAO Contacts 42
Acknowledgments 42

Tables Table 1: Major Categories of R& D Related to Pipeline Safety 20
Table 2: Views of Experts from Three Subgroups on Pipeline Safety

R& D Priorities 24 Figures Figure 1: OPS*s R& D Budget, Fiscal Years 1998-
2003 9

Figure 2: OPS*s Planned Allocation of R& D Funding for Fiscal Year 2003 12
Figure 3: OPS*s R& D Funding by Area of R& D, Fiscal Years 2001- 03 14

Figure 4: Expert Ratings of Categories of Pipeline Safety R& D 21

Abbreviations

DOE Department of Energy DOT Department of Transportation MMS Minerals
Management Service

NIH National Institutes of Health

OPS Office of Pipeline Safety R& D research and development

RSPA Research and Special Programs Administration

This is a work of the U. S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

June 30, 2003 Let er t The Honorable Ernest Istook, Jr. Chairman The
Honorable John Olver Ranking Minority Member Subcommittee on
Transportation, Treasury, and Independent Agencies Committee on
Appropriations House of Representatives

The Honorable Richard C. Shelby Chairman The Honorable Patty Murray
Ranking Minority Member Subcommittee on Transportation, Treasury, and
General Government Committee on Appropriations United States Senate
Pipelines transport nearly all of the natural gas and nearly two- thirds
of the crude oil and refined oil products in the United States. Although
pipelines have a better safety record than other modes of freight
transportation, their cargo is dangerous and leaks or ruptures can have
serious consequences, including fatalities, harm to the environment, and
property damage. For

example, pipeline ruptures in Bellingham, Washington, in 1999 and in
Carlsbad, New Mexico, in 2000 together resulted in a total of 15 deaths
and property and other damages totaling about $46 million. Investigators
have determined that one of the probable causes of the Bellingham accident
was excavation damage and that the cause of the Carlsbad accident was
severe internal corrosion.

The Office of Pipeline Safety (OPS), within the Department of
Transportation*s (DOT) Research and Special Programs Administration, is
responsible for pipeline safety regulation and research. The agency
operates a research and development (R& D) program aimed at enhancing the
safety and reducing the potential environmental impacts of transporting
natural gas and hazardous liquids through pipelines. Specifically, the
program seeks to advance the most promising technological solutions to
problems that impede pipeline safety, such as damage to pipelines from
excavation or corrosion. From fiscal years 2001 through 2003, the budget
of OPS*s R& D program more than tripled, from

$2.8 million to $8.7 million, partly as a result of congressional interest
in achieving technological advances that can improve pipeline safety.

In House Report 107- 722, which accompanied the Department of
Transportation and Related Agencies Appropriations Bill for fiscal year
2003, the House Appropriations Committee raised concerns regarding the
effective management and utilization of these significant increases in
funding for the department*s pipeline safety R& D program. The committee
directed GAO to review the effectiveness of the program. In subsequent
discussions with staff of the Subcommittee on Transportation, Treasury,
and Independent Agencies of the House Appropriations Committee, we agreed
to determine (1) OPS*s distribution of funding among various areas of
pipeline safety R& D since fiscal year 2001 and the extent to which this
funding is aligned with the agency*s mission and pipeline safety goals,
(2) the views of experts on pipeline safety R& D priorities, and (3) how
OPS evaluates the outcomes of the pipeline safety R& D it funds.

To carry out this work, we reviewed legislation and agency documents
pertaining to the R& D program and interviewed agency officials
responsible for this program. We also interviewed key experts and
stakeholders regarding their views on R& D priorities and gaps and on
OPS*s management of its R& D program, including the alignment of the
agency*s research agenda with its mission and goals. We identified best
practices for evaluating the outcomes of R& D through a review of relevant

literature. In addition, we sent a questionnaire to selected experts to
obtain their views on pipeline safety R& D priorities. We selected experts
who are informed about pipeline safety or the development of new pipeline
safety technologies, including representatives of federal and state
agencies, pipeline safety advocacy groups, industry associations, pipeline
companies, technical and consulting organizations, and research
institutes. We received responses from 49 of 55 experts we contacted, for
a response

rate of 89 percent. Our results pertaining to experts* views on R& D
priorities represent the views of only the experts who responded to our
questionnaire and cannot be generalized to a broader population. (See app.
II for additional details on our scope and methodology.)

Results in Brief OPS distributes its R& D budget to three major areas
involving the research and development of pipeline safety technologies as
well as to a fourth

area* efforts to improve the agency*s pipeline mapping and information
systems. For example, in fiscal year 2003, OPS plans to allocate its $8.7
million R& D budget as follows:

 $4.0 million (46 percent) to developing new technologies for preventing
damage to pipelines and detecting leaks,

 $1.9 million (21 percent) to improving technologies for operating,
controlling, and monitoring the condition of pipelines,

 $1.7 million (19 percent) to improving pipeline materials, and  $1.2
million (14 percent) on efforts to improve pipeline mapping data

and data on the safety performance of pipelines. 1 On the basis of our
work, we believe that OPS*s R& D funding is generally aligned with its
mission and pipeline safety goals. The agency has obtained the views of
external experts and stakeholders in determining what types of R& D are
aligned with its mission of ensuring the safe, reliable, and
environmentally sound operation of the nation*s pipeline transportation
system. OPS has also recently improved coordination with other federal
agencies that fund pipeline R& D in order to avoid overlap between their
R& D programs. Both expert review and coordination among agencies are
recognized as best practices that help ensure that federal agencies* R& D
activities are relevant to their missions and do not overlap. OPS has also
described, in various plans, how its R& D efforts can lead to new and
improved technologies that can help achieve its performance goals of
reducing the impacts of pipeline incidents, including fatalities and
injuries, and reducing spills of hazardous material. Key experts and
stakeholders we contacted generally told us that, in their view, the
agency has chosen appropriate R& D areas to fund.

The pipeline safety R& D priorities of the experts who completed our
questionnaire are generally consistent with OPS*s R& D priorities. The
ranking of the major R& D areas based on the responses to our
questionnaire is similar to the relative levels of funding OPS has
assigned to these areas:

 92 percent (45 of 49) of the experts assigned a high priority to the
development of new technologies for preventing damage to pipelines and
detecting leaks,

1 Figures do not add to total due to rounding.

 80 percent (39 of 49) assigned a high priority to improvements in
technologies for operating, controlling, and monitoring the condition of
pipelines, and

 31 percent (15 of 49) assigned a high priority to improvements in
pipeline materials.

However, the experts* level of support for improvements in pipeline
materials was much lower than that for the other two main R& D areas that
OPS is funding and this level of support differed across different groups
of experts. Although 70 percent (7 of 10) of experts from research
organizations indicated that this area should receive high priority, only
21 percent (8 of 39) of the remaining experts* from government, public
interest, industry, and technical and consulting organizations* indicated
that it should receive high priority. OPS officials told us that they are
currently updating their research agenda, using the input of experts and
stakeholders, and that they will consider our questionnaire results in
this process.

Despite the significant growth in its R& D budget since fiscal year 2001,
OPS has not developed a systematic process for evaluating the outcomes of
the R& D it funds. For example, the agency tracks and disseminates
information on the progress of individual R& D projects but has not

developed a process for assessing and reporting on the results of its R& D
program as a whole. Without such a process, OPS cannot determine and
demonstrate the progress of its R& D program in achieving intended
results, such as the development and use of new and improved technologies
that can enhance pipeline safety. The agency has taken some preliminary
steps toward developing an evaluation process for its R& D program, such
as identifying possible measures of program results, and could benefit
from adopting identified best practices for systematically evaluating the
outcomes of federal R& D programs. Leading research organizations, the
Office of Management and Budget, and GAO have identified a number of

such practices, including setting clear R& D goals and measuring progress
toward these goals, using expert review to evaluate the quality of
research outcomes, and reporting periodically on evaluation results. The
results of evaluations can be used to refocus R& D priorities
periodically, as

necessary, to ensure that program resources are most effectively utilized.
The Pipeline Safety Improvement Act of 2002 requires that, starting in
December 2003, DOT, the Department of Energy (DOE), and the National
Institute of Standards and Technology jointly provide annual reports to
Congress on their pipeline R& D efforts but does not fully specify what

types of information should be included in these reports. This requirement
provides an opportunity for OPS to keep Congress informed about the
results of evaluations of its R& D program.

To improve OPS*s ability to demonstrate the effectiveness of its R& D
program and to make the most effective use of program resources, we are
recommending that the agency develop a systematic process for evaluating
program outcomes, using identified best practices, and include the results
of R& D evaluations in the required annual reports to Congress on pipeline
R& D. We provided DOT with a draft of this report for its review and
comment. DOT officials generally agreed with the report*s findings and
conclusions. They emphasized that they have started to develop a

framework for evaluating the effectiveness of their pipeline safety R& D
program and that they intend to follow our recommendations as they move
forward in developing and implementing this framework. Background Three
primary types of pipelines form a 2.2 million- mile network across the

nation.  Natural gas transmission pipelines transport natural gas over
long distances from sources to communities.

 Natural gas distribution pipelines continue to transport natural gas
from transmission lines to consumers.

 Hazardous liquid pipelines transport crude oil to refineries and refined
oil products, such as gasoline, to product terminals.

OPS, within DOT*s Research and Special Programs Administration (RSPA), is
responsible for enhancing the safety of and reducing the potential
environmental impacts of transporting natural gas and hazardous liquids

through pipelines. The agency primarily carries out this responsibility
through regulation, oversight, enforcement, and R& D. OPS sets and
enforces regulations that pipeline operators must follow in designing,
constructing, maintaining, and operating pipelines. State agencies

responsible for overseeing pipeline safety help OPS to enforce its
regulations. 2 In December 2000, it began implementing a new risk- based
regulatory approach, called *integrity management.* Under this approach,
operators are required, in addition to meeting minimum safety standards,
to better protect pipeline segments where a leak or rupture could have
significant consequences, such as near highly populated areas, by
conducting new tests of these segments, completing repairs according to
specified schedules, and developing comprehensive plans for addressing

the range of risks facing these segments. 3 The agency*s R& D program is
aimed at advancing the most promising technologies for ensuring the safe
operations of pipelines. For example, current R& D projects seek to
develop new and improved techniques for assessing the condition of
pipelines and detecting anomalies* such as leaks, corrosion, and damage
from excavators* that can lead to pipeline accidents. From 1998 through
2002, a total of 1, 770 pipeline accidents occurred, resulting in 100
fatalities and $621 million in property damage. 4

2 In general, OPS retains full responsibility for inspecting and enforcing
regulations on interstate pipelines but certifies states to perform these
functions for intrastate pipelines. In 2003, 49 state agencies, the
District of Columbia, and Puerto Rico were certified for inspecting and
enforcing regulations on intrastate pipelines. In addition, OPS has
agreements with 11 states to inspect segments of interstate pipelines
within their boundaries.

3 We have previously reported on OPS*s implementation of this new
regulatory approach. See U. S. General Accounting Office, Pipeline Safety
and Security: Improved Workforce Planning and Communication Needed, GAO-
02- 785 (Washington, D. C.: Aug. 26, 2002), and

Pipeline Safety: The Office of Pipeline Safety Is Changing How It Oversees
the Pipeline Industry, GAO/ RCED- 00- 128 (Washington, D. C.: May 15,
2000). 4 These figures are based on accidents reported to OPS. For
hazardous liquid pipelines, they include accidents involving any fatality
or injury, a fire or explosion, total costs of $50,000 or more, or
releases of 50 or more barrels of hazardous liquids or 5 or more barrels
of highly volatile liquids. For natural gas pipelines, they include
accidents involving any fatality or injury, total costs of $50,000 or
more, or the emergency shutdown of a liquified natural gas facility, as
well as any accidents considered to be significant by the pipeline
operator.

OPS*s R& D program has undergone major changes in the last several years.
In particular, the agency has developed a new agenda for its R& D program,
using the input of key experts and stakeholders, and has received
significant increases in funding for this program.

 Until 2001, most of the research funded by OPS was aimed at helping the
agency perform its regulatory function or was in response to an accident
investigation or congressional direction. In November 2001, the agency
held an R& D planning workshop to gain the perspectives of a variety of
experts and stakeholders on areas of R& D that have the most potential for
enhancing pipeline safety. Attendees included representatives of federal
and state agencies, research organizations, industry groups, pipeline
companies, and technical organizations that set industry safety standards.
OPS used the R& D priorities identified in this workshop to develop a new
agenda for its R& D program, focusing on three main areas: (1) developing
new technologies for preventing damage and detecting leaks, (2) improving
technologies for operating, controlling,

and monitoring the condition of pipelines, and (3) improving pipeline
materials. From March through December 2002, the agency issued
announcements requesting project proposals in these areas, asking that
prospective funding recipients provide at least 50 percent of the proposed
project*s cost. As of May 2003, it had funded 10 R& D proposals it
received in response to these announcements. 5 In addition, after its
November 2001 R& D workshop, OPS established a Web site on its R& D
program in order to improve communications with experts,

stakeholders, and the public about its R& D agenda and activities. 5 In
March 2002, OPS requested proposals related to damage prevention and leak
detection. It received 82 proposals in response and, in November 2002,
funded 7 of them. In June 2002, the agency requested proposals related to
enhanced pipeline operations, controls, and monitoring. It received 57
proposals in response and, in February 2003, funded 3 of them, based on
the availability of funding. OPS intends to fund 3 more of these proposals
in June 2003. OPS has provided approximately 50 percent of the cost of the
projects to awardees.

In December 2002, the agency requested proposals related to improved
performance of pipeline materials and other pipeline safety improvements.
It expects to make funding decisions about these proposals in summer 2003.

 OPS*s budget for its R& D program has risen more than sevenfold since
fiscal year 1998, with the most significant increases occurring since
fiscal year 2001. Figure 1 shows the agency*s budgeted amounts for

R& D from fiscal years 1998 through 2003. 6 OPS*s budget for R& D rose
steadily from fiscal year 1998 to fiscal year 2001, from $1.3 million to
$2.8 million. In fiscal year 2002, the agency received $4.8 million for
its R& D program, which was $2 million more than RSPA had requested for
the program. Agency officials attribute this funding increase to increased
concerns for pipeline safety within Congress following the tragic pipeline
accidents in Bellingham, Washington (1999), and Carlsbad, New Mexico
(2000), which together caused 15 fatalities. For fiscal year 2003, RSPA
requested and received about $4 million in

additional funding for the program, for a total of $8.7 million. OPS
officials told us that this requested increase was a response to
heightened congressional interest in achieving technological solutions to
pipeline safety, as evidenced by legislative proposals that called for
increased attention to this area. 7 RSPA is proposing funding for OPS*s R&
D program of $9.2 million in fiscal year 2004, an increase of about $0.5
million above the fiscal year 2003 amount. OPS officials explained that
they intend to use most of this increase for a study, required by the

Pipeline Safety Improvement Act of 2002, to assess the performance of
controllers who monitor pipeline operations. Overall, agency officials
also attribute recent increases in funding for OPS*s pipeline safety R& D
program to a recognition of the challenges posed by the agency*s new
integrity management regulatory approach and the criticality of the
nation*s pipeline infrastructure, in the aftermath of the terrorist
attacks

of September 11, 2001. 6 These figures have been adjusted to account for
inflation. They are in constant fiscal year 2003 dollars. 7 In addition,
RSPA*s budget submission for fiscal year 2003 noted that the proposed
pipeline safety R& D budget would consolidate into RSPA a pipeline
infrastructure R& D program operated by DOE. However, according to DOE and
OPS officials, no transfer of funding or projects between the two programs
actually took place.

Figure 1: OPS*s R& D Budget, Fiscal Years 1998- 2003

Note: Figures are in constant fiscal year 2003 dollars.

OPS*s pipeline safety R& D program is continuing to evolve in response to
new directives in the Pipeline Safety Improvement Act of 2002 for the
planning and reporting of federal pipeline R& D efforts. The act, which
became law in December 2002, assigned the Secretary of Transportation
responsibility for developing a 5- year plan for pipeline R& D and
transmitting the plan to Congress by December 2003, in coordination with
DOE and the National Institute of Standards and Technology. (OPS officials
told us that the Secretary has delegated this responsibility to OPS.)

DOE operates an R& D program that is focused on developing future
technologies to improve the integrity, reliability, and security of the
natural gas infrastructure, including pipelines and storage facilities. In
comparison with OPS*s R& D program, which focuses on the development of
quick- to- market technologies that could become available in the short
term (1- 3 years) or midterm (3- 5 years), DOE*s program focuses on
technologies that could become available in the midterm (3- 5 years) or
longer term (5- 8 years). The National Institute of Standards and
Technology does not operate an R& D program focused on pipelines, but,

reflecting its expertise in materials research, the act assigns it a key
role in planning future pipeline R& D. The Department of the Interior*s
Minerals Management Service (MMS),

although not assigned an R& D planning role in the act, funds pipeline R&
D, including research on offshore pipeline safety. Consequently, OPS plans
to include that agency in efforts to develop a 5- year plan for pipeline
R& D. The act requires the heads of DOT, DOE, and the National Institute
of

Standards and Technology to jointly report annually to Congress, beginning
in December 2003, on the status and results of implementation of the plan.
OPS*s R& D Funding Is Since fiscal year 2001, OPS has allocated its rising
R& D funding to three Aligned with Its main areas of pipeline safety R& D
that were identified at its 2001 workshop: (1) developing new technologies
for preventing damage to

Mission and Pipeline pipelines and detecting leaks, (2) improving
technologies for operating,

Safety Goals controlling, and monitoring the condition of pipelines, and
(3) improving

the performance of pipeline materials. The agency has also allocated some
R& D funding to a fourth area, efforts to improve the agency*s mapping and
information systems. On the basis of our work, we believe that the
agency*s R& D funding is generally aligned with its mission and pipeline
safety goals. The agency has obtained the views of external experts and
stakeholders in determining what types of R& D are aligned with its
mission of ensuring the safe, reliable, and environmentally sound
operation of the nation*s pipeline transportation system. The agency has
also recently improved coordination with other federal agencies that fund
pipeline R& D in order to

avoid overlap between their R& D programs. Both of these practices have
been recommended by leading organizations that conduct scientific and
engineering research. OPS has also linked its R& D efforts with its
performance goals of reducing the impacts of pipeline incidents, including
fatalities and injuries, and reducing spills of hazardous material. In its
plans, the agency has described how new and improved technologies
resulting from its R& D funding can help achieve these performance goals.
Finally, a number of key experts and stakeholders told us that, in their
view, the agency has chosen appropriate R& D areas to fund.

OPS Allocates Pipeline OPS allocates its R& D budget to three major areas
involving the research

Safety R& D Funding to Four and development of pipeline safety
technologies as well as to a fourth

Major Areas area* efforts to improve the agency*s pipeline mapping and
information

systems* that does not involve such research and development. Figure 2
shows how the agency plans to distribute its fiscal year 2003 R& D budget
of $8.7 million among these areas. 8 OPS plans to spend the largest share
of its R& D budget, 46 percent, or $4.0 million, on the area of Damage
Prevention and Leak Detection, which

includes the development of new technologies to prevent damage to
pipelines, detect pipeline defects, and quickly and accurately locate and
control pipeline leaks. Damage to pipelines from *third parties,* such as
companies performing excavation work, is the leading cause of pipeline
failures and can lead to property damage and injuries or fatalities. 9 OPS
plans to allocate 21 percent of its R& D budget, $1.9 million, to the area

of Enhanced Operations, Controls, and Monitoring, which includes
improvements in technologies for operating, controlling, and monitoring
the integrity of pipelines to help identify and prioritize pipeline safety
problems and solutions.

The agency intends to spend a slightly lesser amount, 19 percent of its R&
D budget, or $1.7 million, on the area of Improved Materials Performance,
which includes improvements in pipeline materials in order to extend the
integrity and lifetime of installed pipelines and their various
components. Finally, the agency plans to allocate the smallest portion of
its R& D budget,

14 percent, or $1.2 million, to the area of Mapping and Information
Systems, which includes efforts to improve the collection, integration,
and analysis of data on the location and safety performance of pipelines.
These efforts make pipeline mapping information available to federal,
state, and

8 These amounts represent OPS*s planned expenditures in each area.
However, the agency*s actual expenditures in an area depend on the
approval of R& D proposals received and may therefore differ from planned
expenditures. Figures do not add to total due to rounding.

9 *Third parties* are people or companies not associated with a pipeline
company or its contractors. Damage to pipelines can result from such
people or companies digging in the vicinity of buried pipelines without
realizing that the pipelines are there. For example, excavating equipment
can accidentally strike a pipeline and cause a leak or rupture, either
immediately or over time, which poses a hazard to life and property.

local officials and support pipeline inspection activities of OPS and its
state partners.

Figure 2: OPS*s Planned Allocation of R& D Funding for Fiscal Year 2003

Note: Shaded areas represent the major pipeline safety R& D areas funded
by OPS. Dollar figures have been rounded.

Since fiscal year 2001, OPS*s allocation of funding to each of the three
main areas of pipeline safety R& D* Damage Prevention and Leak Detection;
Enhanced Operations, Controls, and Monitoring; and Improved Materials
Performance* has risen significantly, while its allocation to Mapping and
Information Systems efforts has remained level. The tripling of the
agency*s R& D budget* from $2.8 million in fiscal year 2001 to $8.7
million in fiscal year 2003* has enabled it to increase funding for these
three R& D areas. Specifically, OPS has increased funding for R& D efforts
in Damage Prevention and Leak Detection from $1. 3 million in fiscal year
2001 to $4.0 million in fiscal year 2003, an increase of over 200 percent.
The agency

has increased funding for Enhanced Operations, Controls, and Monitoring
from $309,000 in fiscal year 2001 to $1.9 million in fiscal year 2003, an
increase of more than 500 percent. OPS started funding Improved Materials
Performance research in fiscal year 2002, increasing funding in this area
to a level of $1.7 million in fiscal year 2003.

Agency officials explained to us that they allocated funding to these
three R& D areas in fiscal years 2002 and 2003 based on the results of
their 2001 R& D planning workshop. 10 For example, they added Improved
Materials

Performance to their R& D agenda because it was identified as a priority
area at the workshop. They have also considered other factors in deciding
how to allocate funding. For example, the agency significantly increased
funding for R& D in the areas of Damage Prevention and Leak Detection and
Enhanced Operations, Controls, and Monitoring because of a great need for
improved performance in these areas. OPS officials explained that, because
the agency*s new risk- based regulatory approach requires pipeline
operators to assess and mitigate risks to pipeline segments where a leak
or rupture could have significant consequences, these operators

need better tools and methods for monitoring pipelines and making
necessary repairs. They also noted that OPS*s R& D results assist in the
creation of industry standards on the appropriate use of new technologies.
In addition, officials explained that they decided to allocate a
significant portion of their R& D budget to the area of Improved Materials
Performance because, on the basis of current information on the
development of pipeline technologies, they believed that advances in this
area held much promise for improving pipeline safety. Finally, OPS has
allocated about $1.2 million per year to the Mapping and

Information Systems area since fiscal year 2001 in order to maintain
efforts to improve these systems. 11 (See fig. 3.)

10 Another area of pipeline R& D* the development of technologies to
support Arctic and offshore pipeline operations* was identified as a main
area of R& D at OPS*s 2001 workshop. However, the agency did not include
this as a main area of funding in its R& D agenda because it was not
identified as a high- priority area at the workshop and because the
Department of the Interior*s MMS funds some R& D in this area. OPS has
recently cofunded

with MMS several projects and a workshop in this area, at a cost of almost
$148,000. 11 Figures have been adjusted to account for inflation. They are
in constant fiscal year 2003 dollars.

Figure 3: OPS*s R& D Funding by Area of R& D, Fiscal Years 2001- 03

Note: Figures are in constant fiscal year 2003 dollars and represent the
agency*s budgeted amounts for each area. In some cases, OPS spends less
than the budgeted amount in a fiscal year. The agency has been allowed up
to 3 years to spend amounts appropriated for R& D.

OPS has provided $3.0 million in funding to 10 projects related to Damage
Prevention and Leak Detection since fiscal year 2001. Examples of funded
projects include the following:

 OPS provided $0.6 million in funding to five projects focused on
improving in- line inspection techniques, including *smart pigs* and other
technologies, for detecting damage and defects in pipe walls. 12 Such
improved techniques can help to prevent pipeline leaks or ruptures

12 Smart pigs are devices that run inside a pipeline to detect anomalies,
such as corrosion, metal loss, or damage from excavation.

by making possible the early detection and repair of damage and defects. 
In partnership with the U. S. Air Force, OPS provided $1.2 million in

funding to a project focused on developing an approach for detecting
pipeline leaks using an airborne laser system that measures levels of
chemicals in the atmosphere just above the earth*s surface. OPS has
provided $0.9 million in funding to six projects related to

Enhanced Pipeline Operations, Controls, and Monitoring since fiscal year
2001. Most of this funding*$ 0.6 million* has been allocated to two
projects to improve alternative inspection techniques, called direct
assessment, for identifying internal and external corrosion and other
defects in pipelines that cannot accommodate smart pigs. 13 This is a
significant issue for natural gas pipelines. One industry association
estimates that only about 35 percent of the total natural gas pipeline
mileage can accommodate smart pigs, which are typically used to assess the
condition of liquid pipelines. OPS officials told us that they are
planning to fund three additional R& D projects in this area in June 2003.
As of May 2003, OPS has provided $0.1 million in funding to one project in

the area of Improved Materials Performance. This project seeks to develop
a *smart* composite pipe that will allow for real- time monitoring of the
condition of the pipe through a remote monitoring system. The agency
requested proposals in this area in December 2002 and expects to start
funding some of these proposals in the summer of 2003. Among the types

of proposals that OPS has requested are proposals to develop  materials
that better withstand third- party damage, corrosion, and cracking;

 higher grade/ strength steels; and  materials that facilitate the
operation of pipelines at higher design

pressures. 13 Direct assessment involves several steps, including digging
holes at intervals along a pipeline to examine suspected problem areas. In
a notice of proposed rulemaking, OPS has proposed integrity management
regulations for gas transmission pipelines that would allow operators to
use direct assessment techniques. See 68 Fed. Reg. 4278, 4318 (Jan. 28,
2003). We have previously reported on the challenges faced by OPS in
ensuring that operators use these techniques appropriately. See GAO- 02-
785.

Finally, of the roughly $1.2 million that OPS has allocated each year
since fiscal year 2001 to the Mapping and Information Systems area, it
spent or plans to spend

 about $800,000 each year for efforts to improve the National Pipeline
Mapping System, which depicts the location of pipelines in relation to
areas that are populated or environmentally sensitive, and

 about $400,000 each year for efforts to integrate information systems
the agency uses in overseeing pipeline safety in cooperation with the
states.

The agency expects to continue funding this area at this level for the
foreseeable future in order to improve and update these systems
continually. OPS officials explained that these mapping and information
systems assist OPS inspectors and state and local officials in their
efforts to oversee pipelines and protect the community and environment
from pipeline leaks or ruptures.

Expert Review and OPS*s mission is to ensure the safe, reliable, and
environmentally sound

Coordination Help OPS operation of the nation*s pipeline transportation
system. It has indicated in Align Its R& D Funding with

its budget and plans that its R& D program supports this broad mission as
Its Mission and Goals

well as the following more specific performance goals: (1) to reduce
deaths, injuries, property damage, and economic disruptions resulting from
pipeline incidents and (2) to reduce the amount of oil and other hazardous
liquids spilled from pipelines. The agency has described how new and
improved technologies resulting from its R& D funding can help achieve
these performance goals. For example, the number of pipeline incidents and
the amount of hazardous material spilled could be reduced through the

use of improved technologies for detecting third- party damage, corrosion,
and defects and the use of improved pipeline materials that can better
withstand damage and corrosion.

The Committee on Science, Engineering, and Public Policy* a joint
committee of the National Academy of Sciences, the National Academy of
Engineering, and the Institute of Medicine* has recommended the use of
expert review to determine whether a research program is focused on the
subjects most relevant to an agency*s mission. 14 Under this form of
review, experts in related fields as well as potential users of the
research evaluate the relevance of research to an agency*s mission and
goals and its potential value to intended users. OPS has used expert
review to help it develop a research agenda that is aligned with its
mission and goals. At its November 2001 R& D planning

workshop, it asked a variety of experts as well as potential users of
research to identify the types of R& D that would be most likely to
enhance pipeline safety. Participants included representatives from
federal and state agencies with pipeline responsibilities, pipeline
companies and their associations, research groups, and technical
organizations that set industry safety standards for pipelines. The agency
subsequently used the results of this workshop in developing its research
agenda, guided by an R& D planning panel composed of key experts from such
groups. OPS has also used peer review, a form of expert review, in
deciding which

R& D proposals to fund, a practice that is recommended by the Committee on
Science, Engineering, and Public Policy. OPS*s review panels have included
representatives from other federal agencies that conduct pipeline R& D,
industry associations, and associations of state agencies with pipeline
safety responsibilities.

The Pipeline Safety Improvement Act, enacted in December 2002, requires
that the Secretary of Transportation consult with a variety of groups in
preparing a 5- year plan for pipeline safety R& D, which must be provided
to Congress by December 2003. In response, OPS is continuing to involve
various experts and stakeholders in its R& D planning. Agency officials
have told us that, in preparation for developing this 5- year plan, they
are in the process of obtaining updated external views in order to
reassess research priorities. This has involved participating in the
pipeline R& D planning efforts of industry associations and research
organizations, discussing R& D priorities with state agency officials, and
reconvening their

14 Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act (Washington, D. C.: National Academy of Sciences, February
1999).

R& D planning panel of outside experts. In developing the plan, agency
officials also plan to consult with OPS*s two technical advisory
committees. Finally, OPS plans to hold another R& D workshop during the
winter of 2003- 04. Coordination among federal agencies that conduct
related research helps

to avoid duplication and ensure that each agency performs research that is
aligned with its particular mission and goals. The Committee on Science,
Engineering, and Public Policy has recommended that agencies establish a

formal process for coordinating similar fields of research, in order to
improve collaboration, help keep important questions from being
overlooked, and avoid duplication of effort. 15 Since 2001, OPS has
increased efforts to coordinate pipeline R& D with DOE and the Department
of the Interior*s MMS, both of which also conduct research related to
pipelines. This increased coordination has taken the form of mutual
participation in panels that review R& D proposals and workshops

to plan R& D activities. According to OPS officials, officials of these
agencies have used these opportunities to communicate about their
respective pipeline R& D efforts and avoid duplication. However, these
agencies have not had a formal mechanism in place that defines each
agency*s responsibilities for pipeline R& D.

The Pipeline Safety Improvement Act requires that the heads of DOT, DOE,
and the National Institute of Standards and Technology develop a
memorandum of understanding to formally coordinate pipeline R& D efforts.
(Although the institute does not operate an R& D program focused on
pipelines, the act assigned it a key role in pipeline R& D based on its
expertise in materials research.) In response, OPS, DOE, and the institute
have developed such a memorandum and are in the process of finalizing it.
16 The Pipeline Safety Improvement Act also requires that DOT coordinate
with DOE and the National Institute of Standards and Technology in
developing a 5- year plan for pipeline R& D. In response, OPS is involving
DOE and the institute, as well as MMS, in efforts to develop

15 Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act, 11. 16 DOT and the Department of the Interior have a
memorandum of understanding in place to coordinate their regulatory
efforts regarding outer continental shelf pipelines; this memorandum
states that that the two departments will coordinate their respective R& D
projects concerning these pipelines. In addition, OPS and MMS have an
interagency agreement to jointly fund R& D projects related to offshore
pipelines.

such a plan. These agencies are also considering holding joint workshops
on pipeline R& D in the future. In addition, OPS and the National
Institute of Standards and Technology have started to participate in each
others* proposal review panels and are discussing entering into an
agreement to have the institute conduct some research on pipeline
materials.

We asked a number of key experts and stakeholders for their views on the
extent to which OPS*s R& D agenda is aligned with its mission and goals.
These individuals included officials in DOE and MMS, representatives of
four industry associations, a former head of a state agency that regulates
gas pipelines, the heads of two leading pipeline R& D organizations, two
foremost technical experts in pipeline safety, and an environmentalist

active in pipeline safety. Six of these individuals have been or are
members of OPS advisory committees or R& D planning or review panels. They
generally told us that, in their view, the agency has chosen to fund
appropriate areas.

Experts Generally The pipeline safety R& D priorities of the experts who
completed our

Support OPS*s R& D questionnaire are generally consistent with OPS*s R& D
priorities. Of the

three main R& D areas that OPS is currently funding, Damage Prevention
Priorities

and Leak Detection received the most scores of high or very high funding
priority; Enhanced Operations, Controls, and Monitoring received the
second highest number of such scores; and Improved Materials Performance
received the third highest number. This ranking corresponds to the
relative levels of funding OPS has assigned to these areas, as described
in the previous section. However, the experts* level of support for
Improved Materials Performance was much lower than that for the other two
main R& D areas that OPS is funding. OPS officials told us that they are
currently updating their research agenda, using the input of experts and
stakeholders, and that they will consider our questionnaire results in
this process. To obtain the views of experts on pipeline safety R& D
priorities, we asked

55 experts to complete a questionnaire indicating the funding priority
they would assign to various types of pipeline safety R& D, using
categories identified as part of OPS*s 2001 R& D planning workshop. Table
1 provides a description of the main categories of R& D we asked experts
to prioritize. The first three categories correspond to the main areas of
R& D that OPS is currently funding. Although the fourth category* Arctic
and Offshore Technologies* was identified as a main area of pipeline R& D
at its workshop, OPS decided not to include it as a main area in its R& D
agenda.

Agency officials told us that they made this decision because R& D related
to Arctic and Offshore Technologies was not considered to be a high
priority by participants at its workshop and because MMS funds some R& D
in this area and is the primary offshore regulator. We did not include
Mapping and Information Systems* an area that OPS is currently funding
from its R& D budget* as a category for the experts to rate because it was
not identified as a main category of R& D at the 2001 workshop. Tabl e 1:
Major Categories of R& D Related to Pipeline Safety

Category of R& D Description

Damage Prevention and Develop new technologies to prevent third- party
damage, Leak Detection detect pipeline defects, and quickly and accurately
locate and control pipeline leaks. Enhanced Operations,

Improve technology for operating, controlling, and Controls, and
Monitoring monitoring the integrity of pipelines to help identify and

prioritize pipeline safety problems and solutions. Improved Materials

Improve pipeline materials to extend the integrity and Performance
lifetime of installed pipelines and their various

components. Arctic and Offshore Develop safer, more cost- effective
materials and Technologies procedures to support Arctic and offshore
pipeline

applications. Sources: Materials from OPS*s 2001 R& D planning workshop
and other OPS documents related to pipeline safety R& D. Figure 4 shows
how the 49 experts who completed our questionnaire rated

the four categories of pipeline safety R& D. We also asked experts to rate
specific types of R& D within each category. (See app. I for how the
experts rated specific types of R& D within these main categories and for
information on the agency*s funding of these specific types of R& D. See
app. II for information on our methodology for selecting experts and
obtaining their views.)

The experts who completed our questionnaire strongly supported the Damage
Prevention and Leak Detection and Enhanced Operations, Controls, and
Monitoring categories of R& D as important areas for OPS to fund. Ninety-
two percent of the experts (45 of 49) indicated that the

Damage Prevention and Leak Detection category should receive high or very
high funding priority. 17 Within this category, experts assigned the most
scores of high or very high funding priority to the following types of R&
D: improvements in the ability of in- line inspection tools, such as
*smart pigs,*

to detect damage and defects (39 of 49), and the development of new
technologies, such as the innovative application of ultrasonics, that can
be used for inspecting pipelines (38 of 49). Several experts we
interviewed highlighted the need to improve methods for detecting damage
to pipelines, citing the fact that third- party damage is the leading
cause of pipeline accidents. According to both liquid and gas pipeline
associations, current inspection tools cannot reliably detect such damage
to pipelines.

Figure 4: Expert Ratings of Categories of Pipeline Safety R& D

Note: Percentages are based on 49 respondents.

17 Experts assigned a funding priority to each category and specific type
of R& D using the following scale: 1= little or no funding, 2= some
funding priority, 3= moderate funding priority, 4= high funding priority,
and 5= very high funding priority. Experts could also indicate that they
did not know or had no basis to judge the funding priority for a
particular R& D category.

Eighty percent of the experts (39 of 49) indicated that the Enhanced
Operations, Controls, and Monitoring category should receive high or very
high funding priority. Within this category, the type of R& D that
received the most scores of high or very high funding priority (37 of 49)
was the improvement of alternative inspection techniques, called direct
assessment, to identify corrosion and other defects in pipelines that
cannot

accommodate in- line inspection devices known as smart pigs. This is a
significant issue for natural gas pipelines because the majority of these
pipelines cannot currently accommodate smart pigs, which are typically
used to assess the condition of liquid pipelines. In contrast to the
experts* views on the importance of these first two

categories, less than one- third of the experts considered the remaining
two categories of R& D, Improved Materials Performance and Arctic and
Offshore Technologies, to be a high priority for OPS to fund. Thirty- one
percent of the experts (15 of 49) assigned scores of high or very high
funding priority to the Improved Materials Performance category, and 20
percent (10 of 49) assigned such scores to the Arctic and Offshore
Technologies category. However, within the category of Improved Materials
Performance, about half (25 of 49) of the experts indicated that the type
of R& D aimed at developing damage- and defect- resistant materials should
receive high or very high funding priority. Such materials could be used
in the replacement of existing pipe or in the installation of new pipe.
One researcher we interviewed noted that such materials are particularly
important for the gas pipeline industry, which is expanding its
infrastructure in response to increased demands for natural gas. One
industry association estimates that the natural gas industry will need to
install about 49,500 miles of transmission pipeline from 2001 through 2015
to meet increased demand in the United States.

Some differences exist in the views of experts from the following three
subgroups: (1) federal and state government and public interest
organizations, (2) pipeline industry and technical and consulting
organizations, and (3) research organizations. 18 As shown in table 2,
experts from all three subgroups generally gave the category of Damage
Prevention and Leak Detection the highest ranking, followed by the

category of Enhanced Operations, Controls, and Monitoring. However,
experts from research organizations considered the categories of Improved
Materials Performance and Arctic and Offshore Technologies to be more
important for OPS to fund than did experts from the other two subgroups.
For example, 70 percent of experts from research organizations (7 of 10)
rated Improved Materials Performance as a high or very high priority
compared with 19 percent of experts from government and public interest
organizations (3 of 16) and 22 percent of experts from pipeline industry
and technical and consulting organizations (5 of 23). In addition, 60
percent of the researchers (6 of 10) rated Arctic and Offshore
Technologies as a high or very high priority for OPS compared with 19
percent of experts from

government and public interest organizations (3 of 16) and only 4 percent
of experts from pipeline industry and technical and consulting
organizations (1 of 23). 18 We also examined results for experts from
those organizations that had bid on OPS R& D

funding in fiscal year 2002 to see how they compared to those of other
experts who completed our questionnaire. Seven of the experts who
completed our questionnaire are from organizations that had bid on OPS R&
D funding within this time frame. Of these, all seven assigned scores of
high or very high funding priority to the Damage Prevention and Leak
Detection category; six assigned such scores to the Enhanced Operations,
Controls, and Monitoring category; three assigned such scores to the
Improved Materials Performance category; and four assigned such scores to
the Arctic and Offshore Technologies category.

Tabl e 2: Views of Experts from Three Subgroups on Pipeline Safety R& D
Priorities Number of experts who assigned a high or very high funding
priority to category

Experts from Experts from

pipeline government

industry and and public

technical and Experts from

interest consulting research

organizations organizations

organizations Category of R& D

(16) (23)

(10)

Damage Prevention and Leak 16 20 9 Detection Enhanced Operations,
Controls,

13 18 8 and Monitoring Improved Materials Performance 3 5 7

Arctic and Offshore Technologies 3 1 6 Source: GAO.

An OPS official told us that he believed that researchers rated the
Improved Materials Performance category more highly than other experts did
because researchers have the best and most current information about the
*state of the art* in technology development and are more aware of
opportunities in this area. A leading expert from a pipeline research
organization noted that the foundation of pipeline R& D has been the
development of defect- resistant steels and that, as a consequence,
researchers in this area are very interested in R& D that will lead to
further improvements in the performance of pipeline materials. He also
explained that researchers may have rated the Arctic and Offshore
Technologies category more highly than the other types of experts who
completed our questionnaire because researchers may be more aware of the
need for such R& D to support the construction of new pipelines in these
areas in order to

reach new energy supplies. OPS Lacks a

Although OPS has received significant increases in funding for its R& D
Systematic Process for

program in recent years, the agency has not developed a systematic process
for evaluating the effectiveness of its R& D program. For example,
Evaluating R& D the agency tracks and disseminates information on the
progress of

Outcomes individual R& D projects but has not developed a process for
assessing and

reporting on the results of its R& D program as a whole. Such a process is
needed to demonstrate the program*s progress toward achieving its

objectives, such as the development and use of new technologies that can
improve pipeline safety. OPS has taken some preliminary steps toward
developing an evaluation process for its R& D program and could benefit
from adopting identified best practices for systematically evaluating the
outcomes of federal R& D programs. Leading research organizations, the
Office of Management and Budget, and GAO have identified a number of

such practices, including setting clear R& D goals and measuring progress
toward these goals, using expert review to evaluate the quality of
research outcomes, and reporting periodically on evaluation results. The
results of evaluations can be used to refocus the direction of R& D
programs

periodically, as necessary, to ensure that resources are most effectively
utilized.

OPS*s Efforts to Evaluate Although OPS has funded R& D to develop pipeline
safety technologies

Research Outcomes Have since the mid- 1990s, the agency*s efforts to
evaluate the outcomes of this R& D have been limited and have focused on
individual projects. 19 OPS*s

Been Limited R& D contracts define project goals and require research
performers to

meet specific milestones for the development of a technology. Contracts
also require research performers to report quarterly and at the end of the
project on results, including milestones achieved and patents applied for
and received. OPS has made some efforts to disseminate the results to date
of individual R& D projects. For example, it has started to put *success
stories* on its Web site that describe achievements in ongoing projects,
such as the development of product prototypes. These success stories help
to communicate the results of individual projects to industry and other
interested parties. At the program level, OPS has not yet established
specific quantifiable

goals for its R& D program or a method for measuring progress toward these
goals. OPS has indicated, in various planning documents, that its R& D
program will help achieve its performance goals of reducing the impacts of
pipeline incidents, including fatalities and injuries, and reducing spills
of hazardous material. However, agency officials have acknowledged that it
is difficult to show the effect of the R& D program on these performance
goals. A more immediate objective of the program, according

19 We have recently reported that RSPA has not fulfilled a DOT requirement
for overseeing and developing ways to improve research evaluation efforts
throughout the department. See U. S. General Accounting Office,
Transportation Research: Action Needed to Improve Coordination and
Evaluation of Research, GAO- 03- 500 (Washington, D. C.: May 1, 2003).

to agency plans, is to promote the transfer of new and improved pipeline
safety technologies to the market in the near term. In deciding which R& D
proposals to fund, OPS gives preference to those

that plan to bring a new product to market within 5 years. In addition,
agency officials told us that OPS plans to promote the use of new
technologies by providing information to potential users and its state
partners about them and, when appropriate, by encouraging their use
through regulation. 20 Agency officials told us that the R& D program aims
to have 80 percent of its projects result in products on the market within
5 years. Such an objective is specific and measurable, but OPS has not
formally established it as a goal in any plan or developed a method for
measuring progress toward achieving it. Furthermore, since the agency has
not yet established specific goals or outcome measures for its R& D
program, it does not have a process for documenting and reporting on the
extent to which this program is achieving its goals. OPS officials
explained that they have not yet developed a process for

evaluating the outcomes of the agency*s R& D program because, prior to
2001, the program*s budget was relatively low and, since restructuring the
program in 2001, they have focused program efforts on building a process
for setting research priorities. However, officials do recognize the need
for evaluating R& D outcomes and have taken some preliminary steps toward
developing an evaluation process for their R& D program. 20 For example,
in a notice of proposed rulemaking, OPS proposed integrity management

regulations for natural gas transmission pipelines that would allow
pipeline operators to assess the integrity (structural soundness) of their
pipelines using a new technique called direct assessment. See 68 Fed. Reg.
4278, 4318 (Jan. 28, 2003). OPS has funded and is currently funding R& D
to develop and validate this assessment method.

OPS is considering some possible measures of the outcomes of its R& D
program as a whole, such as the number of new patents resulting from R& D
efforts. In addition, agency officials told us that, although tracking the
transfer to the market of new pipeline safety technologies can be
challenging, OPS intends to track the use of new technologies in the
future

through its process for inspecting operators* *integrity management*
programs. 21 For example, OPS inspectors could document the use of new or
improved technologies by companies to evaluate the condition of their
pipelines. Agency officials noted that the agency will develop inspection
protocols that require inspectors to collect data on the use of new
technologies after their proposed integrity management rule for natural
gas transmission pipelines is finalized.

OPS is also considering the number of documented R& D *success stories**
summaries of the accomplishments of individual R& D projects* as a
possible measure of program results. However, in previous reviews of R& D
programs operated by other federal agencies, we have found that the
success story approach is selective and does not adequately assess
programwide performance. 22 In early June 2003, OPS presented a potential
set of performance measures for its R& D program to its R& D planning
panel of outside experts in order

to obtain their views on these measures. This panel includes
representatives of DOE, MMS, the National Institute of Standards and
Technology, pipeline industry associations, state agencies with pipeline
responsibilities, and a key pipeline research organization. OPS intends to
refine its set of measures based on comments received from this panel and
to continue obtaining the views of this panel as it moves forward in
developing an evaluation process for its R& D program.

21 OPS has issued requirements for hazardous liquid pipeline operators to
develop such programs, which are aimed at assessing the integrity
(structural soundness) of pipelines and identifying and addressing risks
to segments where a leak or rupture could have significant consequences,
such as near highly populated areas. See 49 CFR S: 195.452. In a notice of
proposed rulemaking, the agency has proposed such requirements for
operators of natural gas transmission pipelines. See 68 Fed. Reg. 4278
(Jan. 28, 2003).

22 U. S. General Accounting Office, Highway Research: DOT*s Actions to
Implement Best Practices for Setting Research Agendas and Evaluating
Outcomes, GAO- 03- 640T (Washington, D. C.: Apr. 10, 2003); Highway
Research: Systematic Selection and Evaluation Processes Needed for
Research Program, GAO- 02- 573 (Washington, D. C.: May 24, 2002); and

DOE*s Success Stories Report, GAO/ RCED- 96- 120R (Washington, D. C.: Apr.
15, 1996).

Finally, OPS officials also told us that the agency intends to obtain the
views of experts on its R& D outcomes as well as on its future R& D
priorities at its next R& D workshop, scheduled for the winter of 2003-
04. However, OPS is in the beginning stages of planning this workshop and
has

not defined a process for using experts* views to evaluate the outcomes of
its R& D program.

OPS officials told us that they are considering including information on
the effectiveness of the agency*s R& D program in the annual reports to
Congress on pipeline R& D that the agency is required to submit, starting
in

December 2003. The Pipeline Safety Improvement Act requires that DOT, DOE,
and the National Institute of Standards and Technology jointly provide
these annual reports to Congress, but does not fully specify what types of
information should be included in these reports.

Best Practices Help Since OPS is in the beginning stages of developing an
evaluation process

Agencies Systematically for its R& D program, it could benefit from
adopting best practices for Evaluate Research

systematically evaluating federal R& D programs. Leading organizations
Outcomes

that conduct scientific and engineering research, the Office of Management
and Budget, and GAO have identified a number of these best practices.
Although the uncertain nature of research outcomes over time can make it
challenging to demonstrate the results of such R& D programs, these
practices are designed to enable agencies to systematically assess and
report on these results regularly in accordance with the Government
Performance and Results Act of 1993. 23 These assessments can be used to
refocus the direction of R& D programs periodically, as necessary, to
ensure that resources are most effectively utilized. Identified best
practices are

discussed in the following sections. 23 The Government Performance and
Results Act requires all federal agencies to measure and report on the
results of their activities annually.

Setting Clear, Quantifiable Goals We have previously reported that, to be
effective, any R& D program must

and Measuring Progress toward be directed toward a clear, measurable goal.
24 Such goals help ensure a

These Goals direct linkage between R& D program efforts and an agency*s
overall

performance goals and mission. Applied research programs, such as OPS*s R&
D program, are directed toward achieving specific useful outcomes, such as
the development of new technologies, which can help accomplish agency
performance goals. The Committee on Science, Engineering, and Public
Policy recommended in a 1999 report that agencies operating

applied research programs measure progress toward practical outcomes and
noted that such measurement can usually be performed annually using
milestones. 25 Similarly, in May 2002 the Office of Management and Budget
established

investment criteria for federal R& D programs that require these programs
to clearly define goals and track progress toward these goals using
appropriate outcome measures and interim milestones. Indicators that have
been used to measure the outcomes of R& D include the achievement of
specific targets for developing new or improved technologies and patent
applications filed and granted. 26 However, measuring research outcomes
can be challenging. For example, outcomes may not occur for a number of
years and may be difficult to track. 24 U. S. General Accounting Office,
Research and Development: Lessons Learned from

Previous Research Could Benefit FreedomCAR Initiative, GAO- 02- 810T
(Washington, D. C.: June 6, 2002).

25 Committee on Science, Engineering, and Public Policy. Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act.

26 See U. S. General Accounting Office, Measuring Performance: Strengths
and Limitations of Performance Indicators, GAO/ RCED- 97- 91 (Washington,
D. C.: Mar. 21, 1997), and

Intellectual Property: Federal Agency Efforts in Transferring and
Reporting New Technology, GAO- 03- 47 (Washington, D. C.: Oct. 31, 2002).
Also, see Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act.

Using Expert Review to Evaluate In its 1999 report and again in 2001, the
Committee on Science,

the Quality of Research Engineering, and Public Policy recommended the use
of expert review,

Outcomes supplemented by quantitative methods, to evaluate research
regularly. 27 Expert review can be a useful addition to performance
measures because

of the value of the reviewers* deep knowledge of the field. Such review
can be performed on a somewhat longer term basis, rather than annually,
and does not require that the final impact of the research be known. Peer
review, a form of expert review, includes an independent assessment of the
technical and scientific merit or quality of research by peers with
essential subject matter expertise and perspective equal to that of the
researchers. In 1999, we reported that some federal agencies, such as the
Department of

Agriculture, the National Institutes of Health (NIH), and DOE, use peer
review to help them evaluate the performance of programs and determine
whether to continue or renew research projects. 28 The Committee on
Science, Engineering, and Public Policy reported in 2001

on the use of expert review, including peer review, by NIH, DOE, the
National Science Foundation, the Department of Defense, and the National
Aeronautics and Space Administration to evaluate the quality of their
research programs. These agencies used varying methods for carrying out
this review, including convening panels of experts who use defined
evaluation processes and obtaining the views of external advisory
committees. The Committee on Science, Engineering, and Public Policy

has also noted that expert evaluation of applied research programs
requires the input of potential users of the results of the research,
since the ultimate usability of these results is an important factor in
determining the worth of the research. Similarly, key experts and
stakeholders we interviewed noted that the degree to which new
technologies are actually used would be a good indication of the
effectiveness of OPS*s R& D program. One industry association
representative we interviewed noted that a *constant theme* raised by
pipeline companies is the need for R& D efforts to produce new
technologies that they can actually use in operating their pipelines.

27 Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act, and

Implementing the Government Performance and Results Act: A Status Report
(Washington, D. C.: National Academy of Sciences, 2001). 28 U. S. General
Accounting Office, Federal Research: Peer Review Practices at Federal
Science Agencies Vary, GAO/ RCED- 99- 99 (Washington, D. C.: Mar. 17,
1999).

Reporting Periodically on Periodic reporting by applied research programs
on results can help keep

Evaluation Results key stakeholders* including oversight organizations and
potential users of new technologies* up- to- date on program
accomplishments. According to

the Committee on Science, Engineering, and Public Policy, applied research
programs can usually report annually on progress in meeting milestones. In
addition, a retrospective analysis over several years is needed to
evaluate outcomes that take more than 1 year to emerge. The committee also
has recommended that agencies demonstrate the value of

their review processes by publicly describing them to oversight groups,
the potential users of research results, and the general public. One
expert we interviewed stressed the importance of periodic public reporting
by OPS on research goals and outcomes and on the method for evaluating
outcomes, in order to disseminate research results and build support for
its R& D program. Conclusions OPS has made significant progress in
establishing a pipeline safety

research agenda that is aligned with its mission and goals and that
incorporates the views of experts and stakeholders. However, without a
systematic process for evaluating the outcomes of its R& D program, the
agency is not able to demonstrate that it is effectively using its
increased resources for R& D to foster new and improved technologies that
can enhance pipeline safety. Identified best practices for evaluating
federal R& D programs* including setting clear quantifiable R& D goals and
measuring progress toward these goals, using expert review to evaluate the
quality of research outcomes, and reporting periodically on evaluation

results* can guide OPS as it moves forward in developing an evaluation
process for its program. By following such practices, the agency can help
ensure that it develops a systematic evaluation process that will enable
it to determine and demonstrate the results of its investment in pipeline
safety R& D. OPS could use such an evaluation process to periodically
refocus the direction of its program in order to make the most effective
use of resources.

Furthermore, although the Pipeline Safety Improvement Act*s requirement
for annual reports on pipeline R& D, starting in December 2003, does not
specify in detail what information should be included in these reports,
this requirement provides an opportunity for the agency to keep Congress
informed about the results of evaluations of its R& D program. In
addition, such reporting, along with other communication methods already
in use by the agency, can keep other interested parties* including the
pipeline industry, state pipeline safety agencies, pipeline safety
advocates, and

researchers* up- to- date on the program*s progress in advancing the most
promising pipeline safety technologies. Recommendations for

To improve OPS*s ability to demonstrate the effectiveness of its R& D
Executive Action

program and make the most effective use of program funds, we recommend
that the Secretary of Transportation direct OPS to

 develop a systematic process for evaluating the outcomes of its R& D
program that incorporates identified best practices and

 include in the annual reports to Congress, which are required by the
Pipeline Safety Improvement Act, information on the results of R& D
evaluations.

Agency Comments We provided DOT with a draft of this report for review and
comment. DOT officials, including OPS*s Director of Program Development,
provided oral

comments on the draft on June 13, 2003. The officials generally agreed
with the report*s findings and conclusions. They emphasized that they are
starting to develop a framework for evaluating the effectiveness of their

pipeline safety R& D program and that they intend to finalize this
framework by December 2003 by documenting it in the 5- year plan and first
annual report on pipeline R& D that DOT is required to submit to Congress,
jointly with DOE and the National Institute of Standards and Technology.
They also noted that they agree with and intend to implement our

recommendations and provided some technical clarifications, which we have
incorporated as appropriate. We are sending copies of this report to the
Secretary of Transportation, the

Administrator of RSPA, RSPA*s Associate Administrator for Pipeline Safety,
the Director of the Office of Management and Budget, and appropriate
congressional committees. We will make copies available to others upon
request. In addition, this report will be available at no charge on the
GAO Web site at http:// www. gao. gov.

If you or your staffs have any questions about this report, please contact
me at (202) 512- 2834 or guerrerop@ gao. gov. Individuals making key
contributions to this report are listed in appendix III.

Peter Guerrero Director, Physical Infrastructure Issues

Appendi xes Experts* Views on R& D Priorities and OPS*s

Appendi I x R& D Funding, by Type of R& D We asked selected experts to
review the following descriptions of specific types of pipeline safety
research and development (R& D) and assign a funding priority to each,
based on its importance in achieving the Office of Pipeline Safety*s (OPS)
mission of ensuring the safe, reliable, and environmentally sound
operation of the nation*s pipeline transportation system. Experts used the
following scale: 1= little or no funding, 2= some funding priority, 3=
moderate funding priority, 4= high funding priority, and 5= very high
funding priority. Experts could also indicate that they did not know or
had no basis to judge the funding priority for a particular type of R& D.
The following table shows, for each type of R& D, the number of experts
who assigned it a high or very high funding priority and OPS*s

current and planned allocation of funding to it. A total of 49 experts
completed our questionnaire.

Number of experts who assigned it a high or very high funding OPS*s
current and planned allocation of funding

Type of R& D priority to this type of R& D a Damage Prevention and Leak
Detection

In- line inspection for damage and defects: 39 Allocated $592,500 to five
projects in November Improve in- line inspection techniques,

2002 for periods of 9 to 24 months. including *smart pigs* and other
technologies, for detecting and measuring damage, cracking, and defects in
pipe walls

Nondestructive evaluation: Develop new 38 Allocated $500,000 to one
project in November approaches or technologies, such as the 2002 for a
period of 24 months. innovative application of ultrasonics, that can be
used for the nondestructive evaluation of

operational pipelines

Real- time monitoring using sensors 27 Allocated $182,000 to one project
in April 2001 for

attached to pipe: Develop and test real- time period of 12 months.
Requested proposals in sensors applied or attached to the pipe that

March 2002 but did not fund any of those received. can detect possible
third- party contact, leaks, Requested additional proposals in December
2002 or other signs of damage

and plans to make funding decisions in July 2003.

Small leak detection: Improve technologies 22 Requested proposals in
December 2002 and plans

for quickly detecting small pipeline leaks to make funding decisions in
July 2003.

Pipe location: Develop better techniques or 17 Allocated $534,521 to two
projects in July 2002 for materials to locate steel and plastic pipelines,

periods of 23 to 24 months. including determining their depth

Encroachment monitoring using satellites: 16 Requested proposals in March
2002 but did not Develop satellite monitoring for encroachment

fund any of those received. Requested additional and ground movement

proposals in December 2002 and plans to make funding decisions in July
2003.

Number of experts who assigned it a high or very high funding OPS*s
current and planned allocation of funding

Type of R& D priority to this type of R& D a Improved directional
drilling: Improve

14 Requested proposals in March 2002 but did not directional drilling
techniques to avoid fund any of those received. Requested additional
accidental damage to other underground

proposals in December 2002 and plans to make utilities funding decisions
in July 2003.

Real- time right- of- way monitoring without

12 Requested proposals in March 2002 but did not

pipe contact: Develop fiber optic lines that can fund any of those
received. Requested additional

be buried above or alongside pipeline to detect proposals in December 2002
and plans to make nearby movement

funding decisions in July 2003.

Airborne chemical mapping: Develop 11 Allocated $600,000 to one project in
April 2001 for a

approaches using aerial surveillance with period of 12 months. Allocated
an additional

optical technologies for right- of- way monitoring $600,000 to this
project in April 2002 for an or other pipeline safety concerns

additional 12 months. Plans to allocate an additional $600,000 to this
project in May 2003 for an additional 12 months. b

Enhanced Operations, Controls, and Monitoring Direct assessment: Improve
alternative 37 Allocated $572,000 to two projects in January 2003
inspection techniques for *unpiggable* for periods of 12 to 26 months.
pipelines to identify internal and external corrosion, third- party
damage, and other pipe defects

External corrosion control: Improve 30 Allocated $297,000 to one project
in January 2003

techniques for characterizing, detecting, and for a period of 26 months.

preventing external corrosion damage

Internal corrosion control: Improve 30 Allocated $275,000 to one project
in January 2003

techniques for characterizing, detecting, and for a period of 12 months.

preventing internal corrosion damage

Stress corrosion cracking detection: 25 Allocated $675,281 to four
projects in May and July Improve techniques for characterizing,

2002 for periods of 12 to 24 months. detecting, and preventing stress
corrosion cracking

Enhanced repair techniques: Develop 25

enhanced repair techniques that can be implemented without shutdown of
pipeline

Risk assessment: Enhance techniques to 25 Allocated $97, 737 to three
projects in May 2002 for integrate and evaluate risk data to define pipe a
period of 12 months. Allocated $70,000 to an susceptibility to various
threats

additional project in January 2003 for a period of 24 months. Requested
additional proposals in December 2002 and plans to make funding decisions
in July 2003. Pipe strength: Improve methods for

17 Requested proposals in December 2002 and plans characterizing remaining
pipe strength to make funding decisions in July 2003.

Human factors: Study human factors, such as 7 Requested proposals in June
2002 but did not fund

operator fatigue, that influence pipeline any of those received. Requested
additional

integrity and develop technologies or proposals in December 2002 and plans
to make procedures to minimize operator error

funding decisions in July 2003.

Number of experts who assigned it a high or very high funding OPS*s
current and planned allocation of funding

Type of R& D priority to this type of R& D a Trenchless pipe installation:
Develop

5 trenchless pipe installation and replacement techniques, including
techniques that use directional drilling or robotics

Improved Materials Performance Damage- and defect- resistant materials: 25
Requested proposals in December 2002 and plans Develop materials that
better withstand third to make funding decisions in July 2003. party
damage, corrosion, and cracking

Pipe coatings: Develop enhanced field- and 15 Requested proposals in
December 2002 and plans

factory- applied coatings, methods for testing to make funding decisions
in July 2003. coatings, and methods to improve coating

choices

Higher grade/ strength steels: Develop higher 14 Requested proposals in
December 2002 and plans

grade/ strength steels, evaluate their to make funding decisions in July
2003. performance, and develop methods for

determining when to use them

Welding and joining: Develop enhanced 13 Requested proposals in December
2002 and plans

welding and joining techniques and improved to make funding decisions in
July 2003. methods for assessing performance of welds

and joints

Higher design pressure: Develop materials 11 Requested proposals in
December 2002 and plans

that facilitate pipelines operating at higher to make funding decisions in
July 2003. design pressures and methods for determining

when to use higher pressure designs

Composite pipe: Develop pipe made of, or 11 Allocated $98,680 to one
project in November 2002

layered with, materials other than steel that for a period of 6 months.
Requested additional may exceed current performance standards or proposals
in December 2002 and plans to make allow greater flexibility or lower cost
in funding decisions in July 2003.

challenging installation conditions

Plastic pipe: Develop new or improved plastic 11 Requested proposals in
December 2002 and plans

pipe for local distribution company systems to make funding decisions in
July 2003. Arctic and Offshore Technologies Leak detection: Develop
approaches to

17 Allocated $7,781 to one project in May 2002 for a detect, verify, and
respond to leaks period of 12 months.

Inspection and maintenance procedures: 8 Allocated $50, 000 to one project
in May 2001 for a Develop alternative inspection and

period of 12 months. maintenance technologies and procedures

Enhanced performance: Develop materials 8 Allocated $59, 955 to one
project in May 2002 for a

and fabrication techniques to enhance low period of 12 months.

temperature performance

Site evaluation: Develop improved techniques 5 for site evaluation

Number of experts who assigned it a high or very high funding OPS*s
current and planned allocation of funding

Type of R& D priority to this type of R& D a Other Pipeline Safety
Improvements Inspection tools: Evaluate and quantify, 28

where possible, the strengths, limits, and performance of current
inspection tools

Pipeline modeling enhancements: Develop 24 Requested proposals in December
2002 and plans

better mathematical or computational modeling to make funding decisions in
July 2003. techniques to improve ability to detect defects,

including growth defects and small leaks

Impact of multiple utilities: Characterize 18 impact of multiple utilities
in common right- ofway on integrity management practices, such as cathodic
protection

Higher stress levels: Evaluate potential for 17

current piping to operate at higher stress levels Reduction of rupture
impact: Explore means 15 to reduce the impact of a pipeline rupture and
explosion, for example, through additives to gas/ liquid or enhanced
shutoff capability

Impact of past releases: Research the impact 7

of past pipeline releases on their surrounding areas and provide
information that could be used to support local zoning decisions Sources:
OPS data; GAO analysis.

Note: The pipeline safety R& D categories were identified as part of OPS*s
2001 R& D Planning Workshop. The descriptions of the categories are based
on materials from this workshop as well as OPS*s 2002 and 2003
announcements soliciting R& D proposals. The information on OPS*s funding
of each category is based on GAO*s analysis of information provided by
OPS. a This column describes OPS*s funding of R& D projects, by category,
in fiscal years 2001 and 2002 and

planned allocation of funding in fiscal year 2003. Some projects that are
applicable to more than one category of R& D appear more than once. b In
conference reports accompanying appropriations bills for fiscal years
2001, 2002, and 2003,

Congress expressed its intent that the Research and Special Programs
Administration devote $600,000 of its pipeline safety R& D budget to this
project in each of these fiscal years.

Appendi I I x Scope and Methodology To perform our work, we reviewed
Office of Pipeline Safety (OPS) documentation on its research and
development (R& D) funding and analyzed this information to identify
trends; reviewed pertinent legislation and agency documents pertaining to
the R& D program; and interviewed OPS officials regarding their R& D
funding, agenda- setting processes, and processes for evaluating the
outcomes of their R& D program. We also interviewed key experts and
stakeholders concerning OPS*s management of its R& D program, including
the alignment of the agency*s research

agenda with its mission and goals, and their views on R& D priorities and
gaps. These individuals included officials of the Department of Energy
(DOE) and the Department of the Interior*s Minerals Management Service
(MMS) who are responsible for pipeline R& D; representatives of pipeline
industry associations and leading pipeline research organizations; and
several key experts in pipeline safety. Also, we identified best practices
for

evaluating the outcomes of federal R& D through a review of relevant
literature and compared the agency*s processes with these practices.

To determine the views of experts on pipeline safety R& D priorities, we
sought to identify experts considered to be very knowledgeable about the
development of new pipeline safety technologies or pipeline safety issues.
To identify appropriate experts, we obtained recommendations on
individuals to contact from key organizations, contacted those
individuals, and obtained further recommendations from them on additional

individuals to contact. We identified initial individuals to contact
through prior work on pipeline safety issues or through recommendations
from OPS. These initial contacts included officials in DOE and MMS,
representatives of four industry associations, a former head of a state
agency that regulates gas pipelines, the heads of two leading pipeline R&
D organizations, two technical experts in pipeline safety, and an
environmentalist active in pipeline safety. Six of these individuals have
been or are members of OPS advisory committees or R& D planning or review
panels. We obtained recommendations from these individuals on experts who
could provide us with views on pipeline safety R& D priorities.

We based our final selection of experts on the criteria of knowledge,
balance, and independence. We considered indications of their extent of
knowledge of pipeline safety R& D, as evidenced by the number of times
they had been recommended, their participation in OPS*s R& D planning and
review activities, or other relevant factors. We included individuals

from a variety of groups in order to achieve a balanced representation of
experts, including some who are relatively independent of OPS and the
pipeline industry. We included individuals from federal and state
agencies,

pipeline safety advocacy groups, industry associations, pipeline
companies, technical and consulting organizations, and research
organizations. We also provided our list of identified experts to the
National Academy of Sciences and OPS for their review and comment.

We contacted 55 individuals whom we had identified as appropriate experts
for our review and asked them to complete a questionnaire indicating their
views on pipeline safety R& D priorities. Forty- nine individuals
responded, for an 89 percent response rate. Our results pertaining to
experts* views on R& D priorities represent the views of only the experts
who responded to our questionnaire. In a number of cases, these
individuals collaborated with others in their organizations in completing
their questionnaires. Listed below are the organizational

affiliations of experts who completed our questionnaire. 1

Government and Public Interest Organizations

Federal Agencies Federal Energy Regulatory Commission Minerals Management
Service, Department of the Interior National Institute of Standards and
Technology, Department of Commerce National Transportation Safety Board
Office of Fossil Energy, Department of Energy State Agencies and
Associations National Association of Pipeline Safety Representatives
National Association of Regulatory Utility Commissioners New York State
Department of Public Service Railroad Commission of Texas Virginia State
Corporation Commission Washington Utilities and Transportation Commission
Pipeline Safety Advocacy Groups Common Ground Alliance Cook Inlet Keeper
Safe Bellingham 1 Two of the individuals who responded are former
officials of these organizations.

Pipeline Industry and Technical/ Consulting Organizations

Industry Associations American Gas Association American Petroleum
Institute Association of Oil Pipelines Interstate Natural Gas Association
of America Offshore Operators Committee Pipeline Companies BP Pipelines,
North America ConocoPhillips CMS Panhandle Companies Duke Energy El Paso
Corporation Enbridge Pipelines Enron Explorer Pipeline Company ExxonMobil
Pipeline Company KeySpan Energy Peoples Energy Shell Pipeline Company

Technical/ Consulting Organizations Accufacts, Inc. Batten and Associates,
Inc. Duckworth Pipeline Integrity Services, Inc. HSB Solomon Kiefner and
Associates, Inc. National Association of Corrosion Engineers

Research Organizations Advantica, Inc. Battelle CFER Technologies Edison
Welding Institute Gas Technology Institute Ohio State University, Fontana
Corrosion Center Pipeline Research Council International, Inc. Southwest
Research Institute

Texas A& M University, Department of Mechanical Engineering University of
Florida, Department of Chemical Engineering In the questionnaire, we asked
respondents to review descriptions of various main categories of pipeline
safety R& D as well as specific types of R& D within these main categories
and indicate what funding priority they would assign to each. 2 (See table
1 for descriptions of the main R& D categories. See app. I for
descriptions of the types of R& D within these main categories.) We based
the R& D categories and descriptions on materials prepared as part of an
R& D planning workshop held by OPS in

2001, in which a variety of experts and stakeholders participated; on
announcements the agency subsequently issued soliciting proposals for R& D
in various areas; and on other OPS documents related to pipeline safety R&
D.

We compiled the scores obtained from the questionnaires to produce a
ranking of R& D priorities representing the views of the experts who
completed our survey. We also analyzed our results to determine whether
any differences existed in the responses of experts from the three
subgroups: government and public interest organizations, industry and
technical and consulting organizations, and research organizations. In
addition, we identified organizations that had bid on R& D funding from
OPS in fiscal year 2002 and conducted a separate analysis of the responses
of experts from these organizations to determine how they compared with
those of other experts who completed our questionnaire. Seven of the
experts who completed our questionnaire are from organizations that had
bid on OPS R& D funding within this time frame. 3 We conducted our work
from January 2003 through June 2003 in

accordance with generally accepted government auditing standards. 2
Experts assigned a funding priority to each category and subcategory of R&
D using the following scale: 1= little or no funding, 2= some funding
priority, 3= moderate funding priority, 4= high funding priority, and 5=
very high funding priority. We also asked respondents if they wished to
identify any additional R& D categories and, if so, what score they would
assign to these categories.

3 Of the 49 experts who completed our questionnaire, we identified 7 from
organizations that had submitted R& D proposals in response to
announcements issued by OPS in March and June 2002, based on information
provided by OPS. Of these 7, 5 were from organizations that received
funding from OPS. In December 2002, OPS issued another announcement
soliciting R& D proposals. However, because OPS has not yet made funding
decisions about proposals received in response to this announcement,
officials preferred not to provide us with information about these
proposals.

Appendi I I I x GAO Contacts and Staff Acknowledgments GAO Contacts Peter
Guerrero, (202) 512- 2834 Susan Fleming, (202) 512- 4431 Acknowledgments
In addition to those named above, Sharon Dyer, Etana Finkler, Judy

Guilliams- Tapia, Brandon Haller, Bert Japikse, Nancy Kingsbury, Donna
Leiss, Gary Stofko, Ron Stouffer, and Stacey Thompson made key
contributions to this report.

(545025)

From 1998 through 2002, a total of 1,770 pipeline accidents occurred, resulting
                             in 100 fatalities and

From 1998 through 2002, a total of 1,770 pipeline accidents occurred,
resulting in 100 fatalities and $621 million in property damage. The
Office of Pipeline Safety (OPS) within the Department of Transportation
operates a research and development (R& D) program

aimed at advancing the most promising technologies for ensuring the safe
operation of pipelines. In fiscal year 2003, OPS received $8. 7 million
for its R& D program, a sevenfold increase since fiscal year 1998. In
response to a directive from the House Committee on Appropriations, GAO
(1) assessed OPS*s distribution of funding among various areas of R& D and
the alignment of this funding with its mission and goals, (2) surveyed
experts to obtain their views on R& D priorities, and (3) determined how
OPS evaluates R& D outcomes. To better determine the effectiveness of its
R& D program, GAO recommends that OPS

develop a systematic process for evaluating program outcomes, using
recognized best practices, and include the results of R& D evaluations in
its annual reports to Congress. OPS officials told us that they generally
agreed with the report*s findings and will follow our recommendations as
they continue to develop an evaluation process for their R& D program.

a

GAO United States General Accounting Office

OPS distributes its R& D budget among four main areas. For example, in
fiscal year 2003, the office plans to allocate its $8.7 million budget as
follows: * 46 percent ($ 4.0 million) to developing new technologies to
prevent

damage to pipelines and prevent leaks;  21 percent ($ 1.9 million) to
improving technologies for operating, controlling, and monitoring the
condition of pipelines;  19 percent ($ 1.7 million) to improved pipeline
materials, such as materials that are resistant to damage and defects; and

 14 percent ($ 1.2 million) to efforts to improve data on the location
and safety performance of pipelines. On the basis of our work, we believe
that OPS*s R& D funding is generally aligned with its mission and pipeline
safety goals. OPS has taken a number of steps to ensure this alignment.
For example, it obtained the views of a variety of experts and
stakeholders in deciding on its R& D priorities and has

described in various plans how its R& D efforts can lead to new and
improved technologies that can help achieve its safety performance goals,
such as reducing the impacts of pipeline accidents. The pipeline safety R&
D priorities of the experts we surveyed are generally consistent with
OPS*s R& D priorities. For example, most assigned a high

priority to the two areas of R& D that receive the highest amount of
funding from OPS.

OPS*s efforts to evaluate the outcomes of its R& D have been limited. The
agency has taken some preliminary steps toward developing an evaluation
process for its R& D program, such as identifying possible measures of
program results. Leading research organizations, the Office of Management
and Budget, and GAO have identified a number of best practices for
systematically evaluating the outcomes of federal R& D programs, such as
setting clear R& D goals, measuring progress toward goals, and reporting
periodically on evaluation results. These best practices can help OPS to
determine the effectiveness of its R& D program in achieving desired
outcomes, such as the development and use of new and improved technologies
that can enhance pipeline safety.

www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 746. To view the full product,
including the scope

and methodology, click on the link above. For more information, contact
Peter Guerrero at (202) 512- 2834 or guerrerop@ gao. gov. Highlights of
GAO- 03- 746, a report to

congressional subcommittees June 2003

PIPELINE SAFETY

Systematic Process Needed to Evaluate Outcomes of Research and Development
Program

Page i GAO- 03- 746 Pipeline Safety R& D

Contents

Contents

Page ii GAO- 03- 746 Pipeline Safety R& D

Page 1 GAO- 03- 746 Pipeline Safety R& D United States General Accounting
Office Washington, D. C. 20548

Page 1 GAO- 03- 746 Pipeline Safety R& D

A

Page 2 GAO- 03- 746 Pipeline Safety R& D

Page 3 GAO- 03- 746 Pipeline Safety R& D

Page 4 GAO- 03- 746 Pipeline Safety R& D

Page 5 GAO- 03- 746 Pipeline Safety R& D

Page 6 GAO- 03- 746 Pipeline Safety R& D

Page 7 GAO- 03- 746 Pipeline Safety R& D

Page 8 GAO- 03- 746 Pipeline Safety R& D

Page 9 GAO- 03- 746 Pipeline Safety R& D

Page 10 GAO- 03- 746 Pipeline Safety R& D

Page 11 GAO- 03- 746 Pipeline Safety R& D

Page 12 GAO- 03- 746 Pipeline Safety R& D

Page 13 GAO- 03- 746 Pipeline Safety R& D

Page 14 GAO- 03- 746 Pipeline Safety R& D

Page 15 GAO- 03- 746 Pipeline Safety R& D

Page 16 GAO- 03- 746 Pipeline Safety R& D

Page 17 GAO- 03- 746 Pipeline Safety R& D

Page 18 GAO- 03- 746 Pipeline Safety R& D

Page 19 GAO- 03- 746 Pipeline Safety R& D

Page 20 GAO- 03- 746 Pipeline Safety R& D

Page 21 GAO- 03- 746 Pipeline Safety R& D

Page 22 GAO- 03- 746 Pipeline Safety R& D

Page 23 GAO- 03- 746 Pipeline Safety R& D

Page 24 GAO- 03- 746 Pipeline Safety R& D

Page 25 GAO- 03- 746 Pipeline Safety R& D

Page 26 GAO- 03- 746 Pipeline Safety R& D

Page 27 GAO- 03- 746 Pipeline Safety R& D

Page 28 GAO- 03- 746 Pipeline Safety R& D

Page 29 GAO- 03- 746 Pipeline Safety R& D

Page 30 GAO- 03- 746 Pipeline Safety R& D

Page 31 GAO- 03- 746 Pipeline Safety R& D

Page 32 GAO- 03- 746 Pipeline Safety R& D

Page 33 GAO- 03- 746 Pipeline Safety R& D

Page 34 GAO- 03- 746 Pipeline Safety R& D

Appendix I

Appendix I Experts* Views on R& D Priorities and OPS*s R& D Funding, by
Type of R& D

Page 35 GAO- 03- 746 Pipeline Safety R& D

Appendix I Experts* Views on R& D Priorities and OPS*s R& D Funding, by
Type of R& D

Page 36 GAO- 03- 746 Pipeline Safety R& D

Appendix I Experts* Views on R& D Priorities and OPS*s R& D Funding, by
Type of R& D

Page 37 GAO- 03- 746 Pipeline Safety R& D

Page 38 GAO- 03- 746 Pipeline Safety R& D

Appendix II

Appendix II Scope and Methodology

Page 39 GAO- 03- 746 Pipeline Safety R& D

Appendix II Scope and Methodology

Page 40 GAO- 03- 746 Pipeline Safety R& D

Appendix II Scope and Methodology

Page 41 GAO- 03- 746 Pipeline Safety R& D

Page 42 GAO- 03- 746 Pipeline Safety R& D

Appendix III

The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO*s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail

this list to you every afternoon, go to www. gao. gov and select
*Subscribe to daily E- mail alert for newly released products* under the
GAO Reports heading.

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to: U. S. General Accounting Office 441 G Street NW, Room LM
Washington, D. C. 20548 To order by Phone: Voice: (202) 512- 6000

TDD: (202) 512- 2537 Fax: (202) 512- 6061

Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470 Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512-
4800

U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548 GAO*s Mission Obtaining Copies of

GAO Reports and Testimony

Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
Programs Public Affairs

United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested

Presorted Standard Postage & Fees Paid

GAO Permit No. GI00
*** End of document. ***