Transportation Research: Actions Needed to Improve Coordination
and Evaluation of Research (01-MAY-03, GAO-03-500).
The Research and Special Programs Administration (RSPA) within
the Department of Transportation (DOT) is responsible for
coordinating and ensuring the evaluation of DOT research programs
to promote the efficient use of departmental research funds,
which in fiscal year 2002 totaled over $1 billion. RSPA is also
responsible for conducting multimodal research that cuts across
different modes of transportation. The House Committee on
Appropriations directed GAO to examine RSPA's coordination and
evaluation of research within DOT and the status of its own
multimodal research.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-500
ACCNO: A06749
TITLE: Transportation Research: Actions Needed to Improve
Coordination and Evaluation of Research
DATE: 05/01/2003
SUBJECT: Federal aid for transportation
Financial management
Internal controls
Program evaluation
Research program management
Transportation research
Funds management
DOT Advanced Vehicle Technologies
Program
DOT Commercial Remote Sensing and
Spatial Information Technologies Program
DOT Infrastructure Assurance Program
DOT Operator Fatigue Management Program
DOT Technology Sharing Program
DOT Technology Transfer Program
DOT University Transportation Centers
Program
Small Business Innovation Research
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-500
A
Report to the Committee on Appropriations, House of Representatives
May 2003 TRANSPORTATION RESEARCH Actions Needed to Improve Coordination
and Evaluation of Research
GAO- 03- 500
Letter 1 Results in Brief 3 Background 5 RSPA Has Met Some, but Not All,
Legislative or DOT Requirements
for Coordinating DOT Research Efforts 6 RSPA Has Met Some, but Not All,
Legislative or DOT Requirements
to Evaluate DOT Research 16 RSPA Has Conducted Multimodal Research in Four
Areas but Does
Not Have a Process to Systematically Evaluate Program Results 19
Conclusions 32 Recommendations for Executive Action 32 Agency Comments and
Our Evaluation 33
Appendixes
Appendix I: Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by RSPA 35
Tables Table 1: Information on and Status of Multimodal Research Programs
Conducted by RSPA from Fiscal Years 1999 to 2003 21 Table 2: Advanced
Vehicle Technologies Program 35 Table 3: Commercial Remote Sensing and
Spatial Information
Technologies Program 38 Table 4: Human- Centered Systems: Operator Fatigue
Management
Program 40 Table 5: Transportation Infrastructure Assurance Program 41
Figures Figure 1: Extent to Which RSPA Meets Selected Legislative and DOT
Requirements for Coordinating DOT Research Efforts 7
Figure 2: RSPA Funding Dedicated to Research and Development Planning and
Management Activities (Fiscal Years 1999- 2003) 12 Figure 3: Extent to
Which RSPA Meets Selected Legislative and
DOT Responsibilities for Evaluating DOT Research Efforts 16 Figure 4:
Electric Vehicle at a Recharging Station 22
Figure 5: Satellite- based Photographic Image of U. S. Interstates 25 and
40 in Albuquerque, New Mexico 24 Figure 6: Airline Pilots Participating in
Fatigue Research 25 Figure 7: Global Positioning Satellite 27 Figure 8:
RSPA*s Multimodal Research Funding (Fiscal Years
1999- 2003) 29
Abbreviations
DOT Department of Transportation FAA Federal Aviation Administration FHWA
Federal Highway Administration FMCSA Federal Motor Carrier Safety
Administration FRA Federal Railroad Administration
FTA Federal Transit Administration GPRA Government Performance and Results
Act of 1993 MARAD Maritime Administration NASA National Aeronautics and
Space Administration RSPA Research and Special Programs Administration
TEA- 21 Transportation Equity Act for the 21st Century TRB Transportation
Research Board TSA Transportation Security Administration USCG United
States Coast Guard
This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.
Letter
May 1, 2003 The Honorable C. W. Bill Young Chairman The Honorable David R.
Obey Ranking Minority Member Committee on Appropriations House of
Representatives
In fiscal year 2002, the Department of Transportation*s (DOT) research and
development budget totaled more than $1 billion. This sum supported the
many individual projects undertaken by the Federal Highway Administration,
the Federal Aviation Administration, and the other operating
administrations that constitute DOT. This research is vital to meeting the
department*s key transportation priorities, such as increasing
transportation safety, enhancing mobility for all Americans, supporting
the nation*s economic growth, and protecting the environment. The Congress
has recognized the importance of coordinating and evaluating research
throughout DOT and established requirements in the Transportation Equity
Act for the 21 st Century to ensure that those tasks are accomplished. In
turn, the department has given responsibility to the Research and Special
Programs Administration*s Office of Innovation, Research, and Education
(hereafter referred to as RSPA) for coordinating, and ensuring the
evaluation of, DOT research programs to promote the efficient use of
research funds. RSPA is additionally responsible for conducting multimodal
research* research that applies to more than one mode of
transportation* for the department that contributes to the safe,
effective, and efficient transportation of people and goods.
In House Report 107- 722, accompanying the DOT and Related Agencies
Appropriations Bill for Fiscal Year 2003, the House Committee on
Appropriations, directed us to examine RSPA*s role in coordinating
research activities and conducting multimodal research throughout the
department. Specifically, in subsequent discussions with Committee staff,
we agreed to address the following questions: (1) To what extent has RSPA
fulfilled requirements for coordinating DOT research efforts? (2) To what
extent has RSPA fulfilled requirements for evaluating research within DOT?
and (3) What types of multimodal research has RSPA conducted since 1999,
and what have been the results? To address questions regarding RSPA*s
efforts to coordinate and evaluate the department*s research efforts, we
examined pertinent legislation, DOT
policy guidance, and DOT performance reports and plans as well as reports
and documents provided by RSPA, including the department*s Research,
Development, and Technology Plan. Although we did not review the
individual research programs and agendas of each DOT modal administration,
we reviewed external assessments conducted by us and the
National Research Council*s Transportation Research Board regarding RSPA*s
role and efforts in coordinating DOT research. 1 Further, we interviewed
RSPA officials, including RSPA*s Associate Administrator for Innovation,
Research, and Education (hereafter Associate Administrator); officials
from the Volpe National Transportation Systems Center in
Cambridge, Massachusetts; and all of the members of the department*s
Research and Technology Coordinating Council (hereafter Coordinating
Council), to discuss research coordination efforts and identify potential
improvements. At the time of our review, the council was made up of 15
members representing the department*s Office of the Assistant Secretary
for Transportation Policy, Office of Intelligence and Security, Office of
the Assistant Secretary for Budget, Office of Intermodalism, United States
Coast Guard, Federal Aviation Administration, Federal Highway
Administration, Federal Railroad Administration, National Highway Traffic
Safety Administration, Federal Transit Administration, Maritime
Administration, RSPA, Bureau of Transportation Statistics, Federal Motor
Carrier Safety Administration, and Transportation Security Administration.
2 In addition, to determine the types and status of multimodal research
that RSPA conducted, we (1) reviewed and analyzed
RSPA budget data from fiscal years 1999 through 2003 and (2) reviewed
RSPA*s multimodal project plans agreements and published project results
for the same period. (One of RSPA*s multimodal research programs* the
Transportation Infrastructure Assurance Program* is also the subject of a
separate GAO review.) 3 1 The Transportation Research Board is a unit of
the National Research Council, a private,
nonprofit institution that is the principal operating agency of the
National Academy of Sciences and the National Academy of Engineering. The
board*s mission is to promote innovation and progress in transportation by
motivating and conducting research, facilitating the dissemination of
information, and encouraging the implementation of research results.
2 The U. S. Coast Guard and the Transportation Security Administration
were transferred to the Department of Homeland Security in March 2003. 3
U. S. General Accounting Office, Transportation Security Research:
Coordination Needed in Selecting and Implementing Infrastructure
Vulnerability Assessments, GAO- 03- 502 (Washington, D. C.: May 1, 2003).
We conducted our review from September 2002 through February 2003 in
accordance with generally accepted government auditing standards.
Results in Brief RSPA has met some, but not all, legislative and DOT
requirements for coordinating departmental research efforts. To meet the
requirements of the Transportation Equity Act for the 21 st Century, RSPA
developed an
annual surface transportation research and development plan and holds
monthly meetings of the department*s Research and Technology Coordinating
Council to coordinate surface transportation research and technology
development activities. The council also provides a venue for discussing
research processes, procedures, and policies as well as a forum for
networking among the department*s researchers to meet DOT requirements for
routinely sharing research information. However, RSPA has not met all of
its legislative and DOT requirements for coordinating
ongoing research efforts to avoid unnecessary duplication of effort
because it does not review all DOT research projects. According to RSPA*s
Associate Administrator, RSPA has not reviewed all DOT research projects
to identify unnecessary duplication because (1) RSPA does not have ready
access to information on all research activities across the department
because efforts to implement a DOT- wide computer- based research tracking
system have stalled and (2) staff and resources dedicated to research
coordination activities have declined. The Associate Administrator said
that he did not believe that unnecessary duplication of research projects
occurred, and that even if such duplication were identified, RSPA*s
limited authority within DOT would hinder efforts to eliminate that
duplication.
RSPA has met some, but not all, legislative and DOT requirements for
evaluating research within DOT. The Secretary of Transportation delegated
responsibility to RSPA for measuring the results of federal surface
transportation research* a legislative requirement* and overseeing and
developing ways to improve research evaluations throughout the department,
which is a DOT requirement. Of these requirements, RSPA partially meets
one: it oversees research evaluation by discussing the issue
at Coordinating Council meetings. According to RSPA officials, they do not
measure the results of surface transportation research throughout DOT
because the operating administrations perform their own evaluations and
RSPA lacks the resources needed to review the individual research
evaluation efforts of each of the operating administrations. Because RSPA
does not oversee specific research evaluation efforts, it cannot ensure
that
evaluations are being conducted or assess the quality of DOT*s operating
administrations* evaluations. Neither RSPA nor DOT has developed a
strategy to address the resource limitations cited by RSPA officials, and
our previous work indicates that more specific oversight is warranted. For
example, we have previously reported that the Federal Highway
Administration does not have an agencywide systematic process to evaluate
whether its research projects are achieving intended results and does not
generally use a peer review approach, 4 consistent with federal research
best practices. 5 Since 1999, RSPA has conducted multimodal research in
the following four
areas: using technology to improve energy efficiency, reduce emissions,
and reduce transportation dependence on petroleum; using satellite images
to improve transportation safety and disaster planning; developing more
effective means to reduce the fatigue of drivers and pilots; and assessing
key transportation systems* vulnerabilities to damage from disasters or
terrorist threats. RSPA budgeted about $37 million to conduct these four
multimodal research programs from fiscal years 1999 through 2003.
According to RSPA*s Associate Administrator, all four programs have
resulted or will result in significant contributions and improvements to
the transportation industry. For example, he said that research aimed at
advanced vehicle technologies has resulted in the testing and development
4 Peer review is a process that includes an independent assessment of the
technical and scientific merit or quality of research by peers with
essential subject area expertise and perspective equal to that of the
researchers.
5 U. S. General Accounting Office, Highway Research: Systematic Selection
and Evaluation Processes Needed for Research Program, GAO- 02- 573
(Washington, D. C.: May 24, 2002).
of components currently used in the production of commercially available
hybrid electrical vehicles. RSPA officials said they use expert or peer
review to assess their multimodal research process and status of their
research programs, an approach that is widely recognized as a research
evaluation best practice. However, we found that RSPA has not established
a systematic approach for using peer or expert review to evaluate the
results of all of its multimodal research programs. For example, RSPA has
no plans to evaluate the results of its Transportation Infrastructure
Assurance Program. As a result, RSPA is limited in its ability to
determine the extent to which these programs are achieving their intended
goals. This report contains recommendations to the Secretary of
Transportation
and the Administrator of RSPA for actions to improve RSPA*s ability to
meet its legislative and DOT requirements pertaining to research
coordination and evaluation. In commenting on this draft, DOT officials
generally agreed with our findings and provided technical comments that we
incorporated, as appropriate. They did not comment on the report*s
recommendations.
Background RSPA has both legislative and departmental responsibilities for
coordinating and evaluating DOT*s research and development programs,
which, in fiscal year 2002, amounted to about $1 billion. The
Transportation
Equity Act for the 21 st Century (TEA- 21) 6 made DOT responsible for
establishing a strategic plan for surface transportation research. The
plan is to include a discussion of efforts to coordinate federal surface
transportation research and technology development activities to avoid
unnecessary duplication of effort. It also is to contain a description of
program evaluations and a schedule for future evaluations of DOT research
projects, among other things. DOT policy delegates the responsibility for
meeting this and other legislative mandates related to research and
development activities to RSPA*s Associate Administrator as chair of the
department*s Coordinating Council. Moreover, DOT policy requires RSPA*s
Associate Administrator, through the Coordinating Council, to coordinate
all research activities by developing processes for sharing information
about research and technology and reviewing the status of all research and
technology projects throughout DOT. 6 Section 5108 codified at 23 U. S. C.
S: 508.
RSPA*s Office of Innovation, Research, and Education had nine full- time
employees and a budget of about $560,000 for fiscal year 2003. In addition
to coordination and evaluation duties, RSPA manages and supports a variety
of other programs for the department, including its Technology Transfer
and Technology Sharing Programs, Small Business Innovation Research
Program, and University Transportation Centers Program. 7 RSPA also
conducts research on multimodal issues that affect the U. S.
transportation system as a whole in contrast to the other operating
administrations within DOT that focus on specific sectors of the U. S.
transportation system. 8 In fiscal year 2003, RSPA conducted and managed
four major multimodal research programs. Participants in RSPA*s multimodal
research programs include stakeholders from the departments*
operating administrations; other federal departments and agencies; state
departments of transportation; private and state universities; private-
sector partners; and various consortia. 9 (See app. I for a listing of
project stakeholders and researchers involved in RSPA*s multimodal
research programs.) RSPA Has Met Some,
Although RSPA has developed an annual plan and taken other steps to but
Not All, Legislative
facilitate research coordination, it has not fully met legislative and DOT
requirements for coordinating departmental research. Figure 1 summarizes
or DOT Requirements
these requirements and the extent to which RSPA has met them. for
Coordinating DOT Research Efforts
7 Through the University Transportation Centers Program, DOT supports 33
university- based research centers to advance transportation research and
education. In fiscal year 2003, RSPA received almost $30 million in
reimbursable funds from the Federal Highway Administration and the Federal
Transit Administration to manage the program.
8 Operating administrations include the Federal Aviation Administration,
Federal Highway Administration, Federal Motor Carrier Safety
Administration, Federal Railroad Administration, Federal Transit
Administration, Maritime Administration, and National Highway Traffic
Safety Administration. 9 A consortium focuses on research and development
of products in a priority area and
includes technical application and demonstration projects carried out in
partnership with industries and service providers.
Figure 1: Extent to Which RSPA Meets Selected Legislative and DOT
Requirements for Coordinating DOT Research Efforts
RSPA Facilitates Research RSPA has met its legislative requirement under
TEA- 21 to develop an Coordination by Developing
annual surface transportation research and technology development plan an
Annual Plan and
to coordinate and document research efforts. The plan covers not only
Conducting Monthly
surface transportation but also air and maritime. Now in its fourth
edition, Meetings
the plan discusses state and local transportation research activities;
describes each operating administration*s mission; and conveys priorities
for the department*s research activities, including identifying examples
of research programs that are necessary to achieve the department*s
strategic goals. According to the Associate Administrator, the plan is an
important resource for the department*s budget and program deployment
processes
and helps to avoid unnecessary duplication of research among the operating
administrations. In March 2000, the National Research Council, acting
through the Transportation Research Board (TRB), assessed DOT*s strategic
planning process, including the first edition of its research plan,
and generally commended RSPA*s efforts in coordinating the department*s
research activities. 10 In September 2001, we reported that the
department*s research plan achieved a number of important strategic
functions, including supporting the department's budget and program
development process, conveying priorities, linking research and
development initiatives occurring throughout the department to specific
strategic goals, and focusing on research and technology needs that
concern the department as
a whole. 11 However, upon reviewing the fiscal year 2003 research plan, we
found that it does not summarize results of the previous year*s surface
transportation research programs. Such a summary is also absent from the
department*s overall performance plan and reports, where it is required by
TEA- 21. 12 Since the explicit intent of the research plan is to focus the
department*s research efforts, and the department is required to gather
summary information on the previous years* research results, this
information could provide continuity and context for observations about
planned research for future years. The research plan is the culmination of
monthly Coordinating Council
meetings in which the Associate Administrator meets with members from each
of DOT*s operating administrations as well as representatives from DOT*s
Office of Policy, Office of Budget, Office of Intelligence and Security,
and Office of Intermodalism. These meetings also serve as a means to
fulfill the legislative requirement to coordinate surface transportation
research and technology development and the DOT requirement to coordinate
research across all modes within DOT. Agendas and minutes from these
meetings indicate that the Coordinating Council*s discussions involve
research processes, procedures, policy, 13 and presentations from council
10 TRB study conducted by the Committee for Review of the National
Transportation Science and Technology Strategy (Washington D. C.: Mar. 28,
2000) 3. 11 U. S. General Accounting Office, Combating Terrorism: Selected
Challenges and Related Recommendations, GAO- 01- 882 (Washington D. C.:
Sept. 20, 2001) 85. 12 According to the Government Performance and Results
Act of 1993, each federal department is required to submit a performance
plan to the Congress annually. The TEA- 21 requirement to include results
of the year*s research in the department*s annual
performance plan is at 23 U. S. C. S: 508 (c) (4) (A). 13 For example, in
October 2002, the Coordinating Council was delegated responsibility for
coordination of all actions related to research misconduct, including
providing guidance in research misconduct policy implementation. The
federal policy on research misconduct was released in December 2000 by the
Executive Office of the President*s Office of Science and Technology
Policy and directs all federal agencies that conduct or support research
to implement this policy. In addition, the council is revising DOT Order
1700.18B dealing with acquisition, publication, and dissemination of DOT
scientific and technical reports.
members and guest speakers. According to the Associate Administrator,
these meetings further provide a forum for networking among DOT
researchers, a venue for DOT operating administrations to learn about each
other*s research, and an opportunity for information sharing and
technology transfer. In addition, in instances where research has
multimodal applicability, the Associate Administrator said RSPA staff and
representatives from the other operating administrations have been active
in sharing information through working groups, such as the Human Factors
Coordinating Committee, which shares information on research conducted by
each of the operating administrations focusing on identifying ways to
better manage human operator fatigue.
RSPA Has Not Reviewed All RSPA does not fully meet its legislative
requirement to coordinate surface DOT Research Projects to
transportation and technology development activities because it does not
Identify Unnecessary
review all surface transportation research projects to determine whether
Duplication
surface transportation researchers within DOT are unnecessarily
duplicating research efforts. A similar DOT requirement broadens this
responsibility to make RSPA responsible for reviewing the status of all
research projects throughout DOT for the same purpose. In its March 2000
report, TRB reported the absence of information on the current status of
all DOT research programs in the department*s annual research plan. 14 We
discovered that this information also remained absent from DOT*s fiscal
year 2003 research plan. The Associate Administrator said that RSPA lacked
(1) readily accessible data on the research activities of other parts of
DOT and (2) the staff and resources to review all research projects across
the department to, at a minimum, identify and report on the extent of
unnecessary duplication, if any, across the department. He also said that
even if RSPA reviewed all of the department*s research activities and
identified any unnecessary duplication of effort, RSPA does not have the
program and budget authority to direct changes in other operating
administrations* research activities.
14 TRB study, 13.
RSPA*s Associate Administrator explained that RSPA does not have readily
accessible data on the research activities of other parts of DOT because
efforts to develop a computer- based tracking system to share DOT research
program and budget information have stalled. RSPA*s 1998 strategic plan
discussed the need to create and deploy such a system to meet its
strategic goal for coordinating research and technology. In fiscal year
1999, RSPA planned to allocate $200, 000 annually for 4 years to develop
and implement a system. According to the Associate Administrator, as of
January 2003, RSPA had spent about $500,000 of its allocation and
completed development of a prototype database. He said that implementing
the centralized information system would (1) make basic project
information (such as project methodologies, funding levels and sources,
schedules, and
planned products) across the department more accessible; (2) provide
greater levels of organization and clarity on historical research; (3)
facilitate strategic planning and coordination; and (4) improve the
department*s annual research plan by providing decision- makers with more
complete, accurate, and timely information on all DOT research activities.
According to the Associate Administrator and Coordinating Council members,
some operating administrations do not support implementation of the system
because they believe that the system would put additional demands on
limited resources and would produce little in terms of tangible results.
Coordinating Council members also said it would duplicate existing
information systems already in place at some operating administrations and
the new system would not be integrated into their
other, modal- unique information systems (such as budget and accounting
information systems). According to the RSPA official in charge of
developing the database, each operating administration would require up to
approximately 2 full- time employees for up to 1 year to input the
historical research project data going back 5 years, and an additional
1/2 to 1 full- time employees per year to manage and update the database.
The Associate Administrator also said that RSPA did not review the status
of all operating administrations* research projects to identify any
unnecessary duplication because his office lacks sufficient staff and
resources to do so. He noted, for example, that RSPA*s total research and
technology budget for fiscal year 1999 was about $3.7 million* of which
$2.2 million was allocated for research and development planning and
management activities 15 *and 13 full- time employees. However, in fiscal
year 2003, this decreased to a total budget of about $2.9 million* of
which $560, 000 was allocated for research and development planning and
management activities* and 9 full- time employees. The Associate
Administrator said that the decline in RSPA*s staff and resources* the
only such staff and resources in the department for conducting long- term
transportation research planning and coordinating research plans and
programs* has also severely limited RSPA*s efforts to coordinate with
transportation research stakeholders outside of DOT, such as state, local,
and other federal agencies. Figure 2 shows RSPA*s funding levels for its
research and development planning and management activities from fiscal
years 1999 through 2003.
15 According to RSPA, research and development planning and management
includes funding for transportation research and development strategic
planning, DOT research facilitation, coordination and partnerships, DOT
technology transfer and technology sharing programs, National Science and
Technology Council activities, Small Business Innovation Research Program
support, and international research and development activities.
Figure 2: RSPA Funding Dedicated to Research and Development Planning and
Management Activities (Fiscal Years 1999- 2003)
RSPA*s Associate Administrator said he believes that little or no
duplication of research activities occurs. He said that, because the
monthly Coordinating Council meetings provided a forum for discussing
ongoing and planned research, unnecessary duplication of research efforts
would be identified. Also, he said that most DOT operating administrations
have discrete research programs and budgets that support their mode-
specific regulatory and safety mandates. For example, the Federal Highway
Administration research focuses on public roads and highways, and its
primary users are state and local transportation departments that seek
better ways to repair the public infrastructure and find improved
materials for pavements. Similarly, the Federal Railroad Administration
focuses on the rail industry*s privately owned infrastructure and these
owners* freight
railroads, Amtrak, commuter railroads, and shippers* look to Federal
Railroad Administration to conduct research that will reduce track
failure, equipment failure, and human error. According to members of the
Coordinating Council, apparent duplication
in research programs might reflect a number of conditions* for example,
research that was intended to validate previous research results, expand
research applications, and address different needs (such as pavement
research for airport runways and highways)* or an effort to explore
alternative approaches before selecting one for further development. None
of the council members, however, could provide us with specific examples
of research projects that reflected these conditions. Other Coordinating
Council members with whom we spoke, however, said that the council should
take further steps to more effectively coordinate DOT research. For
example, one member said the Coordinating Council should review all of the
current projects across the department to improve the level and quality of
the department*s research coordination efforts. Another member said that
the primary functions of the Coordinating Council should be to universally
review DOT research projects to eliminate
unnecessary duplication of effort and to provide opportunities for joint
research partnerships on similar work. Such opportunities might include
research focusing on safety, environmental, training, and human factor
issues. However, Coordinating Council members said that DOT support for
the council and research coordination in general had declined in recent
years (e. g., lower ranking members of the operating administrations
attend the meetings, instead of the more senior personnel that had once
attended, and meetings were shortened from 2 hours to 1 hour per month),
and that
greater departmental support for this effort was warranted. The Associate
Administrator said that RSPA did not have the program and budget authority
over the department*s operating administrations* research activities to
direct changes in research programs, even if opportunities for greater
joint efforts or elimination of unnecessary duplicative research were
found. In a 1996 report examining the status of the department*s
coordination of surface transportation research, we identified RSPA*s lack
of internal clout within the department as an obstacle to its ability to
function effectively as its research strategic planner because it had no
direct program or budget authority over the department*s operating
administrations* research activities. 16 Although DOT proposed the
creation of such an organization to have budgeting and program authority
for DOT research in its fiscal year 1996 budget submission, the Congress
did not approve the agency*s proposal. According to the Associate
Administrator, there are no current legislative or budget initiatives to
pursue this proposal.
16 U. S. General Accounting Office, Surface Transportation: Research
Funding, Federal Role, and Emerging Issues, RCED- 96- 233 (Washington, D.
C.: Sept. 6, 1996).
Although DOT*s earlier effort to overcome RSPA*s lack of internal
authority was not implemented, neither the department nor RSPA has
developed alternative approaches to overcome this obstacle as well as to
address the information and resource limitations that continue to hinder
RSPA*s efforts
to meet legislative and DOT requirements for coordinating departmental
research. Developing a strategy that incorporates information about the
costs involved in reviewing research projects throughout DOT to ensure
that unnecessary duplication does not occur, and that determines whether
finalizing the development and implementation of the DOT- wide research
tracking system database could serve this purpose, is an important first
step for RSPA to meet the legislative and DOT requirements entrusted to
it. A strategy is also critical for communicating to the Congress and the
Secretary of Transportation the challenges RSPA faces, and the specific
actions it can take, in meeting the requirements with the resources it
possesses.
RSPA Has Not Developed RSPA has not established performance measures to
systematically Performance Standards
document the results and benefits of coordinating DOT research activities.
17 In the absence of systematically gathered data on research Against
Which to Measure activities across the department and associated
performance measures, it Its Coordination Efforts
is difficult to determine RSPA*s overall success in coordinating DOT*s
billion- dollar research program. Demonstrated successes could garner
greater departmental support for RSPA*s research coordination efforts. In
its fiscal year 2003 budget submission, RSPA cited the difficulty in
defining and measuring the effectiveness of research coordination
activities. According to RSPA, it is because of this difficulty that it
relies upon
external program assessments to provide independent evaluation of its
research and coordination activities. 18
17 In RSPA*s fiscal year 2003 budget submission to the Congress, it
reported one quantifiable performance measure. This measure is aimed at
gauging RSPA*s progress in administrating the University Transportation
Centers Program and focuses on the number of students graduating with
advanced degrees from universities funded under the program. For fiscal
year 2003, RSPA*s performance goal is 1,228 students.
18 TRB conducted such an assessment in March 2000.
Although we support the use of external assessments, we have reported that
quantifiable measures are necessary to assess agency performance to meet
the intent of the Government Performance and Results Act of 1993 (GPRA).
19 Among the stated purposes of GPRA is the improvement of federal program
effectiveness and public accountability. For agencies to successfully
become high- performing organizations, their leaders need to foster
performance- based cultures, find ways to measure performance, and
use performance information to make decisions. A fundamental element in an
organization*s efforts to manage for results is its ability to set
meaningful goals for performance and, using performance information,
measure performance against those goals. High- performing, results-
oriented organizations establish a set of measures to gauge progress over
various dimensions of performance. In crafting GPRA, the Congress
expressed its interest in American
taxpayers* getting quality results from the programs they pay for as well
as its concern about waste and inefficiency in federal programs. The
fundamental reason for collecting information on a program*s performance
is to take action in managing the program on the basis of that
information. By using performance information to assess the way a program
is conducted, managers can consider alternative approaches and processes
in areas where goals are not being met and enhance the use of program
approaches and processes that are working well. Performance information
also allows program managers to compare their programs* results with goals
and thus determine where to target program resources to improve
performance. When managers are forced to reduce their resources, the same
analysis can help them target the reductions to minimize the impact on
program results.
GPRA*s emphasis on results implies that federal programs contributing to
the same or similar outcomes should be closely coordinated to ensure that
goals are consistent and complementary, and that program efforts are
mutually reinforcing. Thus, measuring the effectiveness of RSPA*s
coordination of DOT research is a critical element of fulfilling its
legislative
and departmental coordination responsibilities* an element RSPA has not
yet addressed.
19 GPRA requires federal agencies to set strategic goals and establish
performance measures for management.
RSPA Has Met Some, RSPA has not fully met all legislative and DOT
requirements for evaluating
but Not All, Legislative research within the department. RSPA does not
meet a legislative
requirement for measuring the results of federal surface transportation or
DOT Requirements
research and partially meets a related DOT policy requirement to oversee
to Evaluate DOT
and develop ways to improve research evaluations throughout the Research
department. Figure 3 summarizes these requirements and the extent to which
RSPA has met them.
Figure 3: Extent to Which RSPA Meets Selected Legislative and DOT
Responsibilities for Evaluating DOT Research Efforts
Although the department has delegated to RSPA the responsibility for
meeting legislative and DOT requirements for evaluating research projects
throughout the department, RSPA*s oversight of DOT research evaluation is
limited to facilitating exchange of information on evaluation approaches.
As delegated by the Secretary of Transportation, RSPA is responsible for
measuring the results of federal surface transportation research
activities and how these results impact the performance of the surface
transportation systems of the United States, as stated in TEA- 21. 20
Also, TEA- 21 calls for a strategic planning process 21 that includes
information on research program evaluations conducted and a schedule of
future evaluations. 22 RSPA has not taken steps to meet these legislative
responsibilities for measuring the results of DOT surface transportation
research, describing research program evaluations, and establishing a
schedule for future evaluations. 23 In terms of DOT policy, RSPA is
responsible for overseeing and developing
more efficient, effective, and participative ways to evaluate and measure
research program effectiveness and progress across all operating
administrations. 24 RSPA has taken steps to develop and communicate more
effective means of evaluation by discussing this issue at monthly
Coordinating Council meetings. For example, in October 2002, the council
provided a forum for discussing four different agency approaches to
research evaluation. In addition, according to RPSA officials, the council
has discussed criteria established by the Office of Management and Budget
for federal investment in research and how these criteria can have an
impact on performance evaluation. RSPA, however, does not oversee
operating administrations* research evaluation efforts and therefore
cannot ensure that steps are being consistently taken to improve
evaluation approaches.
20 23 U. S. C. S: 508 (a) (3) and (b) (3). 21 As defined at 5 U. S. C. S:
306. 22 23 U. S. C. S: 508 (a) (1). 23 We noted that the department*s
performance plan for fiscal year 2003 contained an appendix detailing DOT
program evaluation methods, results, and schedule for future evaluations
for programs that represent significant DOT activities (other than
research) that
contribute to its strategic goals. 24 DOT Order 1120.39.
The Associate Administrator said that RSPA does not measure the results of
federal transportation research activities or provide oversight of the
operating administrations* research program evaluation processes for the
following two reasons: (1) the operating administrations have
responsibility for performing and measuring their own research programs
and (2) the resource constraints that have limited RSPA*s ability to
coordinate DOT- wide research also limit the agency*s ability to oversee
research program evaluations across the department.
Coordinating Council members said that increased oversight of DOT- wide
research programs would be beneficial. Our previous work examining DOT*s
research activities also indicates that such oversight is
warranted. For example, in reviewing the Federal Highway Administration*s
research program, which accounts for almost half of DOT*s research budget,
we found that the Federal Highway Administration lacked a systematic
process for conducting research evaluations, and that
the processes it used were not always consistent with federal research
best practices because it generally did not use a peer review approach. 25
Thus, without oversight, RSPA and the department have no assurance that,
at a
minimum, operating administration research programs are routinely
evaluated or that approaches to evaluations are consistent with
established best practices. DOT and RSPA also have not developed a
strategy to meet the requirement
to measure the results of federal transportation research activities and
how they impact the performance of the surface transportation systems of
the United States. Developing such a strategy that incorporates
information about the costs involved in ensuring that evaluations are
completed and performed according to best practices is an important first
step for DOT and RSPA to meet the requirement. A strategy also is critical
for communicating to the Congress and the Secretary of Transportation (1)
the challenges that RSPA and the department face and (2) the specific
actions
that can be taken to meet this requirement given available resources.
After we raised these issues to senior RSPA officials as a result of our
review, they developed a proposed model for reorganizing the Coordinating
Council to provide an opportunity for RSPA to meet legislative and
departmental requirements to oversee DOT research evaluation. They said
they were still considering the proposal when we completed our review.
25 GAO- 02- 573.
RSPA Has Conducted Since 1999, RSPA has conducted four multimodal research
programs*
Multimodal Research advanced vehicle technologies, commercial remote
sensing and spatial
information, operator fatigue management, and transportation in Four Areas
but Does infrastructure assurance. According to RSPA*s Associate
Administrator, Not Have a Process to these four programs have resulted or
will result in significant contributions
Systematically and improvements to the transportation industry. For
example, he said that
research aimed at advanced vehicle technologies has resulted in the
testing Evaluate Program
and development of components currently used in the production of Results
commercially available hybrid electrical vehicles. Nonetheless, RSPA does
not have a process to systematically evaluate the results of all its
multimodal research programs. In the absence of such a process, RSPA
manages its multimodal research programs by monitoring research
contract agreements and using expert or peer review panels to assess the
quality and relevance of ongoing research. By not systematically
evaluating program results, however, RSPA is limited in its ability to
determine the
extent to which its multimodal research programs are achieving their
intended goals. Status of RSPA*s Multimodal
Since 1999, RSPA has conducted four multimodal research programs, of
Research Programs
which two were congressionally mandated. Specifically, TEA- 21 required
DOT to conduct research on using (1) technology to improve energy
efficiency, and reduce emissions and transportation dependence on
petroleum, and (2) satellite images to improve transportation safety and
disaster planning. Transportation research experts within DOT developed a
third RSPA research program to develop more effective means to increase
the endurance and reduce fatigue of drivers and pilots. Finally, the
catalyst
for research in a fourth area that assesses key transportation system
vulnerabilities to damage from disasters or terrorist threats came from
the National Research Council*s TRB. 26 26 TRB, Improving Surface
Transportation Security, A Research and Development Strategy (1999).
According to the Associate Administrator, RSPA identified and selected
individual projects for these multimodal research programs by obtaining
input from experts within and outside DOT. For example, RSPA, in
conjunction with other DOT operating administrations, published a plan in
June 1999 to guide the selection of human fatique- related projects. In
addition, in April 2000, RSPA issued a strategic multimodal research and
development program plan to help focus advanced vehicle technology
research. Also, in December 2000, RSPA and TRB held a conference on remote
sensing and spatial information research to, among other things, discuss
and define issues and possible research needs with representatives
from academia, transportation agencies, remote sensing businesses,
consulting firms, and other groups. 27 (See app. I for a list of project
stakeholders and researchers involved in RSPA*s multimodal research
programs.) Table 1 provides summary data concerning the scope, funding,
and status of RSPA*s four multimodal research programs.
27 See GAO- 03- 502 for a discussion of the coordination issues involved
in developing and implementing RSPA*s Transportation Infrastructure
Assurance Program.
Table 1: Information on and Status of Multimodal Research Programs
Conducted by RSPA from Fiscal Years 1999 to 2003
Dollars in thousands
Direct and Number of
Multimodal research Reasons for initiating Fiscal
reimbursable projects
Program status as of program programs year
funding initiated January 14, 2003
Advanced Vehicle TEA- 21, section 5111 1999 $9,908 21 In process: 12 of 55
Technologies Program 2000 5,000 15
projects still ongoing* planned program 2001 0 11
completion, end of 2004. 2002 0 8 2003 0 0
Total $14,908 55
Commercial Remote TEA- 21, section 5113 1999 $4,000 0 In process: 14 of 18
Sensing and Spatial
2000 6,000 9 projects still ongoing*
Information Technologies planned program
Program 2001 8,000 6
completion, end of 2003. 2002 0 3 2003 0 0
Total $18,000 18
Human Centered Generated by research
1999 $750 0 In process: 2 of 4 projects Systems: Operator Fatigue
experts within DOT 2000 0 4 still ongoing* planned Management Program
program completion, end of 2001 300 0
2004. 2002 0 0 2003 0 0
Total $1,050 4
Transportation Identified in TRB*s 1999 0 0 In process: all 4 of the
Infrastructure Assurance
Improving Surface
2000 0 0 projects still ongoing*
Program
Transportation Security, A
planned program
Research and
2001 $1,000 2 completion, end of 2003.
Development Strategy 2002 1,000 2 (1999).
2003 1,000 0
Total $3,000 4
Source: GAO presentation of RSPA data.
The Advanced Vehicle Technologies Program was mandated in 1998 under
section 5111 of TEA- 21. This program combines transportation technologies
and innovative program elements to produce new vehicles, components, and
infrastructure for medium- and heavy- duty transportation needs. Since
1999, approximately $15 million has been allocated toward 55 separate
research projects, 43 of which have been
completed, with the goal of improving energy efficiency and U. S.
competitiveness while reducing emissions and transportation dependence on
petroleum. According to the Associate Administrator, the program has
resulted in (1) the testing of components that are being
used in the development and production of commercially available hybrid
electrical vehicles and (2) the development and implementation of electric
vehicle recharging stations in Hawaii. A picture of an electric
vehicle at a recharging station is shown in figure 4. RSPA plans to have
all of the projects completed by the end of 2004.
Figure 4: Electric Vehicle at a Recharging Station
The Commercial Remote Sensing and Spatial Information Technologies
Program was mandated in 1998 under section 5113 of TEA- 21. The joint
program between RSPA and the National Aeronautics and Space Administration
(NASA) began in 1999. This research program focuses in part on using
satellite images to assess transportation hazards and improve disaster
recovery; provide opportunities to monitor and evaluate regional traffic
flow, including the movement of freight; plan for improvements in the
maintenance and security of transportation infrastructures; and aid in
transportation corridor planning. Figure 5 shows an example of satellite-
based photography of interstate highways. According to the Associate
Administrator, RSPA has also supported transportation security technology
project activities
in the aftermath of the terrorist attacks of September 11, 2001. For
instance, according to RSPA officials, the Remote Sensing Program
reoriented two unmanned aerial vehicle projects toward monitoring for
security as well as traditional transportation applications. RSPA has
allocated $18 million to this research program since it began in 1999 and
has disseminated program information; results to date have been
disseminated through Web sites, publications, 28 workshops, and
conferences. Eighteen separate research projects constitute the program; 4
have been completed, and RSPA plans to complete the remaining 14 projects
by the end of 2003.
28 In April 2002, DOT and NASA issued Achievements of the DOT- NASA Joint
Program on Remote Sensing and Spatial Information Technologies:
Application to Multimodal Transportation, which presented 3- year
accomplishments from the program.
Figure 5: Satellite- based Photographic Image of U. S. Interstates 25 and
40 in Albuquerque, New Mexico
The Human Centered Systems: Operator Fatigue Management Program was
conceived by DOT researchers and experts with the primary goal to develop
techniques that transportation operating companies can employ to ensure
endurance and fatigue- free performance of their workforces. Since fiscal
year 1999, approximately $1 million has been allocated to the Human
Centered Systems: Operator Fatigue Management Program. According to the
Associate Administrator, this program has resulted in significant benefit
to the
varied DOT transportation community stakeholders. For example, in January
2003, the program resulted in the production of the Commercial
Transportation Operator Alertness Management Handbook, which describes
measures to better manage driver and pilot fatigue. According to RSPA*s
Associate Administrator, this handbook has been in high demand by the U.
S. Coast Guard, the Maritime Administration, and the Federal Transit
Administration. Two of the four projects being conducted are ongoing. RSPA
plans to complete the program by the end of 2004. In figure 6, airline
pilots participate in NASA research in this area.
Figure 6: Airline Pilots Participating in Fatigue Research
The Transportation Infrastructure Assurance Program 29 consists of
assessments on four separate transportation vulnerabilities. These include
assessing the (1) relationship between the safety and security of the
nation*s transportation infrastructure and some of the nation*s other
critical infrastructures, such as energy, e- commerce, banking and
finance, and telecommunications; (2) transportation and logistical
requirements for emergency response teams in response to terrorist attacks
using biochemical, nuclear, and explosive weapons of mass destruction; (3)
costs, benefits, and practicality of alternative backup systems for the
global positioning system; and (4) trade- offs between
the different modes of transportation and security for hazardous
materials. Since fiscal year 2001, the Congress has appropriated $3
million to conduct these assessments. In total, RSPA plans to publish 11
formal reports on the four vulnerabilities being assessed and develop a
series of presentations and workshops to further disseminate the
information. Figure 7 shows a picture of a global positioning satellite.
29 GAO- 03- 502.
Figure 7: Global Positioning Satellite
From fiscal years 1999 through 2003, RSPA budgeted about $37 million to
conduct these four major multimodal research programs. Of this $37
million, about 9 percent, or $3.3 million, came to RSPA from direct
congressional appropriations. For example, in fiscal year 2001, the
Congress appropriated $1 million for the Transportation Infrastructure
Assurance Program and $300,000 for the Human Centered Systems: Operator
Fatigue Management Program; in fiscal years 2002 and 2003, RSPA received
an additional $1 million for the Transportation Infrastructure Assurance
Program. The remaining $33.7 million for these programs was provided
through reimbursable funding from other DOT administrations. 30 Figure 8
summarizes RSPA*s annual budget for multimodal research from fiscal years
1999 to 2003.
30 RSPA conducts and manages its multimodal research programs, excluding
the Transportation Infrastructure Assurance Program, almost exclusively
using reimbursable funds provided by the DOT administration sponsoring the
research* that is, the Federal Highway Administration, Federal Railroad
Administration, Federal Transit Administration, and National Highway
Traffic Safety Administration.
Figure 8: RSPA*s Multimodal Research Funding (Fiscal Years 1999- 2003)
According to the Associate Administrator, RSPA*s current multimodal
research programs are scheduled for completion by the end of fiscal year
2004. He added that RSPA has made budgetary and legislative proposals to
undertake future multimodal research to, among other things, further
examine applications of unmanned aerial vehicles for commercial remote
sensing or examine infrastructure safety issues in hydrogen energy
systems. He noted, however, that there are no approved plans for future
multimodal research, pending the President*s budget proposal for the
department and the reauthorization of TEA- 21, which might affect RSPA*s
multimodal research roles and responsibilities.
RSPA Oversees Research RSPA oversees its multimodal research programs by
monitoring research
Contracts and Assesses the contract agreements. Specifically, the
Associate Administrator said that
Status of Its Ongoing RSPA assesses project progress against contractual
milestones to ensure
Research but Lacks a that the research is being completed on time and
within cost, while meeting
research objectives. He added that researchers must meet or exceed
Systematic Process for
contractual expectations, or corrective actions are taken. These actions
Evaluating the Results of Its
may include project cancellation. RSPA provided a recent example of the
Multimodal Research
impact of its monitoring efforts that dealt with a project being conducted
under its Commercial Remote Sensing and Spatial Information Technologies
Program. The evaluation focused on an unmanned aerial vehicle application
to real- time traffic flow monitoring, a demonstration project with the
Ohio Department of Transportation and several universities and industry
partners. Due to a technical problem, the scheduled demonstration could
not be conducted. The Remote Sensing Program Manager, working with the
Federal Highway Administration Project Manager assigned to the project,
pulled together a technical team to assess the state of the project. The
technical team became convinced that the contractor could remedy the
technical situation, allowed 3
additional months to make the milestone, and successfully encouraged the
project partners to cover the costs of the delay. According to the
Associate Administrator, these actions resulted in a demonstration to a
state partner
with no cost increase to RSPA. In addition to providing this contractual
oversight, RSPA uses the principles of expert or peer review through the
use of multimodal and multiagency program oversight panels to assess the
status, quality, and relevance of its ongoing multimodal research
programs, according to the Associate Administrator. For example, in
December 2000, TRB and RSPA held a conference on remote sensing and
spatial information research with
representatives from academia, transportation agencies, remote sensing
businesses, consulting firms, and other groups. During the conference,
participants met in breakout sessions to discuss and assess research
progress and interim results. As we have reported, expert or peer review
is
a particularly effective means to evaluate federally funded research. 31
The Committee on Science, Engineering, and Public Policy has also reported
that expert review is widely used to evaluate the quality of current
research as compared with other work being conducted in the field and with
the
relevance of research to the agency's goals and mission. 32 However, RSPA
has not established a process or policies for systematically using peer or
expert reviews to evaluate the results of all its multimodal research. For
example, RSPA has no plans to evaluate the results of its Transportation
Infrastructure Assurance Program.
We, among others, recognize that federal agencies that support research in
science and engineering have been challenged to find the most useful and
effective ways to evaluate the performance and results of the research
programs. For example, since GPRA was passed in 1993, some questions have
been raised about its applicability to the research activities of
government agencies. Because the process required by GPRA is based on a
5- year strategic planning horizon, concerns exist that GPRA constrains,
and perhaps prohibits, the long- term thinking and planning that
characterize the federal role in research. This concern is particularly
relevant for basic research, but even successes from highly applied
research (the type sponsored by DOT) can require 5 to 10 years before
achieving widespread recognition. 33 Nonetheless, as we noted in our
report examining DOT
highway research, 34 without systematic program evaluation, it is unclear
as to whether research efforts are having the intended results. Such a
systematic approach to evaluation, according to best practices used in
other federal research programs, includes review of all ongoing and
completed research on a regular basis and in accordance with GPRA
principles. 31 U. S. General Accounting Office, Federal Research: Peer
Review Practices at Federal Science Agencies Vary, GAO/ RCED- 99- 99
(Washington, D. C.: Mar. 17, 1999) 2.
32 Committee on Science, Engineering, and Public Policy, Evaluating
Federal Research Programs: Research and the Government Performance and
Results Act (Washington, D. C.: February 1999) 39. The Committee on
Science, Engineering, and Public Policy is a joint committee of the
National Academy of Sciences, the National Academy of Engineering, and the
Institute of Medicine.
33 Applied research is original work undertaken to develop new knowledge
with a specific, practical application in mind. 34 GAO- 02- 573.
RSPA*s Associate Administrator acknowledged that a documented process for
systematically evaluating the results of its multimodal research programs
would be beneficial, but that the process should be tailored to
match the type of research and its objectives. He added that RSPA had not
developed and implemented a process for systematically evaluating the
results of its multimodal research because of a lack of funding and
staffing resources. For example, he estimated that an external evaluation
to assess
the results of its multimodal programs could cost as much as $100,000 for
each program. Nonetheless, without establishing and implementing a process
for systematically evaluating the results of its research, RSPA cannot
ensure that its multimodal research programs are achieving their
intended goals. Conclusions To its credit, RSPA has taken steps in recent
years to meet its legislative
and department responsibilities for coordinating and overseeing the
evaluation of the department*s transportation research activities.
Nevertheless, ensuring that no unnecessary duplication of research
programs occurs and that research programs* including the ones that RSPA
conducts* are evaluated for results are critical responsibilities, given
the importance of, and amount of money spent on, DOT research.
Without a strategy to meet legislative and DOT requirements to coordinate
and oversee evaluation of departmental research, RSPA may not be able to
meet these responsibilities, particularly given its lack of authority
within the department and resource limitations. In addition, a lack of
performance standards against which to measure coordination efforts limits
RSPA*s ability to identify areas where coordination is working effectively
and areas
that could be improved upon. With a strategy and performance measures in
place, however, RSPA and DOT should be in a better position to assure the
Congress that the department is making the most of its significant
research dollars through effective coordination and evaluation of its
research programs.
Recommendations for To better meet legislative and DOT requirements for
coordinating and
Executive Action evaluating transportation research within the department,
we recommend the Secretary, in conjunction with RSPA*s Administrator, work
with DOT
operating administrations to:
Develop a strategy for reviewing all DOT research projects to identify
areas of unnecessary research duplication, overlap, and opportunities for
joint efforts. The strategy should address time frames for implementing
this review as well as discuss the extent to which finalizing the
development and implementation of a DOT- wide research tracking system
database could serve to facilitate this process. Once this
strategy has been developed and implemented, the results of this effort
should be incorporated in the department*s annual research plan and
reported to the Congress on an annual basis.
Develop and apply quantifiable performance measures to assess the
effectiveness of research coordination efforts (once a strategy for review
has been developed and implemented), and document the results of these
efforts in the department*s annual research plan. These measures could
include the number of research projects identified as possible candidates
for joint effort or elimination and/ or the associated reduction in the
department*s research spending.
Develop a strategy to ensure that the results of all DOT*s
transportation research activities are evaluated according to established
best practices. This strategy should include estimates of the costs for
ensuring that evaluations are completed. Once the strategy has been
developed and implemented, the results of these efforts should be
incorporated in the
department*s annual research plan and reported to the Congress on an
annual basis.
Include in the department*s annual research plan a summary of all
research program evaluations conducted and a schedule of future
evaluations. In addition, we recommend that the Secretary direct RSPA*s
Administrator to document RSPA*s process for systematically evaluating the
results of its own multimodal research programs, and apply this process to
any future multimodal research programs that RSPA conducts.
Agency Comments and We obtained oral comments on a draft of this report
from RSPA officials,
Our Evaluation including the Associate Administrator for Innovation,
Research, and
Education. These officials generally agreed with the contents of the draft
report but did not comment specifically on the report*s recommendations.
They also provided technical comments that we incorporated as appropriate.
Regarding RSPA*s evaluation of its own multimodal research, the officials
said that RSPA had conducted evaluation activities* peer and expert
reviews of the progress of three of its four multimodal research programs
and had no plans to evaluate the fourth. We acknowledge that RSPA has used
peer and expert review to evaluate the status of at least one of its
ongoing multimodal research programs* commercial remote sensing and
spatial information technology* and we describe this example in this
report. Nevertheless, we continue to believe that our recommendation for
documenting and applying a process for systematically evaluating the
results of any future multimodal research programs conducted by RSPA is
warranted to ensure that such evaluations are consistently conducted in
accordance with established best practices.
We are sending copies of this report to congressional committees and
subcommittees with responsibilities for transportation, the Secretary of
Transportation, the Research and Special Programs Administration
Administrator, and the Director of the Office of Management and Budget.
We will make copies available to others upon request. In addition, the
report will be available at no charge on the GAO Web site at http:// www.
gao. gov.
If you have questions about this report, please contact me at (202) 512-
2834 or siggerudk@ gao. gov. Other key contributors were Colin Fallon,
Christopher Keisling, Bert Japikse, Steve Morris, and Jason Schwartz.
Katherine Siggerud Acting Director, Physical Infrastructure Issues
Appendi xes Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by
Appendi x I
RSPA Table 2: Advanced Vehicle Technologies Program Project Project title
Project performer( s) stakeholders
Model Park Northeast Advanced Vehicle Consortium/ Boston Edison FTA Hybrid
School Bus Northeast Advanced Vehicle Consortium/ Solectria FTA Extended
Hybrid Electric Heavy Duty Vehicle
Northeast Advanced Vehicle Consortium/ MJ Bradley FTA Emission Test
Certification Jet Vapor Deposition for Catalyzing Fuel Cell
Northeast Advanced Vehicle Consortium/ Jet Process FTA
Membranes Corporation AV 900 Cycler for a 600- 900 Volt System for Heavy
Electricore/ Allison Transmission Division of General Motors
FTA Duty Hybrid Electric Vehicles Corporation Installation of Capstone
Microturbines into AVS
Electricore/ Advanced Vehicle Systems, Inc. FTA Passenger Trams Novel
Silicon Carbide JFET- Gated Thyristor Electricore/ Rutgers University FTA
Electric Vehicle Ready State Hawaii Electric Vehicle Demonstration Project
FTA Zero Emission 100- Passenger Electric Tram for
Hawaii Electric Vehicle Demonstration Project/ U. S. FTA, FAA Airports
Electricar Battery Life Cycle Prediction Hawaii Electric Vehicle
Demonstration Project/ Hawaii
FTA Natural Energy Institute/ University of Hawaii/ SOEST Extension of the
Hybrid Electric HMMWV Power
Southern Coalition for Advanced Transportation/ PEI FTA
Train Development Program Electronics Optimization of Hybrid Electric
Vehicles Mid Atlantic Regional Consortium for Advanced FTA Compression
Ignition Auxiliary Unit Control Strategy Vehicles/ Navistar International
Transportation Corporation for Emissions Reduction and Improved Fuel
Economy
Integrated Simulation and Testing System for Mid Atlantic Regional
Consortium for Advanced FTA
Electric Vehicle Batteries Vehicles/ Pennsylvania Transportation Institute
Smaller Better Inverters with Polymer Multi- Layer Mid Atlantic Regional
Consortium for Advanced FTA Capacitors Vehicles/ Sigma Technologies
International, Inc. NiMH Battery System Development for an Electric
Sacramento Municipal Utility District/ Ovonic Battery FTA Vehicle Bus
Advanced PLI Hybrid Electric Vehicle Battery Sacramento Municipal Utility
District/ Compact Power FTA
Reliable Electric Propulsion System for Medium and CALSTART/ Santa Barbara
Electric Bus Works FTA
Heavy Duty Vehicles All- Purpose Electric Tractor CALSTART/ ISE Research
Corporation FTA, FAA Development of Advanced Electrochemical
CALSTART/ University of California, Davis FTA Capacitors Using Carbon and
Lead- Oxide Electrodes for Hybrid Vehicle Applications
Hybrid Transit Bus with Flywheel CALSTART/ Trinity Flywheel Power FTA
Auxiliary Power Unit Project Using Fuel Cell CALSTART/ Freightliner
Corporation FTA Technology
(Continued From Previous Page)
Project Project title Project performer( s) stakeholders
Caterpillar Heavy Duty Powertrain Applicable to Northeast Advanced Vehicle
Consortium FTA Heavy Duty Machines Design/ Integrate/ Test Auxiliary Power
Units/ Hybrid Northeast Advanced Vehicle Consortium FTA
Electric Vehicles for Deployment in Commercial Delivery Fleet
Low Cost, Modular, Highly Reliable, Universal Electricore FTA
Propulsion Inverter for Heavy- Duty Commercial and Military Hybrid
Vehicles
Design and Fabrication of 4H- SiC Hybrid JBS Electricore/ Rutgers
University FTA
Diode for High Temperature and High Efficiency Inverters for Medium and
Heavy Duty Applications
Electric Vehicle Ready State (Phase II) Hawaii Electric Vehicle
Demonstration Project FTA Hybrid Drive System for Heavy Duty Transit Buses
Hawaii Electric Vehicle Demonstration Project/ Hawaii FTA
and Trucks (Phase I) Technology Development Corporation Georgia 1 -
Design, Manufacture and Test a Low Southern Coalition for Advanced
Transportation FTA Speed Industrial Motor System in Heavy- Duty Vehicles
Texas 1 - Enhanced Safety and Risk Reduction for Southern Coalition for
Advanced Transportation FTA
University of Texas Demonstration Program Simulation and Field Test Hybrid
Ultra- Capacitor Mid Atlantic Regional Consortium for Advanced FTA Battery
Energy Storage System for Electric Transit Vehicles/ Pennsylvania
Transportation Institute Vehicles
Development of Scalable Direct- Methanol Fuel Cell Mid Atlantic Regional
Consortium for Advanced FTA
Stacks Vehicles/ GATE Center for Advanced Energy Storage Zebra Battery
Demonstration in an Electric School Sacramento Municipal Utility District/
Santa Barbara Electric
FTA Bus Bus Works Fuel Cell Auxiliary Power Unit Demonstration in a
Sacramento Municipal Utility District/ ISE Research FTA
Heavy- Duty Truck Corporation DOT Center for Climate Change and
Environmental CALSTART/ WestStart Corporation All modes Forecasting
Conference Support Hybrid Electric Prototype Truck, Phase II Program
CALSTART/ ISE Research Corporation FTA 150 kW Traction Drive/ Hybrid
Auxiliary Power Unit
CALSTART/ Unique Mobility, Inc. FTA System for Large Electric or Hybrid
Electric Vehicle Applications
Hickam Air Force Base Project Development Hawaii Electric Vehicle
Demonstration Project FTA, Department of the Air Force Rapid Chargers
Hawaii Electric Vehicle Demonstration Project FTA, Department of the Air
Force Electric Bus Conversion Hawaii Electric Vehicle Demonstration
Project FTA, Department of the Air Force
U. S. S. Arizona Memorial Tour Boat Study Hawaii Electric Vehicle
Demonstration Project FTA, Department of the Navy, National Park Service
(Continued From Previous Page)
Project Project title Project performer( s) stakeholders
Advanced Vehicles for Great Smoky Mountains Mid Atlantic Regional
Consortium for Advanced FTA, National Park National Park Vehicles/
Electric Power Research Institute Service
Plug- In Hybrid Vehicles Energy Control System Mid Atlantic Regional
Consortium for Advanced Vehicles FTA Phase I Fast Track Fuel Cell Bus
Project Sacramento Municipal Utility District/ Sacramento Electric
FTA Transportation Consortium/ CALSTART/ WestStart Development of NiMH
Battery System for
Sacramento Municipal Utility District/ Sacramento Electric FTA
Application in Heavy Duty Hybrid Electric Vehicles Transportation
Consortium Lightweight Hybrid Electric Transit Bus Program CALSTART/ NOVA
Bus Incorporated FTA Airport Clean Fuel Vehicle Outreach -- Targeted
CALSTART/ WestStart Corporation FTA, FAA Project Development Phase I, Fast
Track Fuel Cell Bus Project CALSTART/ WestStart/ Sacramento Electric
Transportation
FTA Consortium National Conference on Climate Change Northeast Advanced
Vehicle Consortium All modes
Development and Fabrication of a PEM Fuel Cell Northeast Advanced Vehicle
Consortium/ University
FTA Power Plant for Heavy Duty Vehicle Applications Transportation Centers
Fuel Cells Drive Line Development Team and Industry Work Northeast
Advanced Vehicle Consortium FTA
Group Route- Ready Fuel Cell Component Testing Northeast Advanced Vehicle
Consortium/ Concurrent FTA Technologies Corporation Hybrid Electric Bus
(Phase II) Hawaii Electric Vehicle Demonstration Project FTA, Department
of the Air Force
Multi- Vehicle Charging System Hawaii Electric Vehicle Demonstration
Project FTA, Department of the Air Force Data Acquisition Systems Hawaii
Electric Vehicle Demonstration Project FTA, Department of the Air Force
Aircraft Loader Hawaii Electric Vehicle Demonstration Project FTA,
Department of the Air Force Source: RSPA.
Legend FAA Federal Aviation Administration FTA Federal Transit
Administration
Note: This program is performed in partnership with seven major consortia
consisting of (1) Sacramento Electric Transportation Consortium; (2)
Northeast Advanced Vehicle Consortium; (3) CALSTART- WESTSTART; (4)
Electricore, Inc.; (5) Mid- Atlantic Regional Consortium for Advanced
Vehicles; (6) Hawaii Electric Vehicle Demonstration Project; and (7)
Southern Coalition for Advanced Transportation.
Table 3: Commercial Remote Sensing and Spatial Information Technologies
Program Project Project title Project performer( s) stakeholders
Airborne Sensor Fusion: A Fast- Track Approach to NCRST- E/ Earthdata
Int'l of NC FHWA
National Environmental Policy Act Streamlining and Environmental
Assessment
Remote Sensing of Environmental Parameters for NCRST- E/ ICF Consulting
FHWA
Use in National Environmental Policy Act Documentation in Support of
Highway Corridor Studies
Remote Sensing Applications in Transit NCRST- F/ Bridgewater State College
FHWA, FTA Airborne Ground- Penetrating Radar to Support NCRST- H/ Aeris
Inc. RSPA/ Office of Monitoring of Pipeline Safety and Performance
Pipeline Safety
Environmental Impact and Risk Modeling of NCRST- H/ EarthWatch, Inc. RSPA/
Office of Petroleum and Gas Transmission Lines Using Pipeline Safety
Interferometry and High Resolution Imagery from Satellite and Airborne-
based Remote Sensing Systems
Facilitating the Operation Efficiency and Growth of NCRST- I/ ASL
Consulting Engineers (Tetra Tech Corp.) Office of the
Intermodal Freight Traffic: Application of Remote Secretary of
Sensing Technology to the Alameda Corridor, Los Transportation/
Angeles, CA Office of
Intermodalism Remote Sensing Applications Supporting Regional
NCRST- F/ Veridian System Division (formerly Veridian FHWA, FTA Database
for Transportation Planning ERIM) Road Network Planning Tool NCRST- F/
Technology Service Corp. FHWA
Impact of Instant Imagery Access on a Regional NCRST- I/ Orbimage FHWA
Database for Transportation Planning Development of Regional Databases for
NCRST- E/ Veridian Systems Division FHWA, FTA Transportation Planning
Remote Sensing for Airport Development and NCRST- F/ Grafton Technologies,
Inc. FAA, FHWA
Transportation Planning Remote Sensing of Invasive Aquatic Plant NCRST- F/
TerraMetrics, Inc. MARAD Obstruction in Navigable Waterways Using an
Unmanned Airborne Data Acquisition NCRST- F/ GeoData Systems, Inc. FHWA
System (ADAS) for Traffic Surveillance, Monitoring, and Management
The Application of Remote Sensing NCRST- H/ ImageCat, Inc. All modes
Technologies in Post- Disaster Damage Assessment
Long- Term Monitoring of Changes in Transportation NCRST- I/ University of
Massachusetts FHWA
and Land Use Associated with the Central Artery/ Third Harbor Tunnel in
Boston, MA.
(Continued From Previous Page)
Project Project title Project performer( s) stakeholders
Implementing Remote Sensing Applications to NCRST- E/ FHWA/ Mississippi
Department of Transportation FHWA, FRA Develop and Environmental Impact
Statement and Decision Options to Relocate the Current CSX
Railroad from Mississippi Gulf Coast townships to the I- 10 Right of Way
Remote Sensing Applications for
NCRST- E/ Washington State Department of Transportation FHWA Environmental
Analysis in Transportation Planning
Highway Features and Characteristics Database NCRST- I/ Florida Department
of Transportation FHWA Development Using Commercial Remote Sensing
Technology, Combined with Mobile Mapping, GIS and GPS. Source: RSPA.
Legend FAA Federal Aviation Administration FHWA Federal Highway
Administration FRA Federal Railroad Administration FTA Federal Transit
Administration MARAD Maritime Administration RSPA Research and Special
Programs Administration
Note: This program is performed in partnership with four major consortia
consisting of (1) the National Consortia for Remote Sensing in
Transportation - Environmental Assessment/ Application (NCRST- E); (2) the
National Consortia for Remote Sensing in Transportation - Traffic Flow
(NCRST- F); (3) the National Consortia for Remote Sensing in
Transportation - Safety, Hazards, and Disasters (NCRST- H); and (4) the
National Consortia for Remote Sensing in Transportation - Infrastructure
Management (NCRST- I).
Table 4: Human- Centered Systems: Operator Fatigue Management Program
Project Project title Project performer( s) stakeholders
Framework for Multimodal Synthesis and Cost ORC Macro All DOT modes
Benefit Analyses through the Human Factors Coordinating Committee
Work Schedule Representation Analysis Software XIMES GmbH All DOT modes
through the Human Factors Coordinating Committee
Development of a Fatigue Management Reference Battelle Memorial Institute,
Columbus Operations All DOT modes
Handbook through the Human Factors Coordinating Committee
Fatigue Analysis Systems for Transportation Science Applications
International Corporation All DOT modes
Operations: Railroad Applications through the Human Factors Coordinating
Committee
Source: RSPA. Legend DOT Department of Transportation
Table 5: Transportation Infrastructure Assurance Program Project title
Project performer( s) Project stakeholders
Interdependency of the Transportation System with Volpe National
Transportation
Office of Intelligence and Security, TSA, FAA, Other Critical
Infrastructures Systems Center Louisiana Offshore Oil Port, TRB.
Transportation and Logistical Requirements for
Volpe National Transportation DOT Maritime Academy, FAA, Federal
Emergency Response Teams in Dealing with Weapons Systems Center Emergency
Management Adminstration,
of Mass Destruction FHWA, MARAD, RSPA*s Office of Emergency
Transportation. Feasibility of Alternative Backup Systems for the
Volpe National Transportation Booz- Allen Hamilton, DOT Office of the
Global Positioning System Systems Center Secretary, FAA, FRA, Northrop-
Grumman,
USCG. Options to Transition Hazardous Materials
Volpe National Transportation TSA, American Association of Railroads,
Transportation Security Guidelines to Security Systems Center American
Chemistry Council, Bureau of
Requirements Transportation Statistics, DOT Office of the
Secretary, FMCSA, FRA, GAO, TRB, RSPA*s Office of Hazardous Materials
Safety, RSPA*s Office of Pipeline Safety, and USCG, American Institute of
Chemical Engineers, the National Transportation Safety Board, the Inland
Rivers, Ports and Waterways
Association, the Bureau of the Census, the U. S. Army Corps of Engineers,
and Vanderbilt University.
Source: RSPA. Legend DOT Department of Transportation FAA Federal Aviation
Administration FHWA Federal Highway Administration FMCSA Federal Motor
Carrier Safety Administration FRA Federal Railroad Administration MARAD
Maritime Administration RSPA Research and Special Programs Administration
TRB Transportation Research Board
TSA Transportation Security Administration USCG United States Coast Guard
(542013)
GAO*s Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities
and to help improve the performance and accountability of the federal
government for the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO*s commitment to good
government is reflected in its core values of accountability, integrity,
and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost is
through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext GAO Reports and
files of current reports and testimony and an expanding archive of older
Testimony
products. The Web site features a search engine to help you locate
documents using key words and phrases. You can print these documents in
their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this
list to you every afternoon, go to www. gao. gov and select *Subscribe to
GAO Mailing Lists* under *Order GAO Products* heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO
also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov Federal Programs
Automated answering system: (800) 424- 5454 or (202) 512- 7470 Public
Affairs Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512-
4800
U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548
a
GAO United States General Accounting Office
RSPA has met some, but not all, legislative and DOT requirements
pertaining to the coordination of departmental research efforts. For
example, while RSPA develops an annual plan and meets monthly with other
DOT research officials, RSPA does not review the status of all DOT
research activities. Thus, it cannot determine whether duplication of
research efforts within
DOT does or does not occur. Additionally, RSPA has not developed standards
against which to measure its performance in coordinating research within
DOT. Moreover, RSPA has not fully met all legislative and DOT requirements
to measure research results and oversee research evaluations across DOT.
RSPA officials cited a lack of ready information on DOT research
activities budget constraints and a lack of authority over other DOT
agencies as reasons why they served primarily an information- sharing
role, rather than as an overseer and manager of the coordination and
evaluation processes.
Extent to Which RSPA Meets Coordination and Evaluation Requirements
Since 1999, RSPA has budgeted $37 million to conduct four major research
programs with applicability to more than one mode of transportation* for
example, using technology to improve energy efficiency and reduce
emissions and transportation dependence on petroleum. According to the
Associate Administrator for Innovation, Research, and Education, RSPA*s
current multimodal research programs are scheduled for completion by the
end of fiscal year 2004 and have had a variety of positive results.
However, RSPA does not have an evaluation process to systematically
evaluate the
results of its multimodal research programs. TRANSPORTATION RESEARCH
Actions Needed to Improve Coordination and Evaluation of Research
www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 500 To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Kate Siggerud at (202) 512- 2834 or siggerudk@ gao.
gov. Highlights of GAO- 03- 500, a report to the
Committee on Appropriations, House of Representatives
May 2003
The Research and Special Programs Administration (RSPA) within the
Department of Transportation (DOT) is
responsible for coordinating and ensuring the evaluation of DOT research
programs to promote the efficient use of departmental research funds,
which in fiscal year 2002 totaled over $1 billion. RSPA
is also responsible for conducting multimodal research that cuts across
different modes of
transportation. The House Committee on Appropriations directed GAO to
examine RSPA*s coordination and evaluation of research within DOT and the
status
of its own multimodal research. GAO is recommending that DOT and RSPA
develop strategies to identify potential research duplication and ensure
that the
results of all DOT transportation research activities* including those
conducted by RSPA* are evaluated. Further, GAO recommended that RSPA
assess
the effectiveness of its research coordination efforts by developing
appropriate performance measures.
DOT reviewed a draft of this report and generally agreed with its contents
but did not comment on the report*s recommendations.
Page i GAO- 03- 500 Transportation Research
Contents
Contents
Page ii GAO- 03- 500 Transportation Research
Page 1 GAO- 03- 500 Transportation Research United States General
Accounting Office
Washington, D. C. 20548 Page 1 GAO- 03- 500 Transportation Research
A
Page 2 GAO- 03- 500 Transportation Research
Page 3 GAO- 03- 500 Transportation Research
Page 4 GAO- 03- 500 Transportation Research
Page 5 GAO- 03- 500 Transportation Research
Page 6 GAO- 03- 500 Transportation Research
Page 7 GAO- 03- 500 Transportation Research
Page 8 GAO- 03- 500 Transportation Research
Page 9 GAO- 03- 500 Transportation Research
Page 10 GAO- 03- 500 Transportation Research
Page 11 GAO- 03- 500 Transportation Research
Page 12 GAO- 03- 500 Transportation Research
Page 13 GAO- 03- 500 Transportation Research
Page 14 GAO- 03- 500 Transportation Research
Page 15 GAO- 03- 500 Transportation Research
Page 16 GAO- 03- 500 Transportation Research
Page 17 GAO- 03- 500 Transportation Research
Page 18 GAO- 03- 500 Transportation Research
Page 19 GAO- 03- 500 Transportation Research
Page 20 GAO- 03- 500 Transportation Research
Page 21 GAO- 03- 500 Transportation Research
Page 22 GAO- 03- 500 Transportation Research
Page 23 GAO- 03- 500 Transportation Research
Page 24 GAO- 03- 500 Transportation Research
Page 25 GAO- 03- 500 Transportation Research
Page 26 GAO- 03- 500 Transportation Research
Page 27 GAO- 03- 500 Transportation Research
Page 28 GAO- 03- 500 Transportation Research
Page 29 GAO- 03- 500 Transportation Research
Page 30 GAO- 03- 500 Transportation Research
Page 31 GAO- 03- 500 Transportation Research
Page 32 GAO- 03- 500 Transportation Research
Page 33 GAO- 03- 500 Transportation Research
Page 34 GAO- 03- 500 Transportation Research
Page 35 GAO- 03- 500 Transportation Research
Appendix I
Appendix I Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by RSPA Page 36 GAO- 03- 500 Transportation Research
Appendix I Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by RSPA Page 37 GAO- 03- 500 Transportation Research
Appendix I Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by RSPA Page 38 GAO- 03- 500 Transportation Research
Appendix I Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by RSPA Page 39 GAO- 03- 500 Transportation Research
Appendix I Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by RSPA Page 40 GAO- 03- 500 Transportation Research
Appendix I Stakeholders and Researchers Involved in the Four Multimodal
Programs Conducted by RSPA Page 41 GAO- 03- 500 Transportation Research
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested
Presorted Standard Postage & Fees Paid
GAO Permit No. GI00
*** End of document. ***