Transportation Research: Opportunities for Improving the
Oversight of DOT's Research Programs and User Satisfaction with
Transportation Statistics (15-AUG-06, GAO-06-917).
The Department of Transportation's (DOT) research, development,
and technology (RD&T) budget totaled $1.1 billion in fiscal year
2005. DOT's Research and Innovative Technology Administration
(RITA)--which includes the Bureau of Transportation Statistics
(BTS)--oversees DOT's RD&T activities. GAO examined (1) how
RITA's responsibilities for overseeing DOT's RD&T activities
differ from those of its predecessor, the Research and Special
Programs Administration (RSPA); (2) RITA's practices for
coordinating, facilitating, and reviewing RD&T activities; (3)
the progress DOT has made in implementing GAO's 2003
recommendations on how to improve the coordination and evaluation
of RD&T activities; and (4) how BTS identifies and monitors how
well it serves its users. To address these issues, GAO reviewed
relevant documentation and interviewed officials from RITA, BTS,
and three operating administrations.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-917
ACCNO: A58681
TITLE: Transportation Research: Opportunities for Improving the
Oversight of DOT's Research Programs and User Satisfaction with
Transportation Statistics
DATE: 08/15/2006
SUBJECT: Agency missions
Program evaluation
Program management
Research and development
Research program management
Systems evaluation
Transportation
Transportation research
Transportation statistics
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-917
United States Government Accountability Office
GAO
August 2006
TRANSPORTATION RESEARCH
Opportunities for Improving the Oversight of DOT's Research Programs and User
Satisfaction with Transportation Statistics
GAO-06-917
TRANSPORTATION RESEARCH
Opportunities for Improving the Oversight of DOT's Research Programs and
User Satisfaction with Transportation Statistics
What GAO Found
In 2005, RITA took over RSPA's responsibilities for overseeing DOT's RD&T
activities when RSPA was dissolved. While RITA's mission and strategic
objectives are similar to those RSPA had, RITA differs from RSPA in a
number of ways. For example, RITA proposed a $2 million increase in its
fiscal year 2007 budget request for the oversight of DOT's RD&T activities
through its proposed Transportation Futures and Applied Technology
Program, which, among other things, would provide access to technical
experts to RITA on a contract basis. Additionally, RITA's responsibility
for evaluation is less clearly defined than RSPA's. RITA, unlike RSPA, is
not required to measure the results or evaluate the effectiveness of RD&T
activities. However, RITA is not explicitly prevented from evaluating such
activities.
RITA coordinates, facilitates, and reviews DOT's RD&T activities through
various practices. For example, RITA has two coordinating bodies-the RD&T
Planning Council and the RD&T Planning Team-and conducts budget reviews,
among other practices. RITA has not, however, established performance
goals, a clear implementing strategy, or an evaluation plan that
delineates how its coordination, facilitation, and review practices will
further DOT's mission or ensure the effectiveness ofits RD&T investment.
Without such a strategic approach, it is difficult for RITA to ensure that
DOT is making the most of its approximately $1 billion annual RD&T
investment.
RITA has partially implemented four of our recommendations and has not
implemented the other. For example, while RITA, through its two
coordinating bodies, has taken some action to review RD&T activities for
duplication and opportunities for joint efforts, RITA has not established
the scope of RD&T activities to be reviewed, the methodology of the
review, or how the results will be used to make decisions about future
RD&T activities.
BTS does not have a systematic process for identifying its primary users,
soliciting ongoing feedback from those users, and determining whether or
how that feedback should be incorporated. For example, rather than
identify specific users of BTS data products and services, BTS considers
its users to be those broad categories of intended users describedin
federal legislation. Further, rather than routinely soliciting user
feedback on all data products and services, such as through a customer
satisfaction survey, BTS only solicits limited feedback from some users of
specific products through conferences, workshops, or other meetings.
Finally, BTS relies onits program managers to evaluate and determine how
best to address feedback from its users; however, the managers are not
required to-and often do not-report the results of whether or how they
considered user feedback. Without a systematic process for identifying its
users, soliciting ongoing feedback, and determining whether or how that
feedback should be incorporated, BTS is limitedin its ability to consider
feedback and use it to make improvements to data products.
United States Government Accountability Office
Contents
Letter 1
Results Summary 4
Conclusions 11
Recommendations for Executive Action 12
Agency Comments 14
: Status of RITA's Implementation of GAO's 2003
Recommendations
Appendix I Review of DOT's Research and Innovative Technology
Administration
Appendix II GAO Contact and Staff Acknowledgments
Table
Abbreviations
BTS Bureau of Transportation Statistics
DOT Department of Transportation
EAS Essential Air Service
FAA Federal Aviation Administration
FHWA Federal Highway Administration
FRA Federal Railroad Administration
FTA Federal Transit Administration
GPRA Government Performance and Results Act of 1993
MPO Metropolitan Planning Organization
NHTS National Household Travel Survey
NHTSA National Highway Traffic Safety Administration
OMB Office of Management and Budget
OST Office of the Secretary of Transportation
PHMSA Pipeline and Hazardous Materials Safety Administration
PRA Paperwork Reduction Act
R&D research and development
RD&T research, development, and technology
RITA Research and Innovative Technology Administration
RSPA Research and Special Programs Administration
SAFETEA-LU Safe, Accountable, Flexible, Efficient Transportation Equity
Act: A Legacy for Users TEA-21 Transportation Equity Act for the 21st
Century TSAR Transportation Statistics Annual Report UTC University
Transportation Centers
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributedin
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office Washington, DC 20548
August 15, 2006
The Honorable Christopher Bond Chairman The Honorable Patty Murray Ranking
Minority Member Subcommittee on Transportation, Treasury, the Judiciary,
Housing and
Urban Development, and Related Agencies Committee on Appropriations United
States Senate
The Honorable Joe Knollenberg Chairman The Honorable John W. Olver Ranking
Minority Member Subcommittee on Transportation, Treasury, Housing and
Urban
Development, the Judiciary, the District of Columbia, and Independent
Agencies Committee on Appropriations House of Representatives
In fiscal year 2005, the Department of Transportation's (DOT) research,
development, and technology1 (RD&T) budget totaled approximately $1.1
billion, including projects undertaken by DOT's operating administrations,
such as the Federal Highway Administration and the Federal Aviation
Administration. RD&T activities are vital to meeting DOT's key
transportation priorities, such as increasing transportation safety and
enhancing mobility for all Americans. Prior to 2005, DOT's Research and
Special Programs Administration (RSPA) was responsible for overseeing
DOT's RD&T activities, the Office of Pipeline Safety, and the Office of
Hazardous Materials Safety. Over the years, GAO and others have raised
concerns about RSPA's capabilities for improving RD&T coordination and
1OMB Circular No. A-11 (2005) defines research and development activities
to include "work undertaken on a systematic basis ... to increase the
stock of knowledge, including knowledge of man, culture and society, and
the use of this stock of knowledge to devise new applications." DOT
defines technology to include demonstration projects and other related
activities associated with research and development activities.
Page 1 GAO-06-917 Research and Innovative Technology Administration
evaluation across DOT, and, in 2003, GAO made several recommendations to
improve those efforts.2
In response to these concerns and to help delineate and clarify roles and
responsibilities for overseeing DOT's RD&T activities, the Norman Y.
Mineta Research and Special Programs Improvement Act of 2004 (Pub. L. No.
108-426) dissolved RSPA and created two new administrations-the Pipeline
and Hazardous Materials Safety Administration3 (PHMSA) and the Research
and Innovative Technology Administration (RITA). RITA was created to
provide DOT with a more focused research organization and to assist in
avoiding DOT-wide research duplication and inefficiency, among other
purposes.4 The act also transferred the Bureau of Transportation
Statistics (BTS) and the Office of Intermodalism to RITA. Through this
act, RITA is responsible for coordinating, facilitating, and reviewing
DOT's RD&T programs and activities, which include the activities conducted
by DOT's operating administrations as well as other RD&T and statistical
programs managed by RITA (e.g., BTS, the Office of Intermodalism, and
University Transportation Centers).
In Senate Report 109-109, accompanying the DOT and related agencies
appropriation bill for fiscal year 2006, the House and Senate Committees
on Appropriations directed GAO to assess how RITA is resolving the
concerns of its predecessor administration and better coordinating DOT's
RD&T and statistical activities. Specifically, this report discusses (1)
how RITA's responsibilities for overseeing DOT's RD&T activities differ
from RSPA's; (2) RITA's practices for coordinating, facilitating, and
reviewing RD&T activities; (3) the progress RITA has made in implementing
GAO's 2003 recommendations; and (4) how BTS identifies its users and
monitors how well it is serving those users.
To determine how RITA's oversight responsibilities differ from RSPA's, we
collected information through legislative histories, document reviews, and
interviews with officials within RITA and used it to compare RITA and
2
GAO, Transportation Research: Actions Needed to Improve Coordination and
Evaluation of Research, GAO-03-500 (Washington, D.C.: May 1, 2003). In
this report, GAO uses the broad term "research" to refer to DOT's
research, development, and technology activities.
3PHMSA includes the Office of Pipeline Safety and the hazardous materials
safety activities formerly in RSPA.
4
H.R. Rep. No. 108-749 (2004).
RSPA with respect to mission, organizational structure, responsibility for
overseeing RD&T activities, work activities, budgetary resources, and
strategic goals. To determine RITA's practices for coordinating,
facilitating, and reviewing DOT's RD&T activities and what progress RITA
and RSPA have made in implementing GAO's 2003 recommendations, we reviewed
pertinent documentation and conducted semistructured interviews with
officials in RITA, the Office of the Secretary of Transportation, and
three of nine operating administrations that received RD&T funds in fiscal
year 2005. The three operating administrations we selected for
interviews-Federal Highway Administration, Federal Aviation
Administration, and Federal Transit Administration-are among those with
the largest RD&T budgets.5 To determine how BTS identifies its users and
monitors how well it is serving those users, we (1) identified criteria
for monitoring user satisfaction, including federal and other agency
requirements and guidelines set by DOT, the National Research Council, and
GAO; (2) through document reviews and interviews with BTS and operating
administration officialsidentified above, obtained information on BTS's
processes for identifying key users of its data products and for
soliciting and incorporating feedback from those users; and (3) compared
BTS's processes to the criteria to identify major gaps. We assessed the
reliability of the information contained in this report through interviews
with knowledgeable officials and reviews of documentation and
corroborating information, and we determined that it was sufficiently
reliable for our purposes. We conducted our work from November 2005
through August 2006 according to generally accepted government auditing
standards. This report summarizes the information we provided to your
staff during our May 30, 2006, briefing and, in addition, contains
recommendations to the Secretary of Transportation to improve RITA's
oversight of the department's RD&T activities and user satisfaction with
transportation statistics. The briefing slides are included in appendix I.
Taken together, the RD&T budget authority for these three operating
administrations accounted for over 80 percent of DOT's total RD&T budget
authorityin fiscal year 2005.
Page 3 GAO-06-917 Research and Innovative Technology Administration
Results Summary
RITA Differs from RSPA in Proposed Budgetary Levels, Responsibility for
Evaluation, and Extent of Multimodal Focus
Beginning in 2005, RITA took over RSPA's responsibilities for overseeing
DOT's RD&T activities. While RITA continues to have a similar mission and
strategic objectives as RSPA-and still does not have the authority to
direct changes in the operating administrations' RD&T activities-RITA
differs from RSPA in several ways. First, RITA intends to focus
significantly more resources in fiscal year 2007 on the oversight of DOT's
RD&T activities. In DOT's fiscal year 2007 budget request, RITA proposes a
new program-the Transportation Futures and Applied Technology Program-to
build upon RITA's existing RD&T Coordination Program and provide access to
highly skilled, specialized technical experts to RITA on a contract basis.
If enacted, funding for these two programs would result in an
overallincrease of about $2 million over the enacted fiscal year 2006
budget for the RD&T Coordination Program-from $536,000 in fiscal year 2006
to about $2.5 million in fiscal year 2007.6 Second, RITA's responsibility
for evaluation is less clearly defined than RSPA's had been. RSPA had both
legislative and departmental responsibility for measuring the results of
DOT's RD&T programs and developing more efficient, effective, and
participative ways to evaluate and measure RD&T program effectiveness.7
However, RITA does not have the same legislative responsibility for
evaluating DOT's RD&T programs because the Safe, Accountable, Flexible,
Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU)8
removed the statutory responsibility for evaluation
6In RITA's fiscal year 2007 budget request, the amount RITA requested for
its RD&T Coordination Program ($247,000) is less than half of what was
enacted for those activities in fiscal year 2006 ($536,000). According to
the 2007 budget request, this decrease represents the transfer of
multimodal RD&T initiatives from the RD&T Coordination Program to the
Transportation Futures and Applied Technology Program. DOT's documentation
does not provide a clear transfer of all of the multimodal initiatives
from one program to the other; however, DOT officials told us that they
plan to review all of the department's multimodal initiatives as part of
the Transportation Futures and Applied Technology Program. As of June
2006, the House Committee on Appropriations denied RITA's request for $2.2
million for the Transportation Futures and Applied Technology Program,
saying that RITA should develop a more robust level of in-house RD&T
expertise beforeit relies on outside contractors for multimodal research
coordination and analysis. The House Committee on Appropriations also
provided $540,000 for RITA's ongoing RD&T Coordination Program.
7
Transportation Equity Act for the 21st Century (TEA-21), formerly codified
in 23 USC 508. Pub. L. No. 105-178, 112 Stat. 107 (1998).
8
Pub. L. No. 109-59, 119 Stat. 1144 (2005).
RITA Has Several Coordination, Facilitation, and Review Groups and
Practices but Lacks Performance Goals and a Plan for Evaluating Its Own
Efforts
activities that had been specifically outlined for DOT, which DOT
delegated to RSPA. Under the Norman Y. Mineta Act, RITA is charged with
reviewing DOT's RD&T activities, which, according to RITA officials, means
that RITA is not required to evaluate RD&T activities to determine whether
they are achieving intended goals, although RITA is not explicitly
prevented from evaluating such activities. Third, RITA's organizational
structure is more multimodally9 focused because the Office of Pipeline
Safety and Office of Hazardous Materials Safety were moved to the newly
created PHMSA, and the remaining offices in RSPA and other program
offices-BTS and the Office of Intermodalism-were transferred to the newly
created RITA.
RITA Has Several Coordination, Facilitation, and Review Groups and
Practices but Lacks Performance Goals and a Plan for Evaluating Its
Own Efforts
RITA coordinates, facilitates, and reviews DOT's RD&T activities in a
variety of ways, including through its two coordinating bodies-the RD&T
Planning Council and the RD&T Planning Team-and budget reviews, among
others. The briefing slides in appendix I contain a summary table and
detailed descriptions of RITA's activities. RITA officials and officials
from the three operating administrations we interviewed provided some
examples of the usefulness of RITA's coordination, facilitation, and
review practices; for example, RITA officials told us that the budget
review process results in a more consistent approach for the operating
administrations to show how their RD&T activities support DOT's strategic
objectives, secretarial priorities, and multimodal initiatives. Officials
from all three operating administrations told us that RITA's Program
Review Working Group10 meetings provide them with opportunities to share
best practices relating to program management issues, such as how to apply
the Office of Management and Budget's (OMB) R&D Investment Criteria11 to
RD&T activities.
While RITA and other DOT officials provided anecdotal examples of the
positive effects of RITA's coordination, facilitation, and review
practices,
9Individual transportation modes include mass transit systems, roads,
aviation, maritime, and railroads. RITA and its program offices oversee
and conduct RD&T and statistical activities that apply to more than one
mode of transportation.
10
The Program Review Working Group reports to the RD&T Planning Team and
provides a forum for the operating administrations to share information
about areas of research and identify opportunities for coordination.
11
OMB developed the R&D Investment Criteria-relevance, quality, and
performance-for all types of research and development programs across all
agencies to assist in budget allocation decisions and to demonstrate
results.
RITA has not established performance goals or an implementing strategy
that delineates how the activities and results of its coordination,
facilitation, and review practices will further DOT's mission and ensure
the effectiveness of the department's RD&T investment; in addition, RITA
does not monitor or evaluate the effects of its efforts. Also, RITA has
not worked with the operating administrations to develop common
performance measures for DOT's RD&T activities. RITA officials told us
that they were working with the operating administrations to develop an
RD&T strategic plan, which is due to Congress in the fall of 2006, but
they did not expect the plan toinclude goals and measures, as discussed
above. Setting meaningful goals for performance, and using performance
information to measure performance against those goals, is consistent with
requirements in the Government Performance and Results Act of 1993
(GPRA).12 Developing an evaluation plan and analyzing performance
information against set goals for its own coordination, facilitation, and
review practices could assist RITA in identifying any problem areas and
taking corrective actions.13 Without such goals and an evaluation plan, it
is difficult for RITA to determine its success in overseeing and ensuring
the effectiveness of DOT's RD&T activities. Also, without common
performance measures for the RD&T activities of the operating
administrations, RITA and the operating administrations lack the means to
monitor and evaluate the collective results of those activities and ensure
that they are achieving their intended (or other) results and furthering
the Secretary's and DOT-wide priorities. Linking performance goals with
the planning and budget process, such as DOT's annual budget process, can
also help RITA determine where to target its resources to improve
performance.14
12
Pub. L. No.103-62, 107 Stat. 285 (1993).
13
Use of performance goals can help ensure that programs are meeting their
intended goals, allows programs to assess the efficiency of their
processes, and promotes continuous improvement. Where activities may be
fragmented or overlap, performanceinformation can also help identify
performance variations and redundancies and lay the foundation for
improved coordination, program consolidation, or elimination of unneeded
programs. GAO, Managing for Results: Using the Results Act to Address
Mission Fragmentation and Program Overlap, AIMD-97-146 (Washington, D.C.:
August 1997).
14
GAO, Managing for Results: Enhancing Agency Use of Performance Information
for Management Decision Making, GAO-05-927 (Washington, D.C.: Sept. 9,
2005).
RITA Has Made Some Progress in Implementing GAO's 2003 Recommendations
In 2003, GAO made five recommendations to DOT and RSPA
to improve the coordination and evaluation of
RD&T activities.15 The recommendations generally remain
relevant for RITA. As shown in table 1, RITA has partially
implemented four recommendations and has not implemented one
recommendation. The briefing slides in appendix I
contain more detailed information on RITA's actions with regard to our
recommendations.
Table 1: Status of RITA's Implementation of GAO's 2003 Recommendations
Recommendations Status of implementation
Develop a strategy for reviewing all of DOT's research projects to
identify areas of unnecessary Partially implemented research duplication,
overlap, and opportunities for joint efforts.
o Include time frames for implementing this review and discuss the
development and implementation of a DOT-wide research tracking system
database.
o Incorporate the results of this effort into DOT's annual research plan
and report to Congress on an annual basis.
Develop and apply quantifiable performance measures to assess the
effectiveness of research Not implemented coordination efforts and
document the results of these efforts in DOT's annual research plan.
Develop a strategy to ensure that the results of all of DOT's
transportation research activities are Partially implemented evaluated
according to established best practices.
o Include estimates of the costs for ensuring that evaluations are
completed.
o Incorporate the results of these efforts in DOT's annual research plan
and report to Congress on an annual basis.
Include in DOT's annual research plan a summary of all research program
evaluations conducted Partially implemented and a schedule of future
evaluations.
Document RSPA's process for systematically evaluating the results of its
own multimodal research Partially implemented programs, and apply this
process to any future multimodal research programs that RSPA conducts.
Source: GAO analysis of RITA information.
RITA has made progress, but much remains to be done for RITA to fully
implement the recommendations. The following is a summary, for each
recommendation, of the actions taken by RITA (and RSPA) and the type of
efforts that are still needed to fully implement the recommendations:
o The first recommendation focuses on developing a strategy for
identifying areas of unnecessary research duplication, overlap, and
opportunities for joint efforts. While RITA officials have taken some
actions to identify areas of unnecessary research duplication and
opportunities for joint efforts through meetings of its RD&T Planning
Council and Planning Team,
15 GAO-03-500 .
among others, none of these groups have developed a strategy that
describes (1) the scope of the RD&T projects or programs that will be
reviewed for duplication or joint efforts, (2) the methodology for how all
research projects will be reviewed or how duplication or joint efforts
will be identified, (3) a timeline and the frequency for reviews to occur,
or (4) how the results of reviews-the identification of duplication or an
opportunity for joint effort-will be used to make decisions about future
RD&T activities.16 The first recommendation also includes the development
of a DOT-wide research tracking system database, which, according to a
RITA official, was dropped from DOT's priorities after the creation of
RITA. While RITA's proposed Transportation Futures and Applied Technology
Program includes the development of a Web-based database for DOT's RD&T
programs, it is uncertain whether this proposed program will be funded.
RITA has not yet established a catalog of all of the research projects
within DOT; as such, DOT officials do not have readily accessible data on
research activities throughout DOT. Without such a strategy-supported by a
comprehensive database of ongoing RD&T projects-RITA is unable to ensure
that areas of research duplication, overlap, and opportunities for joint
efforts are systematically identified and managed.
o DOT and RSPA did not concur with the second recommendation in 2003,
citing their views that (1) the most useful and effective performance
measures focus on results, while coordination is a process and (2)
that existing coordination processes have been effective in preventing
unnecessary research duplication. While RITA officials told us that
they have not developed or applied quantifiable performance measures
for coordination because they do not believe that a metric approach is
well suited for assessing the effects of the coordination process,
they indicated their willingness to develop-in concert with the
operating administrations-common performance measures for DOT's RD&T
activities.
o The third recommendation calls for developing a strategy for ensuring
the evaluation of DOT's RD&T activities. RITA officials told us that
they ensure the evaluation of RD&T activities by reviewing the
operating administrations' application of OMB's R&D Investment
Criteria- relevance, quality, and performance-through its budget
review process.
Strategies help align an agency's activities, core processes, and
resources to support achievement of the agency's strategic goals and
mission. GAO, Results-Oriented Government: GPRA Has Established a Solid
Foundation for Achieving Greater Results, GAO-04-38 (Washington, D.C.:Mar.
10, 2004).
Page 8 GAO-06-917 Research and Innovative Technology Administration
According to RITA officials and OMB documents, the R&D Investment Criteria
are rooted in best practices and include peer review as a mechanism for
assessing program quality. However, RITA has neither developed nor
communicated a strategy for this process that describes (1) the scope of
RD&T activities of the operating administrations that RITA will ensure
were evaluated according to best practices, (2) the methodology for how
RITA will ensure evaluation of RD&T activities took place according to
established best practices, (3) a timeline for when the RD&T evaluations
should occur, and (4) how the results of the RD&T evaluations will inform
future research. Without such a strategy, RITA is less able to ensure the
quality and effectiveness of RD&T activities and investments to determine
whether they are achieving their intended (or other) goals.
o The fourth recommendation focuses on publishing a summary of research
program evaluations and a schedule of future evaluations. RSPA, in its
fiscal year 2005 annual RD&T plan, published such a summary, but it
consisted only of the results of its reviews of the operating
administrations' application of OMB's R&D Investment Criteria. This
summary did not include other research program evaluations or a
schedule of future evaluations. Since RSPA was dissolved, RITA has not
continued to publish the results of these types of reviews because
SAFETEA-LU removed the requirement for RITA to submit an annual RD&T
plan. Publishing a current inventory of the evaluation of research
activities and a schedule for future evaluations on a regular basis
could provide information about research results and planned research
for future years.
o The fifth recommendation addresses RITA's process for systematically
evaluating the results of its own current multimodal research
programs, such as the Hydrogen Safety Program, and future multimodal
research programs. While RITA officials told us they oversee contracts
and evaluate the results of RD&T activities that are conducted under
these contracts through a peer review process, RITA has not
systematically documented this process and it is not clear whether and
how this process would apply to future multimodal research programs.
Without a systematic process for evaluating current and future program
results, RITA is limitedin its ability to determine the extent to
which its multimodal RD&T programs are achieving their intended (or
other) goals.
BTS's Process for Identifying Its Users and Monitoring How Well It Is
Serving Those Users Is Not Systematic
BTS does not have a systematic process in place for identifying its
primary users, soliciting ongoing feedback from those users, and
determining whether or how that feedback should be incorporated. First,
BTS has not established a systematic process for comprehensively
identifying its primary users, it does not track specific users, and it
does not have information on the overall number of users of its data
products. BTS officials told us that their users are primarily identified
in SAFETEA-LU, which only defines broad categories of data users, such as
the federal and state governments. BTS officials told us they also
identify some specific users through other methods, such as direct
inquiries received through BTS's Web site and by telephone. One BTS
official also commented that it is difficult to track individual users,
other than through Web site hits and tracking the number of publications
ordered. Also, the official said that BTS is limited in its ability to
collect information on individual data users because of privacy concerns.
Without a systematic process for comprehensively identifying primary users
for each of its products and services, BTS cannot solicit feedback from
these users on an ongoing basis to improve those products and services.
Secondly, BTS has not established a systematic process for soliciting
feedback from all of its primary users, although it has used a variety of
methods to obtain feedback from some users on a case-by-case basis. For
example, rather than routinely soliciting user feedback on all data
products and services, such as through a customer satisfaction survey, BTS
receives feedback from some data users about specific BTS products from an
online comment form and meetings and workshops held at conferences and
training sessions.17 Without a systematic process for soliciting feedback
on user satisfaction from all of its primary users, BTS cannot ensure it
has a full picture of the needs of those users and how well it is meeting
those needs, which, in turn, hinders BTS's ability to make improvements to
data products that are relevant to users.
Finally, to evaluate the feedback BTS has solicited from its users, BTS
officials told us they rely on program managers to determine how best to
17
In the past, BTS systematically solicited feedback on overall customer
satisfaction of BTS's data products and services through its Customer
Satisfaction Survey. BTS conducted the last such survey in 1998. Also, in
the past, BTS solicited feedback via comment cards, inserted in data
products. According to BTS officials, the survey and comment cards are no
longer used as methods for soliciting feedback as a result of restrictions
stemming from the Privacy Act and the Paperwork Reduction Act. BTS
officials told us they are working with the Chief Counsel's office at RITA
to reinstitute the use of comment cards.
Conclusions
address feedback from its users; however, program managers are not
required to-and often do not-report the results of how feedback on user
satisfaction was considered. Although BTS measures the number of
congressional and governmental agency contacts regarding BTS information
and the average number of daily unique visitors to the TranStats data Web
site,18 these indicators are only simple counts, not measures of user
satisfaction with BTS information. GAO's internal control standards
suggest that ongoing monitoring take place to determine user satisfaction
and that policies and procedures be put in place to ensure that feedback
is evaluated so that improvements to products can be made.19 Without a
systematic process for identifying the primary users for each of its data
products and services, soliciting feedback on user satisfaction from its
users, and incorporating that feedback, BTS is limited in its ability to
consider feedback and make improvements to data products based on user
input.
Conclusions
Since it became operational in 2005, RITA has taken some positive steps to
meet its vision of becoming a departmentwide resource for managing and
ensuring the effectiveness of RD&T activities. In particular, RITA
established several coordinating bodies and review processes, and it has
also proposed additional initiatives, such as the Transportation Futures
and Applied Technology Program, to build upon its current coordination,
facilitation, and review practices. However, RITA lacks performance goals,
a clear implementing strategy, and an evaluation plan that collectively
delineates how the activities and results of its coordinating bodies,
review processes, and proposed initiatives will further DOT's mission and
ensure the effectiveness of the department's RD&T investment. Establishing
these mechanisms for managingits own performance-and linking them to DOT's
annual budget process-could provide RITA with a clear road map for
investing its own limited resources; enhance RITA's ability to identify
areas where its coordination, facilitation, and review efforts are working
effectively and where they could be improved; and help RITA provide
assurance that the department's RD&T activities are adequately
coordinated, routinely evaluated, and achieve their intended (or other)
results. In addition, RITA and the operating administrations have not
18
Required by SAFETEA-LU, the TranStats data Web site contains hundreds of
transportation-related databases and is made available to the public on
BTS's Web site.
19
GAO, Internal Control Management and Evaluation Tool, GAO-01-1008G
(Washington, D.C.: August 2001).
Page 11 GAO-06-917 Research and Innovative Technology Administration
worked together to develop common performance measures for RD&T activities
that are needed to evaluate the RD&T efforts departmentwide. With
performance goals, an implementing strategy, and an evaluation plan for
RITA-and common performance measures for RD&T activities- RITA and DOT
could be in a better position to assure Congress that DOT is making the
most of its approximately $1 billion annual RD&T investment. Additionally,
the strategy and performance measures could serve as a communication tool
to establish expectations and anticipated results with the operating
administrations.
BTS, as part of RITA, has solicited and evaluated some feedback on user
satisfaction, but it does not have a systematic process for identifying
primary users of its transportation data products and services, soliciting
feedback from those users, or incorporating the feedback it solicits. As a
result, BTS cannot ensure that it has a comprehensive picture of who uses
BTS data products, what their needs are, and how well the agency is
meeting those needs with its data products. By establishing more
systematic processes for identifying its primary users, soliciting
feedback from those users, and evaluating feedback on user satisfaction,
BTS could make more informed decisions on how to allocate limited
resources to make improvements to its data products. While BTS tracks and
reports the number of congressional and government agency contacts and the
number of visitors to its TranStats data Web site, developing performance
indicators that measure the overall degree to which products and services
are useful and responsive to the needs of its users will allow BTS to
understand how well it is serving its users over time.
Recommendations for Executive Action
To enhance RITA's ability to manage and ensure the effectiveness of
RD&T activities in furthering the department's mission, we
recommend that the Secretary direct the RITA Administrator
to take the following seven actions:
* Develop and incorporate the following into RITA's fiscal year 2008
budget process, and the annual budget process thereafter:
o performance goals and an overallimplementing strategy that
delineate how the activities and results of its coordination,
facilitation, and review practices will further DOT's mission
and ensure the effectiveness of the department's RD&T
investment. The strategy should include an evaluation plan for
monitoring and evaluating its performance against set goals to
assist RITA in better allocating its resources to improve
performance.
o common performance measures related to DOT's RD&T activities,
which should be developed in consultation with the operating
administrations.
* Develop and incorporate the following into RITA's fiscal year 2008
budget process, the annual budget process thereafter, and the
upcoming RD&T strategic plan:20
o a strategy for identifying and reviewing all of DOT's RD&T
projects to determine areas of unnecessary duplication,
overlap, and opportunities for joint efforts. The strategy
should address (1) the scope of the RD&T projects or programs
that will be reviewed for duplication or joint efforts, (2)
the methodology for how all RD&T projects will be reviewed or
how duplication or joint efforts will beidentified, (3) a
timeline and the frequency for reviews to occur, and (4) how
the results of the reviews-the identification of duplication
or an opportunity for joint effort-will be reported and used
to make decisions about future RD&T activities.
o a strategy to ensure that the results of all of DOT's RD&T
activities are evaluated according to established best
practices. This strategy should include (1) which RD&T
activities of the operating administrations RITA will ensure
were evaluated according to best practices, (2) the
methodology for how RITA will ensure evaluation of RD&T
activities took place according to established best practices,
(3) a timeline for when the RD&T evaluations should occur, and
(4) how the results of the RD&T evaluations will inform future
research.
o a DOT-wide database of all of DOT's RD&T projects that will
support RITA's coordination, facilitation, and review efforts
and will assist in the implementation of the strategies
discussed above. Information on the status of these efforts
should be included in the upcoming RD&T strategic plan to be
issued in the fall of 2006.
o a summary of all of DOT's RD&T program evaluations conducted
by the department for the past 3 years, including ongoing and
completed evaluations, and a schedule of future evaluations.
20The following five recommendations reiterate and expand on four of the
five recommendations GAO made to RSPA in 2003, with the exception of the
recommendation on quantifiable performance measures for RD&T coordination,
which DOT did not concur with, as previously discussed.
Page 13 GAO-06-917 Research and Innovative Technology Administration
o a description of RITA's process for systematically evaluating the
results of its own multimodal research programs and how this process will
be applied to future multimodal research programs that RITA conducts.
To help ensure that BTS's data products meet the needs of its users, we
recommend that the Secretary direct the RITA Administrator and BTS
Director to take the following action:
o Develop and implement a systematic process for BTS to identify its
primary users, solicit and incorporate feedback from those users, and
measure the satisfaction of its users. This process should contain the
following elements: (1) that primary users of BTS's data products and
services are identified and documented in a comprehensive manner; (2) that
feedback on user satisfaction is solicited on a periodic basis from those
users; (3) that user feedback is documented and evaluated at BTS's
agencywide level and against established criteria, to ensure consistency
in decisions about what improvements should be made to data products; and
(4) that performance indicators that measure data users' satisfaction are
developed and applied.
Agency Comments
We obtained oral comments on a draft of this report from DOT officials,
who generally agreed with our findings and recommendations. These
officials also provided technical clarifications that we incorporated into
the report, as appropriate.
We are sending copies of this report to the appropriate congressional
committees and to the Secretary and other appropriate officials in the
Department of Transportation. We will also make copies available to others
upon request. In addition, the report will be available at no charge on
the GAO Web site at http://www.gao.gov.
If you or your staff have any questions regarding this report, please
contact me at (202) 512-2834 or [email protected]. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on the
last page of this report. GAO staff who made major contributions to this
report are listed in appendix II.
Mathew Scire Acting Director, Physical Infrastructure Issues
Appendix I: Review of the Department of Transportation's Research and
Innovative Technology Administration (RITA)
Briefing for the Committees on Appropriations
U.S. Senate and House of Representatives May 30, 2006
Briefing Overview
o Introduction and Objectives
o Approach
o Background
o Results of GAO Work
Introduction
o In fiscal year 2005, the Department of Transportation's (DOT)
research, development, and technology (RD&T)1 budget totaled
approximately $1.1 billion. RD&T activities are vital to meeting DOT's
key transportation priorities, including safety and mobility.
o Prior to 2005, DOT's Research and Special Program Administration
(RSPA) was responsible for overseeing DOT's RD&T activities, the
Office of Hazardous Materials Safety, and the Office of Pipeline
Safety.
o GAO and others have raised concerns about RSPA-for example, GAO made
several recommendations in a 2003 report to improve RSPA's
coordination and evaluation of DOT's RD&T activities.
* In November 2004, the Norman Y. Mineta Research and Special
Programs Improvement Act (P.L. 108-426) dissolved RSPA and
created two new administrations:
o the Research and Innovative Technology Administration
(RITA). The Mineta Act transferred the Bureau of
Transportation Statistics (BTS) and the Office of
Intermodalism to RITA.
o the Pipeline and Hazardous Materials Safety Administration,
which includes the Office of Pipeline Safety and the
hazardous materials safety activities that were formerly in
RSPA.
1 OMB Circular No. A-11 (2005) defines research and development activities
to include "work undertaken on a systematic basis ... to increase the
stock of 3 knowledge, including knowledge of man, culture and society, and
the use of this stock of knowledge to devise new applications." DOT
defines technology to include demonstration projects and other related
activities associated with research and development activities.
Objectives
o In response to a mandate from the Senate and House Appropriations
Committees, GAO addressed the following questions:
1. How, if at all, do RITA's responsibilities for overseeing DOT's RD&T
activities differ from RSPA's?
2. What are RITA's practices for coordinating, facilitating, and
reviewing RD&T activities and what progress has RITA made in
implementing GAO's 2003 recommendations?
3. How does BTS identify its users and monitor how well it is serving
those users?
Overall Approach
o Legislative & Document Review -Reviewed laws and DOT documentation to
understand RSPA's and RITA's responsibilities and BTS' legislative
requirements for producing certain data.
o Interviews -Interviewed DOT officials in: RITA, including the RD&T
Office, BTS, and others; three operating administrations that conduct
and manage significant research programs, including the Federal
Highway Administration (FHWA), Federal Aviation Administration (FAA),
and Federal Transit Administration (FTA); and the Office of the
Secretary (OST).
o Recommendation Follow-up -Reviewed the extent to which RITA (and RSPA)
have addressed the recommendations in GAO's 2003 report.2
o Analysis of selected Bureau of Transportation Statistics' (BTS)
processes -Analyzed BTS' processes for identifying its data users and
for monitoring and incorporating feedback on user satisfaction and
compared those processes to relevant criteria.
o We conducted our work from November 2005 through May 2006 according to
generally accepted government auditing standards.
2 GAO, Transportation Research: Actions Needed to Improve Coordination and
Evaluation of Research, GAO-03-500 (Washington, D.C.: May 1, 2003).
Results in Brief
o RITA took over RSPA's responsibilities for overseeing DOT's RD&T
activities. While RITA continues to have a similar mission and
strategic objectives as RSPA, it still does not have the authority to
direct changes in the operating administrations' RD&T activities. RITA
differs from RSPA in several ways-RITA intends to focus significantly
more resources in fiscal year 2007 on the oversight of DOT's RD&T
activities; RITA's authority for evaluation is less clearly defined;
and RITA's organizational structure is more multi-modally focused.
o RITA coordinates, facilitates, and reviews DOT's RD&T activities
through multiple groups and practices, including its two coordinating
bodies-the RD&T Planning Council and the RD&T Planning Team-and budget
reviews, among others. Of the five recommendations GAO made in 2003 on
how to improve the coordination and evaluation of RD&T activities,
RITA has partially implemented four of the recommendations and has not
implemented the other.
o BTS does not have a systematic process in place for identifying its
users, soliciting ongoing feedback from all its major users, and
determining whether or how that feedback should be incorporated.
Background: RITA's legislative responsibilities
The Norman Y. Mineta Act lays out five broad responsibilities for RITA:
o Coordination, facilitation, and review of DOT's research and
development programs and activities;
o Comprehensive transportation statistics research, analysis, and
reporting;
o Advancement, and research and development, of innovative technologies,
including intelligent transportation systems;
o Education and training in transportation and transportation-related
fields; and
o Activities of the Volpe National Transportation Center, which conducts
a range of transportation research and development projects on a
fee-for-service basis.
Background: Scope of RITA's authority
o RITA's Office of RD&T is responsible for "coordinating, facilitating,
and reviewing" DOT's RD&T activities.
* According to the Norman Y. Mineta Act, RITA's scope of authority
includes coordinating, facilitating, and reviewing all of the
Department's RD&T programs and activities, with one
exemption-RITA was not given any authority over the research and
other programs, activities, standards, or regulations
administered by the National Highway Traffic Safety
Administration (NHTSA).
o The exemption does not apply to NHTSA's activities already
in effect on November 30, 2004, the date of the Mineta Act's
enactment.
o Officials in RITA's Office of RD&T told us that in spite of
the NHTSA exemption in the Mineta Act, NHTSA voluntarily
participates in all of RITA's coordination, facilitation,
and review activities.
Background: BTS' mission and legislative requirements
o The mission of BTS-a principal federal statistical agency established
in 1992-is to develop high quality transportation data and information
and advance the use of this data and information in both public and
private transportation decision-making.
* BTS' legislative requirements in the Safe, Accountable, Flexible,
Efficient Transportation Equity Act: A Legacy for Users
(SAFETEA-LU) (P.L. 109-59) include, among others:
o Coordinating collection of information with the operating
administrations within DOT and with other federal agencies;
o Issuing guidelines for the collection of statistical
information by DOT;
o Reviewing and reporting on the sources and reliability of
the statistics proposed by the operating administrations as
requested by the Secretary; and
o Submitting to the President and Congress a Transportation
Statistics Annual Report (TSAR).
Objective 1: Similarities between RITA and RSPA
o RITA, like RSPA before it, has the responsibility to oversee DOT's
RD&T activities but does not have the authority to direct changes in
the operating administrations' RD&T activities, including budgetary
changes.
o RITA's mission, like RSPA's before it, is focused on using research to
meet DOT's goals, including enhancing transportation safety and
efficiency.
o Two of RITA's three strategic goals--providing research and analytical
capabilities dedicated to furthering DOT's mission and promoting
multimodal, cross-cutting transportation technologies and innovations-
are similar to RSPA's goals. RITA's third strategic goal is to ensure
the effectiveness of the public investment in transportation RD&T.
Differences between RITA and RSPA: RITA intends to focus significantly
more resources on its oversight of DOT's RD&T activities
o RITA's fiscal year 2007 budget request proposes a new program-the
Transportation Futures and Applied Technology Program-to undertake
certain oversight and coordination activities, and also continues
RITA's existing RD&T Coordination program. If enacted, funding for
these two programs would result in an overall increase of about $2
million over the enacted fiscal year 2006 budget for RD&T
Coordination-from $536,000 in fiscal year 2006 to about $2.5 million
in fiscal year 2007-although the Transportation Futures and Applied
Technology Program also funds activities other than coordination.3
o According to senior DOT officials in OST and RITA, although the
Transportation Futures and Applied Technology Program and the RD&T
Coordination program were listed as separate programs in DOT's 2007
budget request, DOT views them as one unified program, with the intent
that the Transportation Futures and Applied Technology Program will
build on and enhance the existing RD&T Coordination program. A
significant element of the funding request for the Transportation
Futures and Applied Technology Program is to enable RITA to have
access to highly skilled, specialized technical experts that RITA
views as critical to enhancing its ability to effectively coordinate,
facilitate, and review DOT's RD&T activities and enable RITA to more
fully participate in strategic planning, priority setting, and
decision-making related to those activities. Further, both programs
are requesting funding to support intermodal and interagency teams
that are working to better leverage research results and advance the
collaborative integration and deployment of new technologies.
3 These other activities include conducting technology scans and
accelerating the deployment of technologies and innovations to users.
Differences between RITA and RSPA: RITA intends to focus significantly
more resources on its oversight of DOT's RD&T activities, continued
o The presentation of RITA's fiscal year 2007 budget request does not
provide sufficient detail to determine the total amounts of funding
that RITA is requesting for the two programs since the budget only
shows the amount of proposed contract dollars for the two programs,
including $2.2 million for the Transportation Futures and Applied
Technology Program and $247,000 for the RD&T Coordination program. The
2007 budget request also shows that RITA is requesting about $5.2
million for salaries and administrative expenses for all research and
development activities, but it does not specify what portion of that
amount will be dedicated to the two programs.
* The amount that RITA requested for its RD&T Coordination Program
in fiscal year 2007 ($247,000) is less than half of what was
enacted for those activities in fiscal year 2006 ($536,000).
According to the 2007 budget request, this decrease represents
the transfer of cross-modal RD&T initiatives from the RD&T
Coordination program to the Transportation Futures and Applied
Technology Program. DOT's documentation does not provide a clear
cross-walk showing the transfer of all the cross-modal
initiatives from one program to the other. However, a senior
official in OST told us that DOT plans to review all of its
crossmodal initiatives as part of the Transportation Futures and
Applied Technology Program for the purpose of re-focusing them to
ensure they are addressing the most important DOT-wide research
priorities.
* Differences between RITA and RSPA: RITA's authority for
evaluation is less clearly defined
o RSPA had both legislative and departmental responsibility for
evaluating DOT's RD&T programs; specifically RSPA was required to:
o measure the results of federal transportation research activities and
how those activities impact the performance of the surface transportation
systems of the United States4and
o oversee DOT's RD&T programs and develop more efficient, effective, and
participative ways to evaluate and measure program effectiveness and
progress across all operating administrations and other selected
projects.5
o RITA does not have the same legislative responsibility for evaluating
DOT's RD&T programs because SAFETEA-LU removed the statutory
responsibility for evaluation activities that had been specifically
outlined for DOT, which DOT delegated to RSPA.
o RITA officials told us that their interpretation of "review" authority
under the Norman Y. Mineta Act is that RITA is not required to
evaluate RD&T activities to determine whether they are achieving
intended goals, although RITA is not explicitly prevented from
evaluating such activities.
4 As stated in the Transportation Equity Act for the 21st Century
(TEA-21), formerly codified in 23 USC 508. 15 5 DOT policy delegated this
responsibility to RSPA.
Objective 2: RITA's Coordination, Facilitation, and Review Practices
RITA's Office of RD&T coordinates, facilitates, and reviews DOT's RD&T
activities through various groups/practices, as summarized below:
Group / Process Function Participants
RD&T Planning Advises the DOT Secretary on RD&T Operating
Council policies and priorities necessary to Administrators,
support the DOT
Strategic Plan and Administration and the
Secretarial policy and objectives. Undersecretary
Ensures cross-modal for
collaboration and coordination in RD&T Policy, OST;
initiatives within DOT and with external Chair, RITA
entities. Reports
to the Secretary. Administrator
RD&T Planning Assists the RD&T Planning Council and the Associate
Team RITA Administrator in ensuring Operating
cross-modal
collaboration and coordination in DOT's Administrators
RD&T initiatives. Assures the application for RD&T,
of
Administration and Secretarial RD&T OST; Chair,
priorities and policies and guidance from RITA's
the RD&T
Planning Council. Associate
Administrator
for RD&T
Conducts the annual review of modal RD&T Operating
Program Review programs to ensure they are aligned with Administrations'
DOT
Working Group strategic goals and implement the Office RD&T managers
of Management and Budget's (OMB) Research
and
Development (R&D) investment criteria. 8
Reports to RD&T Planning Team.
RITA's annual assessment of the operating
Budget Review administrations' RD&T budget-first RITA, Operating
performed on
fiscal year 2007 budget submissions--to Administrations,
ensure alignment with DOT's strategic OST's
objectives,
Secretarial priorities, and cross-modal Budget Office
initiatives and proper application of
OMB's R&D
Investment Criteria.
Advances U.S. technology and expertise in
University transportation through the mechanisms of RITA, Operating
education,
Transportation research and technology transfer at Administrations
Centers university-based centers of excellence.
RITA administers
Program grants (mostly legislatively designated)
to universities.
Peer Review Coordinates the implementation of OMB's
Task Information Quality Bulletin on Peer RITA, Operating
Review for
Force and influential and highly influential Administrations'
Steering research (as determined by the agency or RD&T
OMB) and oversees
Committee compliance with these requirements. managers, Data
Prepares annual report to OMB. Quality
experts.
RD&T Strategic Develops the details of the RD&T Strategic Plan due to
Congress in the fall of 2006. The draft RITA, Operating Planning Task
Force plan will be reviewed by the National Research Council in June 2006.
Administrations' RD&T
managers Various multi-modal Examples include: RITA, Operating RD&T groups
managed x Hydrogen Working Group which coordinates, facilitates and
informs all RD&T activities Administrations' policy by RITA related to
hydrogen fueled and fuel cell powered vehicles and stationary
applications. and technical staff and
x Center for Climate Change which shares information, builds partnerships,
and coordinates other representatives activities related to climate
change. Source: GAO analysis of RITA information.
8 OMB developed the R&D Investment Criteria-quality, relevance, and
performance-for all types of R&D programs across all agencies to assist 16
in budget allocation decisions and to demonstrate results.
RITA's Coordination, Facilitation, and Review Practices, continued
* RD&T Planning Council
o According to DOT Order 1120.39A, the Planning Council is to
meet quarterly. The Planning Council has convened three times,
most recently on May 15, 2006. RITA officials told us the
Council expects to meet more regularly as it reviews the
development of the draft RD&T Strategic Plan before it is
submitted to Congress-several meetings have been scheduled for
the summer of 2006.
o According to the Planning Council's meeting minutes from May
and June, 2005, the Planning Council reviewed and approved the
RD&T budget guidance that was distributed to the operating
administrations and discussed future strategic directions for
DOT research, the fiscal year 2007 RD&T budget submissions,
and other topics.
* RD&T Planning Team
o According to DOT Order 1120.39A, the Planning Team is to meet
quarterly, which it has done. The Planning Team has convened
four times, most recently in February 2006, and another
meeting is scheduled for May 24, 2006.
o According to the Planning Team's meeting minutes from June and
November, 2005 and February, 2006, the Planning Team discussed
results of RITA's budget review for fiscal year 2007, effects
of SAFETEA-LU on research programs, and updates on the
development of the RD&T strategic plan and the University
Transportations Centers program.
RITA's Coordination, Facilitation, and Review Practices, continued
* Program Review Working Group:
* o Established through a DOT Order that allowed the RD&T Planning
Team to create ad hoc working groups, this group met nine times
during 2005 and 2006 (as of April). The group provides a forum for
the operating administrations to share information about areas of
research and identify opportunities for coordination. The group
also schedules presentations throughout the year which it shares
with the RD&T Planning Team. Past presentations have addressed how
the operating administrations have applied OMB's R&D Investment
Criteria to their RD&T activities and how the operating
administrations' RD&T activities support DOT's and the Secretary's
strategic priorities.
* Budget Review
o In preparation for their first budget review, RITA officials
developed the 2007 RD&T budget guidance for the operating
administrations, advising them to include discussions of how
their RD&T activities are linked to the Secretary's priorities
and how they have applied OMB's R&D Investment Criteria. This
guidance was reviewed by the Planning Team, approved by the
Planning Council, and distributed by OST's Budget Office.
o The results of RITA's budget review were discussed at the June
2005 Planning Council meeting. Also, RITA developed
recommendations-to DOT's Assistant Secretary for Budget and
Programs/Chief Financial Officer-that RITA continue to work
with the operating administrations to improve the Department's
RD&T budget submissions and that RITA develop guidance for
defining RD&T to ensure consistency in the way that the
Department classifies and accounts for its RD&T programs.
RITA's Coordination, Facilitation, and Review Practices, continued
o Management of the University Transportation Centers (UTC) Program9
* RITA's RD&T Office manages the UTC program, which was significantly
expanded by SAFETEALU.
o SAFETEA-LU authorized up to about $76.7 million per year for
UTC grants during the period from fiscal year 2005 to 2009, an
increase of about 136 percent compared to the $32.5 million
authorized under TEA-21.
o Under SAFETEA-LU, the UTC grants provide funding to establish
and operate up to 60 UTCs, an increase of about 82 percent
compared to the 33 centers authorized under TEA-21. Twenty of
the 60 centers are scheduled for competitive selection during
2006, and 40 centers are located at institutions named in the
legislation.
o To foster a closer connection between UTC research and needs of the
operating administrations and DOT, RITA hosted a two-day workshop in
April 2006 where officials from operating administrations made
presentations on their RD&T activities and DOT's priorities to UTC
members. RITA officials noted that this is the first of a planned
series of program meetings, technical workshops, and other outreach
efforts to increase UTC connections to DOT's RD&T priorities and
programs. RITA is also working with the operating administrations to
increase technical oversight of the individual UTCs' selection and
evaluation processes for research.
9 The UTC program was initiated in 1987 under the Surface Transportation
and Uniform Relocation Assistance Act, which authorized the
19
establishment and operation of transportation centers in each of the ten
federal regions. The Intermodal Surface Transportation Efficiency Act of
1991 (ISTEA) reauthorized the UTC program for an additional six years and
added four national centers and six University Research Institutes.
Effects of RITA's coordination, facilitation, and review activities
* RITA has not established performance goals for its coordination,
facilitation, and review activities, nor does RITA monitor or
evaluate the effects of those activities. Also, RITA has not worked
with the operating administrations to develop common performance
measures for the Department's RD&T activities.
o Setting meaningful goals for performance, and using
performance information to measure performance against those
goals, is required by the Government Performance and Results
Act of 1993.10 GAO has also reported on key practices-such as
defining common outcomes and developing mechanisms to monitor,
evaluate, and report on results-that can enhance and sustain
collaboration among organizations that conduct cross-cutting
activities.11
o Establishing performance goals and an evaluation plan for its
own coordination, facilitation, and review activities could
assist RITA in identifying any problem areas and better
allocating its resources to improve performance. Without such
goals and an evaluation plan, it is difficult for RITA to
determine its success in overseeing and ensuring the
effectiveness of DOT's RD&T activities. Also, without common
performance measures for the RD&T activities of the operating
administrations, RITA and the operating administrations lack
the means to monitor and evaluate the collective results of
those activities and ensure that they are achieving their
intended (or other) results and furthering the Secretary's and
DOT-wide priorities.
o The RITA Administrator told us that he was open to developing
performance goals and an evaluation plan for RITA's coordination,
facilitation, and review activities and to working with the operating
administrations to develop common performance measures for DOT's RD&T
activities.
10 Pub. L. No. 103-62, 107 Stat. 285 (1993). Under GPRA, federal agencies
are required to develop strategic plans, performance plans, and
performance reports that set long-term and annual goals along with the
means for accomplishing the goals and report on achieving them. 20
11 GAO, Results-Oriented Government: Practices That Can Help Enhance and
Sustain Collaboration among Federal Agencies, GAO-06-15 (Washington, D.C.:
October 21, 2005).
Effects of RITA's coordination, facilitation, and review activities,
continued
o Although RITA lacks performance goals and measures, officials from the
three operating administrations we interviewed and from RITA noted some
positive effects of RITA's efforts:
o Officials from all three operating administrations told us that RITA's
Program Review Working Group meetings provide them with opportunities
to share best practices relating to program management issues, such as
how to apply the OMB R&D Investment Criteria to RD&T activities. An
official from one of the operating administrations said that RITA has
been helpful in trying to strengthen the connection between the UTCs
and the DOT's RD&T priorities through programs such as the 2-day UTC
workshop hosted in April 2006. Officials from another operating
administration told us that RITA has helped them to better align their
RD&T activities with DOT's and the Secretary's priorities by focusing
on how their research activities will support DOT's RD&T Strategic
Plan.
o Officials from RITA's Office of RD&T believe that RITA's budget
guidance and review process is helping to ensure that the operating
administrations have a more consistent approach to show how their RD&T
activities support DOT's strategic objectives, Secretarial priorities,
and crossmodal initiatives. One official noted that, over time, this
process should give RITA the ability to critically review and make
recommendations to OST on how RD&T activities could better support
DOT's strategic objectives, Secretarial priorities, and cross-modal
initiatives. Also, the Program Review Working Group has identified an
opportunity for FHWA and FAA to coordinate pavement research.
Follow-up to GAO Recommendations
o In 2003, GAO made five recommendations to DOT and RSPA to improve the
coordination and evaluation of RD&T activities.
o The recommendations generally remain relevant for RITA.
o RITA has partially implemented four recommendations and has not
implemented one recommendation. RITA has made progress, but much
remains to be done for RITA to fully implement the recommendations.
Recommendation 1
Develop a strategy for reviewing all DOT research projects to identify
areas of unnecessary research duplication overlap, and opportunities for
joint efforts.
o Include time frames for implementing this review and discuss the
development and implementation of a DOT-wide research tracking system
database.
o Incorporate the results of this effort into DOT's annual research plan
and report to Congress on an annual basis.
Status: Partially Implemented
o RITA has not developed a strategy to review all DOT research projects
to identify areas of unnecessary research duplication overlap and
opportunities for joint efforts, although they have taken some action,
as discussed below. RITA officials told us that they expect to
identify such areas through the coordination and review activities of
the RD&T Planning Council, RD&T Planning Team, and the Program Review
Working Group. However, although the DOT Order that created the RD&T
Planning Council, Planning Team, and indirectly the Program Review
Working Group, lays out as part of its purpose the responsibility to
prevent unnecessary duplication of RD&T efforts in DOT, none of these
groups has developed a strategy that describes (1) the scope of the
RD&T projects or programs that will be reviewed for duplication or
joint efforts, (2) the methodology for how all research projects will
be reviewed or how duplication or joint efforts will be identified,
(3) a time line and frequency for the review to occur, and (4) how the
results of the review-the identification of duplication or an
opportunity for joint effort-will be used to make decisions about
future RD&T activities.
o A RITA official told us that the research tracking system database was
dropped from DOT's priorities after the creation of RITA; however,
RITA's proposed Transportation Futures and Applied Technology Program
includes the development of a Web-based database for DOT's RD&T
programs. RITA has not yet established a catalog of all the research
activities within DOT. Without such a catalog, DOT officials do not
have readily accessible data on research activities throughout DOT.
Recommendation 1
Develop a strategy for reviewing all DOT research projects to identify
areas of unnecessary research duplication overlap, and opportunities for
joint efforts.
o Include time frames for implementing this review and discuss the
development and implementation of a DOT-wide research tracking system
database.
o Incorporate the results of this effort into DOT's annual research plan
and report to Congress on an annual basis.
Status: Partially Implemented
o While the results of RITA's coordination and review activities are
discussed during Planning Team meetings, the results are not reported
to Congress because RITA is no longer required to submit an annual
RD&T plan to Congress. RITA officials told us that they plan to report
these results annually in the DOT Performance Plans and Reports
required by SAFETEA-LU. This reporting is expected to begin in Fiscal
Year 2008, as the first reporting against the new RD&T Strategic Plan.
o Without such a strategy-supported by a comprehensive database of
ongoing RD&T projects-RITA is unable to ensure that areas of
unnecessary research duplication, overlap, and opportunities for joint
efforts are systematically identified and managed.
Recommendation 2
Develop and apply quantifiable performance measures to assess the
effectiveness of research coordination efforts and document the results of
these efforts in DOT's annual research plan.
Status: Not Implemented
o DOT and RSPA did not concur with this recommendation in 2003, citing
their views that: (1) the most useful and effective performance
measures focus on results, while coordination is a process; (2) there
are no performance measures capable of quantifying effective
coordination; and (3) existing coordination processes have been
effective in preventing unnecessary research duplication.
o RITA officials also told us that they did not develop performance
measures to assess the effectiveness of research coordination effort
because they do not believe that a metric approach is well suited for
assessing the effects of the coordination process.
o While RITA officials told us that they have not developed or applied
performance measures for coordination, they indicated their
willingness to develop--in concert with the operating
administrations-common performance measures for DOT's RD&T activities.
12 GAO, Highway Research: Systematic Selection and Evaluation Processes
Needed for Research Program, GAO-02-573 (Washington, D.C.: May 24, 2002).
Peer review includes an independent assessment of technical and scientific
merit or quality and is 26 considered appropriate for all RD&T programs.
Recommendation 4
Include in DOT's annual research plan a summary of all research program
evaluations conducted and a schedule of future evaluations.
Status: Partially Implemented
o RSPA, for one year, published a summary of all research program
evaluations conducted, in that it published the results of its review
of the operating administrations' application of OMB's R&D Investment
Criteria in its fiscal year 2005 annual RD&T plan. A schedule of
future evaluations was not included in this plan.
o Since RSPA was dissolved, RITA has not continued to publish the
results of its reviews of the operating administrations' application
of OMB's R&D Investment Criteria because SAFETEA-LU no longer requires
RITA to submit an annual research plan.
o Publishing a current inventory of the evaluation of research
activities and a schedule for future evaluations on a regular basis
could provide continuity and context for the observations about
research results and planned research for future years.
Recommendation 5
Document RSPA's process for systematically evaluating the results of its
own multi-modal research programs, and apply this process to any future
multimodal research programs that RSPA conducts.
Status: Partially Implemented
o RITA has not documented its process for systematically evaluating the
results of its own multi-modal research programs, or how this process
would apply to any future multi-modal research programs that it
conducts, although RITA has taken some action, described below.
o RITA manages multi-modal RD&T activities, such as the Hydrogen Safety
Program and various grant programs in SAFETEA-LU delegated to it by
DOT. According to a RITA official, RITA oversees contracts and
evaluates the results of RD&T activities that are conducted under
these contracts through the peer review process. RITA has not
documented its process for conducting these peer reviews and it is
unclear whether the peer review process will be applied to future
multi-modal RD&T activities.
o Without systematically evaluating program results, RITA is limited in
its ability to determine the extent to which its multi-modal RD&T
programs are achieving their intended (or other) goals.
Objective 3: BTS' process for identifying its users is not systematic
o BTS does not have a systematic process, but uses a variety of methods,
for identifying specific users of BTS information.
o BTS officials told us they primarily rely on a Congressional
definition of broad categories of data users, contained in SAFETEA-LU.
This definition states that users of BTS information include the
Federal government, State and local governments, metropolitan planning
organizations, transportation-related associations, the private sector
(including the freight community), and the public.
* BTS officials told us that they identify specific users through a
variety of methods such as:
o Conferences, focus groups, and training sessions;
o Meetings of professional associations;
o Direct inquiries via the BTS Web site, in the form of email,
letters, and telephone; and
o News media citations--tracked by RITA's Office of
Governmental, International and Public Affairs--that mention
BTS data.
o BTS does not track specific users nor does it have information on the
overall number of users of its data products. According to one BTS
official, the reason for this is because it is very difficult to track
individual users, other than through Web site hits and tracking the
number of publications ordered. Also, the official mentioned BTS is
limited in its ability to collect information on individual data users
due to privacy concerns.
x Without a systematic process for identifying primary users for each of
its programs, BTS cannot solicit feedback from these users on an ongoing
basis.
29
Examples of BTS data users
o Examples of users of BTS information
BTS Data Users Data product used and for what purpose
Department of x DOT's Office of the Secretary uses BTS airline data to
determine eligibility for the Essential Air Transportation Service (EAS)
program, monitor the impact of airline service on the traveling public,
negotiate international air service agreements, and provide oversight of
the airline industry. x The Federal Highway Administration uses Commodity
Flow Survey data in its Freight Analysis Framework, a departmental
planning tool.
x DOT's Federal Railroad Administration (FRA) uses data collected by BTS
for the Confidential Close Call Reporting System Demonstration Project to
help FRA and railroad carriers identify safety issues that require
corrective action.
Other Federal x BTS worked with the Commerce Department's Bureau of
Economic Analysis to develop a agencies Transportation Satellite Account,
which enhanced the measurement of transportation's contribution to the
Gross Domestic Product. x The Department of Homeland Security`s
Transportation Security Administration used BTS' Omnibus Survey Program
data on the amount of baggage brought to airports.
Policymakers x BTS is developing capital stock values (a commonly used
economic measure of the capacity of the transportation system) for
airports, waterways, and transit systems for use by policymakers to better
estimate the amount of investment needed to accommodate current or future
levels of traffic on those modes.
Planners x States use National Household Travel Survey (NHTS) data for
transit planning; Metropolitan Planning Organizations (MPO) use NHTS data
for highway planning.
Source: GAO analysis of BTS information.
BTS' methods for soliciting feedback on user satisfaction is not
systematic
* BTS' process for soliciting feedback on user satisfaction is not
systematic, but BTS officials reported using a variety of methods
for soliciting feedback from its users, including:
o Data users' meetings and workshops held at conferences or
training sessions, and focus groups conducted with BTS data
users;
o BTS' Reference Services, which tracks and responds to user
feedback; and
o Independent third party reviews of its programs.
o In the past, BTS systematically solicited feedback on overall customer
satisfaction of BTS' products and services through its Customer
Satisfaction Survey. BTS conducted the last such survey in 1998.
o Also, in the past, BTS solicited feedback via comment cards, inserted
in data products. According to BTS officials, the survey and comment
cards are no longer used as methods for soliciting feedback as a
result of restrictions stemming from the Privacy Act and Paperwork
Reduction Act (PRA). BTS officials told us they are working with
RITA's Chief Counsel's office to reinstitute the use of comment cards.
x Without a systematic process for soliciting feedback on user
satisfaction from its users, BTS cannot ensure that it is has a full
picture of the needs of those users and how well it is meeting those
needs, which, in turn, hinders BTS' ability to make improvements to data
programs that are relevant to users.
Examples of how BTS has solicited feedback from users
o Examples of how BTS has solicited feedback from users.
How BTS solicited user Examples of how feedback on user solicitation was
solicited for specific BTS feedback on user satisfaction programs
Data users' meetings and x In 2005, BTS engaged freight data users at a
Commodity Flow Survey Conference for workshops held at conferences input
on changes needed in BTS' freight data collection. or training sessions x
In 2004, BTS engaged users of travel data for their input into current
uses of existing
data and future data needs. The results of this data user's meeting were
published in
Data for Understanding Our Nation's Travel: National Household Travel
Survey Conference.
BTS Reference Services x BTS' National Transportation Library Reference
Services Desk tracks and responds to inquiries received by email via BTS'
Web site or by telephone.
Independent third party reviews x In 2003, BTS asked the Transportation
Research Board and Committee on National Statistics of the National
Academies to conduct a review of its major survey programs, including the
(1) National Household Travel Survey, (2) Omnibus Survey Program, and (3)
Commodity Flow Survey. x In 1999, BTS contracted with an external group to
conduct focus groups with congressional staffers, DOT employees,
transportation statistics experts, and employees of non-profits or
associations in the transportation field to (1) gain feedback on BTS'
publications and services and (2) gather insight about how to improve
these tools for the end user.
Source: GAO analysis of BTS information.
BTS' method for incorporating feedback on user satisfaction is not
systematic
o BTS relies on its program managers, who are responsible for managing a
data collection program and its associated data products, to determine
how best to address feedback from users in identifying areas of
improvement and taking steps to address product weaknesses.
o According to BTS officials, program managers use their professional
judgment and consider such factors as time and resource constraints
when deciding whether to incorporate feedback from users in taking
steps to improve their data products. BTS officials told us that
program managers are not required-and often do not-report the results
of how feedback on user satisfaction was considered and whether
improvements to the data product were made.
o GAO's Internal Control standards suggest that on-going monitoring
should take place to ensure that users are satisfied and that policies
and procedures should be in place to ensure that feedback is evaluated
so that improvements can be made.
o Without policies and procedures in place for incorporating feedback on
user satisfaction, BTS cannot systematically consider feedback and use
it to make improvements to data products.
BTS' performance measures
* BTS reports that it measures the overall degree to which products
and services are useful and responsive to the needs of users by
tracking two performance indicators-the number of congressional and
governmental agency contacts regarding BTS information and the
average number of daily unique visitors to the TranStats data Web
site:13
o BTS reports that the number of congressional and governmental
agency contacts has increased each year, from 84 requests in
2002, to 287 in 2005. BTS officials attribute this increase to
their responsiveness to inquiries and they told us that the
increase indicates that BTS is providing Congress and agencies
with information that they need, and that they find BTS to be
a reliable source.
o BTS reports the average number of daily unique visits to the
TranStats data site has increased from 565 in 2003, to 813 in
2005. According to BTS officials, this indicates that the
information is useful and that awareness among current and
potential new BTS data users is increasing.
o We note, however, that these indicators are simple counts and not
measures of user satisfaction with BTS information.
13 Required by SAFETEA-LU, the TranStats data web site contains hundreds
of transportation-related databases and is made available to the public
via BTS' Web site. 34
DOT's Perspective on Key Issues Related to RITA
o We met with senior DOT officials-including the Department's Chief of
Staff, the Assistant Secretary for Budget and Programs and Chief
Financial Officer, the RITA Administrator, and other officials-to
discuss the factual information presented in these slides. These
officials agreed with the information, providing clarifications, which
we incorporated as appropriate.
o These officials said that their vision is for RITA to play a key
role-in concert with the operating administrations-in shaping the
future of DOT's RD&T activities, although they believe that RITA will
need significant additional resources, especially technical experts,
to achieve this vision. They said that OST will rely heavily on input
from RITA's Planning Council in making its decisions related to the
RD&T budget requests from the operating administrations. They also
said that OST will look to RITA to play a lead role in helping realign
the Department's RD&T activities to focus more closely on key
cross-modal transportation issues facing the nation, including
congestion, alternative fuels, freight capacity, and other issues.
Next Steps
o Prepare final slides, with a short summary of results, for publication
as a GAO report.
o Determine if committee staff want any additional work on RITA.
Appendix II: GAO Contact and Staff Acknowledgments
GAO Contact
Mathew Scire, (202) 512-2834 or [email protected]
Staff Acknowledgments
In addition to the contact named above, key contributors to this report
were Rita Grieco, Assistant Director; Ashley Alley; Carl Barden; Christine
Bonham; Elizabeth Curda; Michelle Dresben; Colin Fallon;
Richard Hung; Sara Ann Moessbauer; Susan Ragland; April Thompson; Deborah
Winters; and Dorothy Yee.
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts GAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone:Voice:(202) 512-6000 TDD:(202) 512-2537 Fax:(202)
512-6061
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
RECYCLED PAPER
*** End of document. ***