Transportation Research: Opportunities for Improving the	 
Oversight of DOT's Research Programs and User Satisfaction with  
Transportation Statistics (15-AUG-06, GAO-06-917).		 
                                                                 
The Department of Transportation's (DOT) research, development,  
and technology (RD&T) budget totaled $1.1 billion in fiscal year 
2005. DOT's Research and Innovative Technology Administration	 
(RITA)--which includes the Bureau of Transportation Statistics	 
(BTS)--oversees DOT's RD&T activities. GAO examined (1) how	 
RITA's responsibilities for overseeing DOT's RD&T activities	 
differ from those of its predecessor, the Research and Special	 
Programs Administration (RSPA); (2) RITA's practices for	 
coordinating, facilitating, and reviewing RD&T activities; (3)	 
the progress DOT has made in implementing GAO's 2003		 
recommendations on how to improve the coordination and evaluation
of RD&T activities; and (4) how BTS identifies and monitors how  
well it serves its users. To address these issues, GAO reviewed  
relevant documentation and interviewed officials from RITA, BTS, 
and three operating administrations.				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-06-917 					        
    ACCNO:   A58681						        
  TITLE:     Transportation Research: Opportunities for Improving the 
Oversight of DOT's Research Programs and User Satisfaction with  
Transportation Statistics					 
     DATE:   08/15/2006 
  SUBJECT:   Agency missions					 
	     Program evaluation 				 
	     Program management 				 
	     Research and development				 
	     Research program management			 
	     Systems evaluation 				 
	     Transportation					 
	     Transportation research				 
	     Transportation statistics				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-06-917

                 United States Government Accountability Office

GAO

                                  August 2006

TRANSPORTATION RESEARCH

 Opportunities for Improving the Oversight of DOT's Research Programs and User
                  Satisfaction with Transportation Statistics

GAO-06-917

TRANSPORTATION RESEARCH

Opportunities for Improving the Oversight of DOT's Research Programs and
User Satisfaction with Transportation Statistics

  What GAO Found

In 2005, RITA took over RSPA's responsibilities for overseeing DOT's RD&T
activities when RSPA was dissolved. While RITA's mission and strategic
objectives are similar to those RSPA had, RITA differs from RSPA in a
number of ways. For example, RITA proposed a $2 million increase in its
fiscal year 2007 budget request for the oversight of DOT's RD&T activities
through its proposed Transportation Futures and Applied Technology
Program, which, among other things, would provide access to technical
experts to RITA on a contract basis. Additionally, RITA's responsibility
for evaluation is less clearly defined than RSPA's. RITA, unlike RSPA, is
not required to measure the results or evaluate the effectiveness of RD&T
activities. However, RITA is not explicitly prevented from evaluating such
activities.

RITA coordinates, facilitates, and reviews DOT's RD&T activities through
various practices. For example, RITA has two coordinating bodies-the RD&T
Planning Council and the RD&T Planning Team-and conducts budget reviews,
among other practices. RITA has not, however, established performance
goals, a clear implementing strategy, or an evaluation plan that
delineates how its coordination, facilitation, and review practices will
further DOT's mission or ensure the effectiveness ofits RD&T investment.
Without such a strategic approach, it is difficult for RITA to ensure that
DOT is making the most of its approximately $1 billion annual RD&T
investment.

RITA has partially implemented four of our recommendations and has not
implemented the other. For example, while RITA, through its two
coordinating bodies, has taken some action to review RD&T activities for
duplication and opportunities for joint efforts, RITA has not established
the scope of RD&T activities to be reviewed, the methodology of the
review, or how the results will be used to make decisions about future
RD&T activities.

BTS does not have a systematic process for identifying its primary users,
soliciting ongoing feedback from those users, and determining whether or
how that feedback should be incorporated. For example, rather than
identify specific users of BTS data products and services, BTS considers
its users to be those broad categories of intended users describedin
federal legislation. Further, rather than routinely soliciting user
feedback on all data products and services, such as through a customer
satisfaction survey, BTS only solicits limited feedback from some users of
specific products through conferences, workshops, or other meetings.
Finally, BTS relies onits program managers to evaluate and determine how
best to address feedback from its users; however, the managers are not
required to-and often do not-report the results of whether or how they
considered user feedback. Without a systematic process for identifying its
users, soliciting ongoing feedback, and determining whether or how that
feedback should be incorporated, BTS is limitedin its ability to consider
feedback and use it to make improvements to data products.

United States Government Accountability Office

Contents

Letter 1
Results Summary 4
Conclusions 11
Recommendations for Executive Action 12
Agency Comments 14
: Status of RITA's Implementation of GAO's 2003
Recommendations

Appendix I Review of DOT's Research and Innovative Technology
Administration

Appendix II GAO Contact and Staff Acknowledgments

Table

Abbreviations

BTS           Bureau of Transportation Statistics                          
DOT           Department of Transportation                                 
EAS           Essential Air Service                                        
FAA           Federal Aviation Administration                              
FHWA          Federal Highway Administration                               
FRA           Federal Railroad Administration                              
FTA           Federal Transit Administration                               
GPRA          Government Performance and Results Act of 1993               
MPO           Metropolitan Planning Organization                           
NHTS          National Household Travel Survey                             
NHTSA         National Highway Traffic Safety Administration               
OMB           Office of Management and Budget                              
OST           Office of the Secretary of Transportation                    
PHMSA         Pipeline and Hazardous Materials Safety Administration       
PRA           Paperwork Reduction Act                                      
R&D           research and development                                     
RD&T          research, development, and technology                        
RITA          Research and Innovative Technology Administration            
RSPA          Research and Special Programs Administration                 

SAFETEA-LU Safe, Accountable, Flexible, Efficient Transportation Equity

Act: A Legacy for Users TEA-21 Transportation Equity Act for the 21st
Century TSAR Transportation Statistics Annual Report UTC University
Transportation Centers

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributedin
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office Washington, DC 20548

August 15, 2006

The Honorable Christopher Bond Chairman The Honorable Patty Murray Ranking
Minority Member Subcommittee on Transportation, Treasury, the Judiciary,
Housing and

Urban Development, and Related Agencies Committee on Appropriations United
States Senate

The Honorable Joe Knollenberg Chairman The Honorable John W. Olver Ranking
Minority Member Subcommittee on Transportation, Treasury, Housing and
Urban

Development, the Judiciary, the District of Columbia, and Independent

Agencies Committee on Appropriations House of Representatives

In fiscal year 2005, the Department of Transportation's (DOT) research,
development, and technology1 (RD&T) budget totaled approximately $1.1
billion, including projects undertaken by DOT's operating administrations,
such as the Federal Highway Administration and the Federal Aviation
Administration. RD&T activities are vital to meeting DOT's key
transportation priorities, such as increasing transportation safety and
enhancing mobility for all Americans. Prior to 2005, DOT's Research and
Special Programs Administration (RSPA) was responsible for overseeing
DOT's RD&T activities, the Office of Pipeline Safety, and the Office of
Hazardous Materials Safety. Over the years, GAO and others have raised
concerns about RSPA's capabilities for improving RD&T coordination and

1OMB Circular No. A-11 (2005) defines research and development activities
to include "work undertaken on a systematic basis ... to increase the
stock of knowledge, including knowledge of man, culture and society, and
the use of this stock of knowledge to devise new applications." DOT
defines technology to include demonstration projects and other related
activities associated with research and development activities.

    Page 1 GAO-06-917 Research and Innovative Technology Administration

evaluation across DOT, and, in 2003, GAO made several recommendations to
improve those efforts.2

In response to these concerns and to help delineate and clarify roles and
responsibilities for overseeing DOT's RD&T activities, the Norman Y.
Mineta Research and Special Programs Improvement Act of 2004 (Pub. L. No.
108-426) dissolved RSPA and created two new administrations-the Pipeline
and Hazardous Materials Safety Administration3 (PHMSA) and the Research
and Innovative Technology Administration (RITA). RITA was created to
provide DOT with a more focused research organization and to assist in
avoiding DOT-wide research duplication and inefficiency, among other
purposes.4 The act also transferred the Bureau of Transportation
Statistics (BTS) and the Office of Intermodalism to RITA. Through this
act, RITA is responsible for coordinating, facilitating, and reviewing
DOT's RD&T programs and activities, which include the activities conducted
by DOT's operating administrations as well as other RD&T and statistical
programs managed by RITA (e.g., BTS, the Office of Intermodalism, and
University Transportation Centers).

In Senate Report 109-109, accompanying the DOT and related agencies
appropriation bill for fiscal year 2006, the House and Senate Committees
on Appropriations directed GAO to assess how RITA is resolving the
concerns of its predecessor administration and better coordinating DOT's
RD&T and statistical activities. Specifically, this report discusses (1)
how RITA's responsibilities for overseeing DOT's RD&T activities differ
from RSPA's; (2) RITA's practices for coordinating, facilitating, and
reviewing RD&T activities; (3) the progress RITA has made in implementing
GAO's 2003 recommendations; and (4) how BTS identifies its users and
monitors how well it is serving those users.

To determine how RITA's oversight responsibilities differ from RSPA's, we
collected information through legislative histories, document reviews, and
interviews with officials within RITA and used it to compare RITA and

2

GAO, Transportation Research: Actions Needed to Improve Coordination and
Evaluation of Research, GAO-03-500 (Washington, D.C.: May 1, 2003). In
this report, GAO uses the broad term "research" to refer to DOT's
research, development, and technology activities.

3PHMSA includes the Office of Pipeline Safety and the hazardous materials
safety activities formerly in RSPA.

4

H.R. Rep. No. 108-749 (2004).

RSPA with respect to mission, organizational structure, responsibility for
overseeing RD&T activities, work activities, budgetary resources, and
strategic goals. To determine RITA's practices for coordinating,
facilitating, and reviewing DOT's RD&T activities and what progress RITA
and RSPA have made in implementing GAO's 2003 recommendations, we reviewed
pertinent documentation and conducted semistructured interviews with
officials in RITA, the Office of the Secretary of Transportation, and
three of nine operating administrations that received RD&T funds in fiscal
year 2005. The three operating administrations we selected for
interviews-Federal Highway Administration, Federal Aviation
Administration, and Federal Transit Administration-are among those with
the largest RD&T budgets.5 To determine how BTS identifies its users and
monitors how well it is serving those users, we (1) identified criteria
for monitoring user satisfaction, including federal and other agency
requirements and guidelines set by DOT, the National Research Council, and
GAO; (2) through document reviews and interviews with BTS and operating
administration officialsidentified above, obtained information on BTS's
processes for identifying key users of its data products and for
soliciting and incorporating feedback from those users; and (3) compared
BTS's processes to the criteria to identify major gaps. We assessed the
reliability of the information contained in this report through interviews
with knowledgeable officials and reviews of documentation and
corroborating information, and we determined that it was sufficiently
reliable for our purposes. We conducted our work from November 2005
through August 2006 according to generally accepted government auditing
standards. This report summarizes the information we provided to your
staff during our May 30, 2006, briefing and, in addition, contains
recommendations to the Secretary of Transportation to improve RITA's
oversight of the department's RD&T activities and user satisfaction with
transportation statistics. The briefing slides are included in appendix I.

Taken together, the RD&T budget authority for these three operating
administrations accounted for over 80 percent of DOT's total RD&T budget
authorityin fiscal year 2005.

Page 3 GAO-06-917 Research and Innovative Technology Administration

  Results Summary

RITA Differs from RSPA in Proposed Budgetary Levels, Responsibility for
Evaluation, and Extent of Multimodal Focus

Beginning in 2005, RITA took over RSPA's responsibilities for overseeing
DOT's RD&T activities. While RITA continues to have a similar mission and
strategic objectives as RSPA-and still does not have the authority to
direct changes in the operating administrations' RD&T activities-RITA
differs from RSPA in several ways. First, RITA intends to focus
significantly more resources in fiscal year 2007 on the oversight of DOT's
RD&T activities. In DOT's fiscal year 2007 budget request, RITA proposes a
new program-the Transportation Futures and Applied Technology Program-to
build upon RITA's existing RD&T Coordination Program and provide access to
highly skilled, specialized technical experts to RITA on a contract basis.
If enacted, funding for these two programs would result in an
overallincrease of about $2 million over the enacted fiscal year 2006
budget for the RD&T Coordination Program-from $536,000 in fiscal year 2006
to about $2.5 million in fiscal year 2007.6 Second, RITA's responsibility
for evaluation is less clearly defined than RSPA's had been. RSPA had both
legislative and departmental responsibility for measuring the results of
DOT's RD&T programs and developing more efficient, effective, and
participative ways to evaluate and measure RD&T program effectiveness.7
However, RITA does not have the same legislative responsibility for
evaluating DOT's RD&T programs because the Safe, Accountable, Flexible,
Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU)8
removed the statutory responsibility for evaluation

6In RITA's fiscal year 2007 budget request, the amount RITA requested for
its RD&T Coordination Program ($247,000) is less than half of what was
enacted for those activities in fiscal year 2006 ($536,000). According to
the 2007 budget request, this decrease represents the transfer of
multimodal RD&T initiatives from the RD&T Coordination Program to the
Transportation Futures and Applied Technology Program. DOT's documentation
does not provide a clear transfer of all of the multimodal initiatives
from one program to the other; however, DOT officials told us that they
plan to review all of the department's multimodal initiatives as part of
the Transportation Futures and Applied Technology Program. As of June
2006, the House Committee on Appropriations denied RITA's request for $2.2
million for the Transportation Futures and Applied Technology Program,
saying that RITA should develop a more robust level of in-house RD&T
expertise beforeit relies on outside contractors for multimodal research
coordination and analysis. The House Committee on Appropriations also
provided $540,000 for RITA's ongoing RD&T Coordination Program.

7

Transportation Equity Act for the 21st Century (TEA-21), formerly codified
in 23 USC 508. Pub. L. No. 105-178, 112 Stat. 107 (1998).

8

Pub. L. No. 109-59, 119 Stat. 1144 (2005).

RITA Has Several Coordination, Facilitation, and Review Groups and
Practices but Lacks Performance Goals and a Plan for Evaluating Its Own
Efforts

activities that had been specifically outlined for DOT, which DOT
delegated to RSPA. Under the Norman Y. Mineta Act, RITA is charged with
reviewing DOT's RD&T activities, which, according to RITA officials, means
that RITA is not required to evaluate RD&T activities to determine whether
they are achieving intended goals, although RITA is not explicitly
prevented from evaluating such activities. Third, RITA's organizational
structure is more multimodally9 focused because the Office of Pipeline
Safety and Office of Hazardous Materials Safety were moved to the newly
created PHMSA, and the remaining offices in RSPA and other program
offices-BTS and the Office of Intermodalism-were transferred to the newly
created RITA.

RITA Has Several Coordination, Facilitation, and Review Groups and
Practices but Lacks Performance Goals and a Plan for Evaluating Its
Own Efforts

RITA coordinates, facilitates, and reviews DOT's RD&T activities in a
variety of ways, including through its two coordinating bodies-the RD&T
Planning Council and the RD&T Planning Team-and budget reviews, among
others. The briefing slides in appendix I contain a summary table and
detailed descriptions of RITA's activities. RITA officials and officials
from the three operating administrations we interviewed provided some
examples of the usefulness of RITA's coordination, facilitation, and
review practices; for example, RITA officials told us that the budget
review process results in a more consistent approach for the operating
administrations to show how their RD&T activities support DOT's strategic
objectives, secretarial priorities, and multimodal initiatives. Officials
from all three operating administrations told us that RITA's Program
Review Working Group10 meetings provide them with opportunities to share
best practices relating to program management issues, such as how to apply
the Office of Management and Budget's (OMB) R&D Investment Criteria11 to
RD&T activities.

While RITA and other DOT officials provided anecdotal examples of the
positive effects of RITA's coordination, facilitation, and review
practices,

9Individual transportation modes include mass transit systems, roads,
aviation, maritime, and railroads. RITA and its program offices oversee
and conduct RD&T and statistical activities that apply to more than one
mode of transportation.

10

The Program Review Working Group reports to the RD&T Planning Team and
provides a forum for the operating administrations to share information
about areas of research and identify opportunities for coordination.

11

OMB developed the R&D Investment Criteria-relevance, quality, and
performance-for all types of research and development programs across all
agencies to assist in budget allocation decisions and to demonstrate
results.

RITA has not established performance goals or an implementing strategy
that delineates how the activities and results of its coordination,
facilitation, and review practices will further DOT's mission and ensure
the effectiveness of the department's RD&T investment; in addition, RITA
does not monitor or evaluate the effects of its efforts. Also, RITA has
not worked with the operating administrations to develop common
performance measures for DOT's RD&T activities. RITA officials told us
that they were working with the operating administrations to develop an
RD&T strategic plan, which is due to Congress in the fall of 2006, but
they did not expect the plan toinclude goals and measures, as discussed
above. Setting meaningful goals for performance, and using performance
information to measure performance against those goals, is consistent with
requirements in the Government Performance and Results Act of 1993
(GPRA).12 Developing an evaluation plan and analyzing performance
information against set goals for its own coordination, facilitation, and
review practices could assist RITA in identifying any problem areas and
taking corrective actions.13 Without such goals and an evaluation plan, it
is difficult for RITA to determine its success in overseeing and ensuring
the effectiveness of DOT's RD&T activities. Also, without common
performance measures for the RD&T activities of the operating
administrations, RITA and the operating administrations lack the means to
monitor and evaluate the collective results of those activities and ensure
that they are achieving their intended (or other) results and furthering
the Secretary's and DOT-wide priorities. Linking performance goals with
the planning and budget process, such as DOT's annual budget process, can
also help RITA determine where to target its resources to improve
performance.14

12

Pub. L. No.103-62, 107 Stat. 285 (1993).

13

Use of performance goals can help ensure that programs are meeting their
intended goals, allows programs to assess the efficiency of their
processes, and promotes continuous improvement. Where activities may be
fragmented or overlap, performanceinformation can also help identify
performance variations and redundancies and lay the foundation for
improved coordination, program consolidation, or elimination of unneeded
programs. GAO, Managing for Results: Using the Results Act to Address
Mission Fragmentation and Program Overlap, AIMD-97-146 (Washington, D.C.:
August 1997).

14

GAO, Managing for Results: Enhancing Agency Use of Performance Information
for Management Decision Making, GAO-05-927 (Washington, D.C.: Sept. 9,
2005).

RITA Has Made Some Progress in Implementing GAO's 2003 Recommendations 

In 2003, GAO made five recommendations to DOT and RSPA
to improve the coordination and evaluation of
RD&T activities.15 The recommendations generally remain
relevant for RITA. As shown in table 1, RITA has partially
implemented four recommendations and has not implemented one
recommendation. The briefing slides in appendix I

contain more detailed information on RITA's actions with regard to our

recommendations.

Table 1: Status of RITA's Implementation of GAO's 2003 Recommendations

    Recommendations Status of implementation

Develop a strategy for reviewing all of DOT's research projects to
identify areas of unnecessary Partially implemented research duplication,
overlap, and opportunities for joint efforts.

     o Include time frames for implementing this review and discuss the
       development and implementation of a DOT-wide research tracking system
       database.
     o Incorporate the results of this effort into DOT's annual research plan
       and report to Congress on an annual basis.

Develop and apply quantifiable performance measures to assess the
effectiveness of research Not implemented coordination efforts and
document the results of these efforts in DOT's annual research plan.

Develop a strategy to ensure that the results of all of DOT's
transportation research activities are Partially implemented evaluated
according to established best practices.

     o Include estimates of the costs for ensuring that evaluations are
       completed.
     o Incorporate the results of these efforts in DOT's annual research plan
       and report to Congress on an annual basis.

Include in DOT's annual research plan a summary of all research program
evaluations conducted Partially implemented and a schedule of future
evaluations.

Document RSPA's process for systematically evaluating the results of its
own multimodal research Partially implemented programs, and apply this
process to any future multimodal research programs that RSPA conducts.
Source: GAO analysis of RITA information.

RITA has made progress, but much remains to be done for RITA to fully
implement the recommendations. The following is a summary, for each
recommendation, of the actions taken by RITA (and RSPA) and the type of
efforts that are still needed to fully implement the recommendations:

o  The first recommendation focuses on developing a strategy for
identifying areas of unnecessary research duplication, overlap, and
opportunities for joint efforts. While RITA officials have taken some
actions to identify areas of unnecessary research duplication and
opportunities for joint efforts through meetings of its RD&T Planning
Council and Planning Team,

15 GAO-03-500 .

among others, none of these groups have developed a strategy that
describes (1) the scope of the RD&T projects or programs that will be
reviewed for duplication or joint efforts, (2) the methodology for how all
research projects will be reviewed or how duplication or joint efforts
will be identified, (3) a timeline and the frequency for reviews to occur,
or (4) how the results of reviews-the identification of duplication or an
opportunity for joint effort-will be used to make decisions about future
RD&T activities.16 The first recommendation also includes the development
of a DOT-wide research tracking system database, which, according to a
RITA official, was dropped from DOT's priorities after the creation of
RITA. While RITA's proposed Transportation Futures and Applied Technology
Program includes the development of a Web-based database for DOT's RD&T
programs, it is uncertain whether this proposed program will be funded.
RITA has not yet established a catalog of all of the research projects
within DOT; as such, DOT officials do not have readily accessible data on
research activities throughout DOT. Without such a strategy-supported by a
comprehensive database of ongoing RD&T projects-RITA is unable to ensure
that areas of research duplication, overlap, and opportunities for joint
efforts are systematically identified and managed.

     o DOT and RSPA did not concur with the second recommendation in 2003,
       citing their views that (1) the most useful and effective performance
       measures focus on results, while coordination is a process and (2)
       that existing coordination processes have been effective in preventing
       unnecessary research duplication. While RITA officials told us that
       they have not developed or applied quantifiable performance measures
       for coordination because they do not believe that a metric approach is
       well suited for assessing the effects of the coordination process,
       they indicated their willingness to develop-in concert with the
       operating administrations-common performance measures for DOT's RD&T
       activities.
     o The third recommendation calls for developing a strategy for ensuring
       the evaluation of DOT's RD&T activities. RITA officials told us that
       they ensure the evaluation of RD&T activities by reviewing the
       operating administrations' application of OMB's R&D Investment
       Criteria- relevance, quality, and performance-through its budget
       review process.

Strategies help align an agency's activities, core processes, and
resources to support achievement of the agency's strategic goals and
mission. GAO, Results-Oriented Government: GPRA Has Established a Solid
Foundation for Achieving Greater Results, GAO-04-38 (Washington, D.C.:Mar.
10, 2004).

Page 8 GAO-06-917 Research and Innovative Technology Administration

According to RITA officials and OMB documents, the R&D Investment Criteria
are rooted in best practices and include peer review as a mechanism for
assessing program quality. However, RITA has neither developed nor
communicated a strategy for this process that describes (1) the scope of
RD&T activities of the operating administrations that RITA will ensure
were evaluated according to best practices, (2) the methodology for how
RITA will ensure evaluation of RD&T activities took place according to
established best practices, (3) a timeline for when the RD&T evaluations
should occur, and (4) how the results of the RD&T evaluations will inform
future research. Without such a strategy, RITA is less able to ensure the
quality and effectiveness of RD&T activities and investments to determine
whether they are achieving their intended (or other) goals.

     o The fourth recommendation focuses on publishing a summary of research
       program evaluations and a schedule of future evaluations. RSPA, in its
       fiscal year 2005 annual RD&T plan, published such a summary, but it
       consisted only of the results of its reviews of the operating
       administrations' application of OMB's R&D Investment Criteria. This
       summary did not include other research program evaluations or a
       schedule of future evaluations. Since RSPA was dissolved, RITA has not
       continued to publish the results of these types of reviews because
       SAFETEA-LU removed the requirement for RITA to submit an annual RD&T
       plan. Publishing a current inventory of the evaluation of research
       activities and a schedule for future evaluations on a regular basis
       could provide information about research results and planned research
       for future years.
     o The fifth recommendation addresses RITA's process for systematically
       evaluating the results of its own current multimodal research
       programs, such as the Hydrogen Safety Program, and future multimodal
       research programs. While RITA officials told us they oversee contracts
       and evaluate the results of RD&T activities that are conducted under
       these contracts through a peer review process, RITA has not
       systematically documented this process and it is not clear whether and
       how this process would apply to future multimodal research programs.
       Without a systematic process for evaluating current and future program
       results, RITA is limitedin its ability to determine the extent to
       which its multimodal RD&T programs are achieving their intended (or
       other) goals.

BTS's Process for Identifying Its Users and Monitoring How Well It Is
Serving Those Users Is Not Systematic

BTS does not have a systematic process in place for identifying its
primary users, soliciting ongoing feedback from those users, and
determining whether or how that feedback should be incorporated. First,
BTS has not established a systematic process for comprehensively
identifying its primary users, it does not track specific users, and it
does not have information on the overall number of users of its data
products. BTS officials told us that their users are primarily identified
in SAFETEA-LU, which only defines broad categories of data users, such as
the federal and state governments. BTS officials told us they also
identify some specific users through other methods, such as direct
inquiries received through BTS's Web site and by telephone. One BTS
official also commented that it is difficult to track individual users,
other than through Web site hits and tracking the number of publications
ordered. Also, the official said that BTS is limited in its ability to
collect information on individual data users because of privacy concerns.
Without a systematic process for comprehensively identifying primary users
for each of its products and services, BTS cannot solicit feedback from
these users on an ongoing basis to improve those products and services.

Secondly, BTS has not established a systematic process for soliciting
feedback from all of its primary users, although it has used a variety of
methods to obtain feedback from some users on a case-by-case basis. For
example, rather than routinely soliciting user feedback on all data
products and services, such as through a customer satisfaction survey, BTS
receives feedback from some data users about specific BTS products from an
online comment form and meetings and workshops held at conferences and
training sessions.17 Without a systematic process for soliciting feedback
on user satisfaction from all of its primary users, BTS cannot ensure it
has a full picture of the needs of those users and how well it is meeting
those needs, which, in turn, hinders BTS's ability to make improvements to
data products that are relevant to users.

Finally, to evaluate the feedback BTS has solicited from its users, BTS
officials told us they rely on program managers to determine how best to

17

In the past, BTS systematically solicited feedback on overall customer
satisfaction of BTS's data products and services through its Customer
Satisfaction Survey. BTS conducted the last such survey in 1998. Also, in
the past, BTS solicited feedback via comment cards, inserted in data
products. According to BTS officials, the survey and comment cards are no
longer used as methods for soliciting feedback as a result of restrictions
stemming from the Privacy Act and the Paperwork Reduction Act. BTS
officials told us they are working with the Chief Counsel's office at RITA
to reinstitute the use of comment cards.

                                  Conclusions

address feedback from its users; however, program managers are not
required to-and often do not-report the results of how feedback on user
satisfaction was considered. Although BTS measures the number of
congressional and governmental agency contacts regarding BTS information
and the average number of daily unique visitors to the TranStats data Web
site,18 these indicators are only simple counts, not measures of user
satisfaction with BTS information. GAO's internal control standards
suggest that ongoing monitoring take place to determine user satisfaction
and that policies and procedures be put in place to ensure that feedback
is evaluated so that improvements to products can be made.19 Without a
systematic process for identifying the primary users for each of its data
products and services, soliciting feedback on user satisfaction from its
users, and incorporating that feedback, BTS is limited in its ability to
consider feedback and make improvements to data products based on user
input.

Conclusions

Since it became operational in 2005, RITA has taken some positive steps to
meet its vision of becoming a departmentwide resource for managing and
ensuring the effectiveness of RD&T activities. In particular, RITA
established several coordinating bodies and review processes, and it has
also proposed additional initiatives, such as the Transportation Futures
and Applied Technology Program, to build upon its current coordination,
facilitation, and review practices. However, RITA lacks performance goals,
a clear implementing strategy, and an evaluation plan that collectively
delineates how the activities and results of its coordinating bodies,
review processes, and proposed initiatives will further DOT's mission and
ensure the effectiveness of the department's RD&T investment. Establishing
these mechanisms for managingits own performance-and linking them to DOT's
annual budget process-could provide RITA with a clear road map for
investing its own limited resources; enhance RITA's ability to identify
areas where its coordination, facilitation, and review efforts are working
effectively and where they could be improved; and help RITA provide
assurance that the department's RD&T activities are adequately
coordinated, routinely evaluated, and achieve their intended (or other)
results. In addition, RITA and the operating administrations have not

18

Required by SAFETEA-LU, the TranStats data Web site contains hundreds of
transportation-related databases and is made available to the public on
BTS's Web site.

19

GAO, Internal Control Management and Evaluation Tool, GAO-01-1008G
(Washington, D.C.: August 2001).

Page 11 GAO-06-917 Research and Innovative Technology Administration

worked together to develop common performance measures for RD&T activities
that are needed to evaluate the RD&T efforts departmentwide. With
performance goals, an implementing strategy, and an evaluation plan for
RITA-and common performance measures for RD&T activities- RITA and DOT
could be in a better position to assure Congress that DOT is making the
most of its approximately $1 billion annual RD&T investment. Additionally,
the strategy and performance measures could serve as a communication tool
to establish expectations and anticipated results with the operating
administrations.

BTS, as part of RITA, has solicited and evaluated some feedback on user
satisfaction, but it does not have a systematic process for identifying
primary users of its transportation data products and services, soliciting
feedback from those users, or incorporating the feedback it solicits. As a
result, BTS cannot ensure that it has a comprehensive picture of who uses
BTS data products, what their needs are, and how well the agency is
meeting those needs with its data products. By establishing more
systematic processes for identifying its primary users, soliciting
feedback from those users, and evaluating feedback on user satisfaction,
BTS could make more informed decisions on how to allocate limited
resources to make improvements to its data products. While BTS tracks and
reports the number of congressional and government agency contacts and the
number of visitors to its TranStats data Web site, developing performance
indicators that measure the overall degree to which products and services
are useful and responsive to the needs of its users will allow BTS to
understand how well it is serving its users over time.

  Recommendations for Executive Action
  
To enhance RITA's ability to manage and ensure the effectiveness of
RD&T activities in furthering the department's mission, we
recommend that the Secretary direct the RITA Administrator
to take the following seven actions:

        * Develop and incorporate the following into RITA's fiscal year 2008
          budget process, and the annual budget process thereafter:
             o performance goals and an overallimplementing strategy that
               delineate how the activities and results of its coordination,
               facilitation, and review practices will further DOT's mission
               and ensure the effectiveness of the department's RD&T
               investment. The strategy should include an evaluation plan for
               monitoring and evaluating its performance against set goals to
               assist RITA in better allocating its resources to improve
               performance.
             o common performance measures related to DOT's RD&T activities,
               which should be developed in consultation with the operating
               administrations.
        * Develop and incorporate the following into RITA's fiscal year 2008
          budget process, the annual budget process thereafter, and the
          upcoming RD&T strategic plan:20
             o a strategy for identifying and reviewing all of DOT's RD&T
               projects to determine areas of unnecessary duplication,
               overlap, and opportunities for joint efforts. The strategy
               should address (1) the scope of the RD&T projects or programs
               that will be reviewed for duplication or joint efforts, (2)
               the methodology for how all RD&T projects will be reviewed or
               how duplication or joint efforts will beidentified, (3) a
               timeline and the frequency for reviews to occur, and (4) how
               the results of the reviews-the identification of duplication
               or an opportunity for joint effort-will be reported and used
               to make decisions about future RD&T activities.
             o a strategy to ensure that the results of all of DOT's RD&T
               activities are evaluated according to established best
               practices. This strategy should include (1) which RD&T
               activities of the operating administrations RITA will ensure
               were evaluated according to best practices, (2) the
               methodology for how RITA will ensure evaluation of RD&T
               activities took place according to established best practices,
               (3) a timeline for when the RD&T evaluations should occur, and
               (4) how the results of the RD&T evaluations will inform future
               research.
             o a DOT-wide database of all of DOT's RD&T projects that will
               support RITA's coordination, facilitation, and review efforts
               and will assist in the implementation of the strategies
               discussed above. Information on the status of these efforts
               should be included in the upcoming RD&T strategic plan to be
               issued in the fall of 2006.
             o a summary of all of DOT's RD&T program evaluations conducted
               by the department for the past 3 years, including ongoing and
               completed evaluations, and a schedule of future evaluations.

20The following five recommendations reiterate and expand on four of the
five recommendations GAO made to RSPA in 2003, with the exception of the
recommendation on quantifiable performance measures for RD&T coordination,
which DOT did not concur with, as previously discussed.

Page 13 GAO-06-917 Research and Innovative Technology Administration

o  a description of RITA's process for systematically evaluating the
results of its own multimodal research programs and how this process will
be applied to future multimodal research programs that RITA conducts.

To help ensure that BTS's data products meet the needs of its users, we
recommend that the Secretary direct the RITA Administrator and BTS
Director to take the following action:

o  Develop and implement a systematic process for BTS to identify its
primary users, solicit and incorporate feedback from those users, and
measure the satisfaction of its users. This process should contain the
following elements: (1) that primary users of BTS's data products and
services are identified and documented in a comprehensive manner; (2) that
feedback on user satisfaction is solicited on a periodic basis from those
users; (3) that user feedback is documented and evaluated at BTS's
agencywide level and against established criteria, to ensure consistency
in decisions about what improvements should be made to data products; and

(4) that performance indicators that measure data users' satisfaction are
developed and applied.

Agency Comments

We obtained oral comments on a draft of this report from DOT officials,
who generally agreed with our findings and recommendations. These
officials also provided technical clarifications that we incorporated into
the report, as appropriate.

We are sending copies of this report to the appropriate congressional
committees and to the Secretary and other appropriate officials in the
Department of Transportation. We will also make copies available to others
upon request. In addition, the report will be available at no charge on
the GAO Web site at http://www.gao.gov.

If you or your staff have any questions regarding this report, please
contact me at (202) 512-2834 or [email protected]. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on the
last page of this report. GAO staff who made major contributions to this
report are listed in appendix II.

Mathew Scire Acting Director, Physical Infrastructure Issues

Appendix I:  Review of the Department of Transportation's Research and
Innovative Technology Administration (RITA)

                 Briefing for the Committees on Appropriations
					  
             U.S. Senate and House of Representatives May 30, 2006

Briefing Overview

     o Introduction and Objectives
     o Approach
     o Background
     o Results of GAO Work

Introduction

     o In fiscal year 2005, the Department of Transportation's (DOT)
       research, development, and technology (RD&T)1 budget totaled
       approximately $1.1 billion. RD&T activities are vital to meeting DOT's
       key transportation priorities, including safety and mobility.
     o Prior to 2005, DOT's Research and Special Program Administration
       (RSPA) was responsible for overseeing DOT's RD&T activities, the
       Office of Hazardous Materials Safety, and the Office of Pipeline
       Safety.
     o GAO and others have raised concerns about RSPA-for example, GAO made
       several recommendations in a 2003 report to improve RSPA's
       coordination and evaluation of DOT's RD&T activities.
          * In November 2004, the Norman Y. Mineta Research and Special
            Programs Improvement Act (P.L. 108-426) dissolved RSPA and
            created two new administrations:
               o the Research and Innovative Technology Administration
                 (RITA). The Mineta Act transferred the Bureau of
                 Transportation Statistics (BTS) and the Office of
                 Intermodalism to RITA.
               o the Pipeline and Hazardous Materials Safety Administration,
                 which includes the Office of Pipeline Safety and the
                 hazardous materials safety activities that were formerly in
                 RSPA.

1 OMB Circular No. A-11 (2005) defines research and development activities
to include "work undertaken on a systematic basis ... to increase the
stock of 3 knowledge, including knowledge of man, culture and society, and
the use of this stock of knowledge to devise new applications." DOT
defines technology to include demonstration projects and other related
activities associated with research and development activities.

Objectives

o  In response to a mandate from the Senate and House Appropriations
Committees, GAO addressed the following questions:

1. How, if at all, do RITA's responsibilities for overseeing DOT's RD&T
       activities differ from RSPA's?
2. What are RITA's practices for coordinating, facilitating, and
       reviewing RD&T activities and what progress has RITA made in
       implementing GAO's 2003 recommendations?
3. How does BTS identify its users and monitor how well it is serving
       those users?

Overall Approach

     o Legislative & Document Review -Reviewed laws and DOT documentation to
       understand RSPA's and RITA's responsibilities and BTS' legislative
       requirements for producing certain data.
     o Interviews -Interviewed DOT officials in: RITA, including the RD&T
       Office, BTS, and others; three operating administrations that conduct
       and manage significant research programs, including the Federal
       Highway Administration (FHWA), Federal Aviation Administration (FAA),
       and Federal Transit Administration (FTA); and the Office of the
       Secretary (OST).
     o Recommendation Follow-up -Reviewed the extent to which RITA (and RSPA)
       have addressed the recommendations in GAO's 2003 report.2
     o Analysis of selected Bureau of Transportation Statistics' (BTS)
       processes -Analyzed BTS' processes for identifying its data users and
       for monitoring and incorporating feedback on user satisfaction and
       compared those processes to relevant criteria.
     o We conducted our work from November 2005 through May 2006 according to
       generally accepted government auditing standards.

2 GAO, Transportation Research: Actions Needed to Improve Coordination and
Evaluation of Research, GAO-03-500 (Washington, D.C.: May 1, 2003).

Results in Brief

     o RITA took over RSPA's responsibilities for overseeing DOT's RD&T
       activities. While RITA continues to have a similar mission and
       strategic objectives as RSPA, it still does not have the authority to
       direct changes in the operating administrations' RD&T activities. RITA
       differs from RSPA in several ways-RITA intends to focus significantly
       more resources in fiscal year 2007 on the oversight of DOT's RD&T
       activities; RITA's authority for evaluation is less clearly defined;
       and RITA's organizational structure is more multi-modally focused.
     o RITA coordinates, facilitates, and reviews DOT's RD&T activities
       through multiple groups and practices, including its two coordinating
       bodies-the RD&T Planning Council and the RD&T Planning Team-and budget
       reviews, among others. Of the five recommendations GAO made in 2003 on
       how to improve the coordination and evaluation of RD&T activities,
       RITA has partially implemented four of the recommendations and has not
       implemented the other.
     o BTS does not have a systematic process in place for identifying its
       users, soliciting ongoing feedback from all its major users, and
       determining whether or how that feedback should be incorporated.

Background: RITA's legislative responsibilities

The Norman Y. Mineta Act lays out five broad responsibilities for RITA:

     o Coordination, facilitation, and review of DOT's research and
       development programs and activities;
     o Comprehensive transportation statistics research, analysis, and
       reporting;
     o Advancement, and research and development, of innovative technologies,
       including intelligent transportation systems;
     o Education and training in transportation and transportation-related
       fields; and
     o Activities of the Volpe National Transportation Center, which conducts
       a range of transportation research and development projects on a
       fee-for-service basis.

Background: Scope of RITA's authority

     o RITA's Office of RD&T is responsible for "coordinating, facilitating,
       and reviewing" DOT's RD&T activities.
          * According to the Norman Y. Mineta Act, RITA's scope of authority
            includes coordinating, facilitating, and reviewing all of the
            Department's RD&T programs and activities, with one
            exemption-RITA was not given any authority over the research and
            other programs, activities, standards, or regulations
            administered by the National Highway Traffic Safety
            Administration (NHTSA).
               o The exemption does not apply to NHTSA's activities already
                 in effect on November 30, 2004, the date of the Mineta Act's
                 enactment.
               o Officials in RITA's Office of RD&T told us that in spite of
                 the NHTSA exemption in the Mineta Act, NHTSA voluntarily
                 participates in all of RITA's coordination, facilitation,
                 and review activities.

Background: BTS' mission and legislative requirements

     o The mission of BTS-a principal federal statistical agency established
       in 1992-is to develop high quality transportation data and information
       and advance the use of this data and information in both public and
       private transportation decision-making.
          * BTS' legislative requirements in the Safe, Accountable, Flexible,
            Efficient Transportation Equity Act: A Legacy for Users
            (SAFETEA-LU) (P.L. 109-59) include, among others:
               o Coordinating collection of information with the operating
                 administrations within DOT and with other federal agencies;
               o Issuing guidelines for the collection of statistical
                 information by DOT;
               o Reviewing and reporting on the sources and reliability of
                 the statistics proposed by the operating administrations as
                 requested by the Secretary; and
               o Submitting to the President and Congress a Transportation
                 Statistics Annual Report (TSAR).

Objective 1: Similarities between RITA and RSPA

     o RITA, like RSPA before it, has the responsibility to oversee DOT's
       RD&T activities but does not have the authority to direct changes in
       the operating administrations' RD&T activities, including budgetary
       changes.
     o RITA's mission, like RSPA's before it, is focused on using research to
       meet DOT's goals, including enhancing transportation safety and
       efficiency.
     o Two of RITA's three strategic goals--providing research and analytical
       capabilities dedicated to furthering DOT's mission and promoting
       multimodal, cross-cutting transportation technologies and innovations-
       are similar to RSPA's goals. RITA's third strategic goal is to ensure
       the effectiveness of the public investment in transportation RD&T.

       Differences between RITA and RSPA: RITA intends to focus significantly
                     more resources on its oversight of DOT's RD&T activities

     o RITA's fiscal year 2007 budget request proposes a new program-the
       Transportation Futures and Applied Technology Program-to undertake
       certain oversight and coordination activities, and also continues
       RITA's existing RD&T Coordination program. If enacted, funding for
       these two programs would result in an overall increase of about $2
       million over the enacted fiscal year 2006 budget for RD&T
       Coordination-from $536,000 in fiscal year 2006 to about $2.5 million
       in fiscal year 2007-although the Transportation Futures and Applied
       Technology Program also funds activities other than coordination.3
     o According to senior DOT officials in OST and RITA, although the
       Transportation Futures and Applied Technology Program and the RD&T
       Coordination program were listed as separate programs in DOT's 2007
       budget request, DOT views them as one unified program, with the intent
       that the Transportation Futures and Applied Technology Program will
       build on and enhance the existing RD&T Coordination program. A
       significant element of the funding request for the Transportation
       Futures and Applied Technology Program is to enable RITA to have
       access to highly skilled, specialized technical experts that RITA
       views as critical to enhancing its ability to effectively coordinate,
       facilitate, and review DOT's RD&T activities and enable RITA to more
       fully participate in strategic planning, priority setting, and
       decision-making related to those activities. Further, both programs
       are requesting funding to support intermodal and interagency teams
       that are working to better leverage research results and advance the
       collaborative integration and deployment of new technologies.

3 These other activities include conducting technology scans and
accelerating the deployment of technologies and innovations to users.

       Differences between RITA and RSPA: RITA intends to focus significantly
          more resources on its oversight of DOT's RD&T activities, continued

     o The presentation of RITA's fiscal year 2007 budget request does not
       provide sufficient detail to determine the total amounts of funding
       that RITA is requesting for the two programs since the budget only
       shows the amount of proposed contract dollars for the two programs,
       including $2.2 million for the Transportation Futures and Applied
       Technology Program and $247,000 for the RD&T Coordination program. The
       2007 budget request also shows that RITA is requesting about $5.2
       million for salaries and administrative expenses for all research and
       development activities, but it does not specify what portion of that
       amount will be dedicated to the two programs.
          * The amount that RITA requested for its RD&T Coordination Program
            in fiscal year 2007 ($247,000) is less than half of what was
            enacted for those activities in fiscal year 2006 ($536,000).
            According to the 2007 budget request, this decrease represents
            the transfer of cross-modal RD&T initiatives from the RD&T
            Coordination program to the Transportation Futures and Applied
            Technology Program. DOT's documentation does not provide a clear
            cross-walk showing the transfer of all the cross-modal
            initiatives from one program to the other. However, a senior
            official in OST told us that DOT plans to review all of its
            crossmodal initiatives as part of the Transportation Futures and
            Applied Technology Program for the purpose of re-focusing them to
            ensure they are addressing the most important DOT-wide research
            priorities.
          * Differences between RITA and RSPA: RITA's authority for
            evaluation is less clearly defined
     o RSPA had both legislative and departmental responsibility for
       evaluating DOT's RD&T programs; specifically RSPA was required to:

o  measure the results of federal transportation research activities and
how those activities impact the performance of the surface transportation
systems of the United States4and

o  oversee DOT's RD&T programs and develop more efficient, effective, and
participative ways to evaluate and measure program effectiveness and
progress across all operating administrations and other selected
projects.5

     o RITA does not have the same legislative responsibility for evaluating
       DOT's RD&T programs because SAFETEA-LU removed the statutory
       responsibility for evaluation activities that had been specifically
       outlined for DOT, which DOT delegated to RSPA.
     o RITA officials told us that their interpretation of "review" authority
       under the Norman Y. Mineta Act is that RITA is not required to
       evaluate RD&T activities to determine whether they are achieving
       intended goals, although RITA is not explicitly prevented from
       evaluating such activities.

4 As stated in the Transportation Equity Act for the 21st Century
(TEA-21), formerly codified in 23 USC 508. 15 5 DOT policy delegated this
responsibility to RSPA.

Objective 2: RITA's Coordination, Facilitation, and Review Practices

RITA's Office of RD&T coordinates, facilitates, and reviews DOT's RD&T
activities through various groups/practices, as summarized below:

Group / Process Function                                  Participants     
RD&T Planning   Advises the DOT Secretary on RD&T                Operating 
Council         policies and priorities necessary to       Administrators, 
                   support the DOT                           
                   Strategic Plan and Administration and     the              
                   Secretarial policy and objectives.        Undersecretary   
                   Ensures cross-modal                       for              
                   collaboration and coordination in RD&T        Policy, OST; 
                   initiatives within DOT and with external       Chair, RITA 
                   entities. Reports                         
                   to the Secretary.                         Administrator    
RD&T Planning   Assists the RD&T Planning Council and the Associate        
Team            RITA Administrator in ensuring            Operating        
                   cross-modal                               
                   collaboration and coordination in DOT's     Administrators 
                   RD&T initiatives. Assures the application        for RD&T, 
                   of                                        
                   Administration and Secretarial RD&T       OST; Chair,      
                   priorities and policies and guidance from RITA's           
                   the RD&T                                  
                   Planning Council.                         Associate        
                                                             Administrator    
                                                             for RD&T         
                   Conducts the annual review of modal RD&T         Operating 
Program Review  programs to ensure they are aligned with  Administrations' 
                   DOT                                       
Working Group   strategic goals and implement the Office  RD&T managers    
                   of Management and Budget's (OMB) Research 
                   and                                       
                   Development (R&D) investment criteria. 8  
                   Reports to RD&T Planning Team.            
                   RITA's annual assessment of the operating                  
Budget Review   administrations' RD&T budget-first        RITA, Operating
                   performed on                              
                   fiscal year 2007 budget submissions--to   Administrations, 
                   ensure alignment with DOT's strategic     OST's            
                   objectives,                               
                   Secretarial priorities, and cross-modal   Budget Office    
                   initiatives and proper application of     
                   OMB's R&D                                 
                   Investment Criteria.                      
                   Advances U.S. technology and expertise in                  
University      transportation through the mechanisms of  RITA, Operating
                   education,                                
Transportation  research and technology transfer at       Administrations  
Centers         university-based centers of excellence.   
                   RITA administers                          
Program         grants (mostly legislatively designated)  
                   to universities.                          
Peer Review     Coordinates the implementation of OMB's                    
Task            Information Quality Bulletin on Peer      RITA, Operating
                   Review for                                
Force and       influential and highly influential        Administrations' 
Steering        research (as determined by the agency or  RD&T             
                   OMB) and oversees                         
Committee       compliance with these requirements.       managers, Data   
                   Prepares annual report to OMB.            Quality          
                                                             experts.         

RD&T Strategic Develops the details of the RD&T Strategic Plan due to
Congress in the fall of 2006. The draft RITA, Operating Planning Task
Force plan will be reviewed by the National Research Council in June 2006.
Administrations' RD&T

managers Various multi-modal Examples include: RITA, Operating RD&T groups
managed x Hydrogen Working Group which coordinates, facilitates and
informs all RD&T activities Administrations' policy by RITA related to
hydrogen fueled and fuel cell powered vehicles and stationary
applications. and technical staff and

x Center for Climate Change which shares information, builds partnerships,
and coordinates other representatives activities related to climate
change. Source: GAO analysis of RITA information.

8 OMB developed the R&D Investment Criteria-quality, relevance, and
performance-for all types of R&D programs across all agencies to assist 16
in budget allocation decisions and to demonstrate results.

RITA's Coordination, Facilitation, and Review Practices, continued

        * RD&T Planning Council
             o According to DOT Order 1120.39A, the Planning Council is to
               meet quarterly. The Planning Council has convened three times,
               most recently on May 15, 2006. RITA officials told us the
               Council expects to meet more regularly as it reviews the
               development of the draft RD&T Strategic Plan before it is
               submitted to Congress-several meetings have been scheduled for
               the summer of 2006.
             o According to the Planning Council's meeting minutes from May
               and June, 2005, the Planning Council reviewed and approved the
               RD&T budget guidance that was distributed to the operating
               administrations and discussed future strategic directions for
               DOT research, the fiscal year 2007 RD&T budget submissions,
               and other topics.
        * RD&T Planning Team
             o According to DOT Order 1120.39A, the Planning Team is to meet
               quarterly, which it has done. The Planning Team has convened
               four times, most recently in February 2006, and another
               meeting is scheduled for May 24, 2006.
             o According to the Planning Team's meeting minutes from June and
               November, 2005 and February, 2006, the Planning Team discussed
               results of RITA's budget review for fiscal year 2007, effects
               of SAFETEA-LU on research programs, and updates on the
               development of the RD&T strategic plan and the University
               Transportations Centers program.

RITA's Coordination, Facilitation, and Review Practices, continued

        * Program Review Working Group:
        * o  Established through a DOT Order that allowed the RD&T Planning
          Team to create ad hoc working groups, this group met nine times
          during 2005 and 2006 (as of April). The group provides a forum for
          the operating administrations to share information about areas of
          research and identify opportunities for coordination. The group
          also schedules presentations throughout the year which it shares
          with the RD&T Planning Team. Past presentations have addressed how
          the operating administrations have applied OMB's R&D Investment
          Criteria to their RD&T activities and how the operating
          administrations' RD&T activities support DOT's and the Secretary's
          strategic priorities.
        * Budget Review
             o In preparation for their first budget review, RITA officials
               developed the 2007 RD&T budget guidance for the operating
               administrations, advising them to include discussions of how
               their RD&T activities are linked to the Secretary's priorities
               and how they have applied OMB's R&D Investment Criteria. This
               guidance was reviewed by the Planning Team, approved by the
               Planning Council, and distributed by OST's Budget Office.
             o The results of RITA's budget review were discussed at the June
               2005 Planning Council meeting. Also, RITA developed
               recommendations-to DOT's Assistant Secretary for Budget and
               Programs/Chief Financial Officer-that RITA continue to work
               with the operating administrations to improve the Department's
               RD&T budget submissions and that RITA develop guidance for
               defining RD&T to ensure consistency in the way that the
               Department classifies and accounts for its RD&T programs.

RITA's Coordination, Facilitation, and Review Practices, continued

o  Management of the University Transportation Centers (UTC) Program9

        * RITA's RD&T Office manages the UTC program, which was significantly
          expanded by SAFETEALU.
             o SAFETEA-LU authorized up to about $76.7 million per year for
               UTC grants during the period from fiscal year 2005 to 2009, an
               increase of about 136 percent compared to the $32.5 million
               authorized under TEA-21.
             o Under SAFETEA-LU, the UTC grants provide funding to establish
               and operate up to 60 UTCs, an increase of about 82 percent
               compared to the 33 centers authorized under TEA-21. Twenty of
               the 60 centers are scheduled for competitive selection during
               2006, and 40 centers are located at institutions named in the
               legislation.
     o To foster a closer connection between UTC research and needs of the
       operating administrations and DOT, RITA hosted a two-day workshop in
       April 2006 where officials from operating administrations made
       presentations on their RD&T activities and DOT's priorities to UTC
       members. RITA officials noted that this is the first of a planned
       series of program meetings, technical workshops, and other outreach
       efforts to increase UTC connections to DOT's RD&T priorities and
       programs. RITA is also working with the operating administrations to
       increase technical oversight of the individual UTCs' selection and
       evaluation processes for research.

9 The UTC program was initiated in 1987 under the Surface Transportation
and Uniform Relocation Assistance Act, which authorized the

19

establishment and operation of transportation centers in each of the ten
federal regions. The Intermodal Surface Transportation Efficiency Act of
1991 (ISTEA) reauthorized the UTC program for an additional six years and
added four national centers and six University Research Institutes.

Effects of RITA's coordination, facilitation, and review activities

        * RITA has not established performance goals for its coordination,
          facilitation, and review activities, nor does RITA monitor or
          evaluate the effects of those activities. Also, RITA has not worked
          with the operating administrations to develop common performance
          measures for the Department's RD&T activities.
             o Setting meaningful goals for performance, and using
               performance information to measure performance against those
               goals, is required by the Government Performance and Results
               Act of 1993.10 GAO has also reported on key practices-such as
               defining common outcomes and developing mechanisms to monitor,
               evaluate, and report on results-that can enhance and sustain
               collaboration among organizations that conduct cross-cutting
               activities.11
             o Establishing performance goals and an evaluation plan for its
               own coordination, facilitation, and review activities could
               assist RITA in identifying any problem areas and better
               allocating its resources to improve performance. Without such
               goals and an evaluation plan, it is difficult for RITA to
               determine its success in overseeing and ensuring the
               effectiveness of DOT's RD&T activities. Also, without common
               performance measures for the RD&T activities of the operating
               administrations, RITA and the operating administrations lack
               the means to monitor and evaluate the collective results of
               those activities and ensure that they are achieving their
               intended (or other) results and furthering the Secretary's and
               DOT-wide priorities.
     o The RITA Administrator told us that he was open to developing
       performance goals and an evaluation plan for RITA's coordination,
       facilitation, and review activities and to working with the operating
       administrations to develop common performance measures for DOT's RD&T
       activities.

10 Pub. L. No. 103-62, 107 Stat. 285 (1993). Under GPRA, federal agencies
are required to develop strategic plans, performance plans, and
performance reports that set long-term and annual goals along with the
means for accomplishing the goals and report on achieving them. 20

11 GAO, Results-Oriented Government: Practices That Can Help Enhance and
Sustain Collaboration among Federal Agencies, GAO-06-15 (Washington, D.C.:
October 21, 2005).

Effects of RITA's coordination, facilitation, and review activities,
continued

o  Although RITA lacks performance goals and measures, officials from the
three operating administrations we interviewed and from RITA noted some
positive effects of RITA's efforts:

     o Officials from all three operating administrations told us that RITA's
       Program Review Working Group meetings provide them with opportunities
       to share best practices relating to program management issues, such as
       how to apply the OMB R&D Investment Criteria to RD&T activities. An
       official from one of the operating administrations said that RITA has
       been helpful in trying to strengthen the connection between the UTCs
       and the DOT's RD&T priorities through programs such as the 2-day UTC
       workshop hosted in April 2006. Officials from another operating
       administration told us that RITA has helped them to better align their
       RD&T activities with DOT's and the Secretary's priorities by focusing
       on how their research activities will support DOT's RD&T Strategic
       Plan.
     o Officials from RITA's Office of RD&T believe that RITA's budget
       guidance and review process is helping to ensure that the operating
       administrations have a more consistent approach to show how their RD&T
       activities support DOT's strategic objectives, Secretarial priorities,
       and crossmodal initiatives. One official noted that, over time, this
       process should give RITA the ability to critically review and make
       recommendations to OST on how RD&T activities could better support
       DOT's strategic objectives, Secretarial priorities, and cross-modal
       initiatives. Also, the Program Review Working Group has identified an
       opportunity for FHWA and FAA to coordinate pavement research.

Follow-up to GAO Recommendations

     o In 2003, GAO made five recommendations to DOT and RSPA to improve the
       coordination and evaluation of RD&T activities.
     o The recommendations generally remain relevant for RITA.
     o RITA has partially implemented four recommendations and has not
       implemented one recommendation. RITA has made progress, but much
       remains to be done for RITA to fully implement the recommendations.

Recommendation 1

Develop a strategy for reviewing all DOT research projects to identify
areas of unnecessary research duplication overlap, and opportunities for
joint efforts.

     o Include time frames for implementing this review and discuss the
       development and implementation of a DOT-wide research tracking system
       database.
     o Incorporate the results of this effort into DOT's annual research plan
       and report to Congress on an annual basis.

      Status: Partially Implemented

     o RITA has not developed a strategy to review all DOT research projects
       to identify areas of unnecessary research duplication overlap and
       opportunities for joint efforts, although they have taken some action,
       as discussed below. RITA officials told us that they expect to
       identify such areas through the coordination and review activities of
       the RD&T Planning Council, RD&T Planning Team, and the Program Review
       Working Group. However, although the DOT Order that created the RD&T
       Planning Council, Planning Team, and indirectly the Program Review
       Working Group, lays out as part of its purpose the responsibility to
       prevent unnecessary duplication of RD&T efforts in DOT, none of these
       groups has developed a strategy that describes (1) the scope of the
       RD&T projects or programs that will be reviewed for duplication or
       joint efforts, (2) the methodology for how all research projects will
       be reviewed or how duplication or joint efforts will be identified,
       (3) a time line and frequency for the review to occur, and (4) how the
       results of the review-the identification of duplication or an
       opportunity for joint effort-will be used to make decisions about
       future RD&T activities.
     o A RITA official told us that the research tracking system database was
       dropped from DOT's priorities after the creation of RITA; however,
       RITA's proposed Transportation Futures and Applied Technology Program
       includes the development of a Web-based database for DOT's RD&T
       programs. RITA has not yet established a catalog of all the research
       activities within DOT. Without such a catalog, DOT officials do not
       have readily accessible data on research activities throughout DOT.

Recommendation 1

Develop a strategy for reviewing all DOT research projects to identify
areas of unnecessary research duplication overlap, and opportunities for
joint efforts.

     o Include time frames for implementing this review and discuss the
       development and implementation of a DOT-wide research tracking system
       database.
     o Incorporate the results of this effort into DOT's annual research plan
       and report to Congress on an annual basis.

      Status: Partially Implemented

     o While the results of RITA's coordination and review activities are
       discussed during Planning Team meetings, the results are not reported
       to Congress because RITA is no longer required to submit an annual
       RD&T plan to Congress. RITA officials told us that they plan to report
       these results annually in the DOT Performance Plans and Reports
       required by SAFETEA-LU. This reporting is expected to begin in Fiscal
       Year 2008, as the first reporting against the new RD&T Strategic Plan.
     o Without such a strategy-supported by a comprehensive database of
       ongoing RD&T projects-RITA is unable to ensure that areas of
       unnecessary research duplication, overlap, and opportunities for joint
       efforts are systematically identified and managed.

Recommendation 2

Develop and apply quantifiable performance measures to assess the
effectiveness of research coordination efforts and document the results of
these efforts in DOT's annual research plan.

      Status: Not Implemented

     o DOT and RSPA did not concur with this recommendation in 2003, citing
       their views that: (1) the most useful and effective performance
       measures focus on results, while coordination is a process; (2) there
       are no performance measures capable of quantifying effective
       coordination; and (3) existing coordination processes have been
       effective in preventing unnecessary research duplication.
     o RITA officials also told us that they did not develop performance
       measures to assess the effectiveness of research coordination effort
       because they do not believe that a metric approach is well suited for
       assessing the effects of the coordination process.
     o While RITA officials told us that they have not developed or applied
       performance measures for coordination, they indicated their
       willingness to develop--in concert with the operating
       administrations-common performance measures for DOT's RD&T activities.

12 GAO, Highway Research: Systematic Selection and Evaluation Processes
Needed for Research Program, GAO-02-573 (Washington, D.C.: May 24, 2002).
Peer review includes an independent assessment of technical and scientific
merit or quality and is 26 considered appropriate for all RD&T programs.

Recommendation 4

Include in DOT's annual research plan a summary of all research program
evaluations conducted and a schedule of future evaluations.

      Status: Partially Implemented

     o RSPA, for one year, published a summary of all research program
       evaluations conducted, in that it published the results of its review
       of the operating administrations' application of OMB's R&D Investment
       Criteria in its fiscal year 2005 annual RD&T plan. A schedule of
       future evaluations was not included in this plan.
     o Since RSPA was dissolved, RITA has not continued to publish the
       results of its reviews of the operating administrations' application
       of OMB's R&D Investment Criteria because SAFETEA-LU no longer requires
       RITA to submit an annual research plan.
     o Publishing a current inventory of the evaluation of research
       activities and a schedule for future evaluations on a regular basis
       could provide continuity and context for the observations about
       research results and planned research for future years.

Recommendation 5

Document RSPA's process for systematically evaluating the results of its
own multi-modal research programs, and apply this process to any future
multimodal research programs that RSPA conducts.

      Status: Partially Implemented

     o RITA has not documented its process for systematically evaluating the
       results of its own multi-modal research programs, or how this process
       would apply to any future multi-modal research programs that it
       conducts, although RITA has taken some action, described below.
     o RITA manages multi-modal RD&T activities, such as the Hydrogen Safety
       Program and various grant programs in SAFETEA-LU delegated to it by
       DOT. According to a RITA official, RITA oversees contracts and
       evaluates the results of RD&T activities that are conducted under
       these contracts through the peer review process. RITA has not
       documented its process for conducting these peer reviews and it is
       unclear whether the peer review process will be applied to future
       multi-modal RD&T activities.
     o Without systematically evaluating program results, RITA is limited in
       its ability to determine the extent to which its multi-modal RD&T
       programs are achieving their intended (or other) goals.

Objective 3: BTS' process for identifying its users is not systematic

     o BTS does not have a systematic process, but uses a variety of methods,
       for identifying specific users of BTS information.
     o BTS officials told us they primarily rely on a Congressional
       definition of broad categories of data users, contained in SAFETEA-LU.
       This definition states that users of BTS information include the
       Federal government, State and local governments, metropolitan planning
       organizations, transportation-related associations, the private sector
       (including the freight community), and the public.
          * BTS officials told us that they identify specific users through a
            variety of methods such as:
               o Conferences, focus groups, and training sessions;
               o Meetings of professional associations;
               o Direct inquiries via the BTS Web site, in the form of email,
                 letters, and telephone; and
               o News media citations--tracked by RITA's Office of
                 Governmental, International and Public Affairs--that mention
                 BTS data.
     o BTS does not track specific users nor does it have information on the
       overall number of users of its data products. According to one BTS
       official, the reason for this is because it is very difficult to track
       individual users, other than through Web site hits and tracking the
       number of publications ordered. Also, the official mentioned BTS is
       limited in its ability to collect information on individual data users
       due to privacy concerns.

x Without a systematic process for identifying primary users for each of
its programs, BTS cannot solicit feedback from these users on an ongoing
basis.

29

Examples of BTS data users

o  Examples of users of BTS information

             BTS Data Users Data product used and for what purpose

Department of x DOT's Office of the Secretary uses BTS airline data to
determine eligibility for the Essential Air Transportation Service (EAS)
program, monitor the impact of airline service on the traveling public,
negotiate international air service agreements, and provide oversight of
the airline industry. x The Federal Highway Administration uses Commodity
Flow Survey data in its Freight Analysis Framework, a departmental
planning tool.

x DOT's Federal Railroad Administration (FRA) uses data collected by BTS
for the Confidential Close Call Reporting System Demonstration Project to
help FRA and railroad carriers identify safety issues that require
corrective action.

Other Federal x BTS worked with the Commerce Department's Bureau of
Economic Analysis to develop a agencies Transportation Satellite Account,
which enhanced the measurement of transportation's contribution to the
Gross Domestic Product. x The Department of Homeland Security`s
Transportation Security Administration used BTS' Omnibus Survey Program
data on the amount of baggage brought to airports.

Policymakers x BTS is developing capital stock values (a commonly used
economic measure of the capacity of the transportation system) for
airports, waterways, and transit systems for use by policymakers to better
estimate the amount of investment needed to accommodate current or future
levels of traffic on those modes.

Planners x States use National Household Travel Survey (NHTS) data for
transit planning; Metropolitan Planning Organizations (MPO) use NHTS data
for highway planning.

                    Source: GAO analysis of BTS information.

BTS' methods for soliciting feedback on user satisfaction is not
systematic

        * BTS' process for soliciting feedback on user satisfaction is not
          systematic, but BTS officials reported using a variety of methods
          for soliciting feedback from its users, including:
             o Data users' meetings and workshops held at conferences or
               training sessions, and focus groups conducted with BTS data
               users;
             o BTS' Reference Services, which tracks and responds to user
               feedback; and
             o Independent third party reviews of its programs.
     o In the past, BTS systematically solicited feedback on overall customer
       satisfaction of BTS' products and services through its Customer
       Satisfaction Survey. BTS conducted the last such survey in 1998.
     o Also, in the past, BTS solicited feedback via comment cards, inserted
       in data products. According to BTS officials, the survey and comment
       cards are no longer used as methods for soliciting feedback as a
       result of restrictions stemming from the Privacy Act and Paperwork
       Reduction Act (PRA). BTS officials told us they are working with
       RITA's Chief Counsel's office to reinstitute the use of comment cards.

x Without a systematic process for soliciting feedback on user
satisfaction from its users, BTS cannot ensure that it is has a full
picture of the needs of those users and how well it is meeting those
needs, which, in turn, hinders BTS' ability to make improvements to data
programs that are relevant to users.

Examples of how BTS has solicited feedback from users

o  Examples of how BTS has solicited feedback from users.

How BTS solicited user Examples of how feedback on user solicitation was
solicited for specific BTS feedback on user satisfaction programs

Data users' meetings and x In 2005, BTS engaged freight data users at a
Commodity Flow Survey Conference for workshops held at conferences input
on changes needed in BTS' freight data collection. or training sessions x
In 2004, BTS engaged users of travel data for their input into current
uses of existing

data and future data needs. The results of this data user's meeting were
published in

Data for Understanding Our Nation's Travel: National Household Travel
Survey Conference.

BTS Reference Services x BTS' National Transportation Library Reference
Services Desk tracks and responds to inquiries received by email via BTS'
Web site or by telephone.

Independent third party reviews x In 2003, BTS asked the Transportation
Research Board and Committee on National Statistics of the National
Academies to conduct a review of its major survey programs, including the
(1) National Household Travel Survey, (2) Omnibus Survey Program, and (3)
Commodity Flow Survey. x In 1999, BTS contracted with an external group to
conduct focus groups with congressional staffers, DOT employees,
transportation statistics experts, and employees of non-profits or
associations in the transportation field to (1) gain feedback on BTS'
publications and services and (2) gather insight about how to improve
these tools for the end user.

                    Source: GAO analysis of BTS information.

BTS' method for incorporating feedback on user satisfaction is not
systematic

     o BTS relies on its program managers, who are responsible for managing a
       data collection program and its associated data products, to determine
       how best to address feedback from users in identifying areas of
       improvement and taking steps to address product weaknesses.
     o According to BTS officials, program managers use their professional
       judgment and consider such factors as time and resource constraints
       when deciding whether to incorporate feedback from users in taking
       steps to improve their data products. BTS officials told us that
       program managers are not required-and often do not-report the results
       of how feedback on user satisfaction was considered and whether
       improvements to the data product were made.
     o GAO's Internal Control standards suggest that on-going monitoring
       should take place to ensure that users are satisfied and that policies
       and procedures should be in place to ensure that feedback is evaluated
       so that improvements can be made.
     o Without policies and procedures in place for incorporating feedback on
       user satisfaction, BTS cannot systematically consider feedback and use
       it to make improvements to data products.

BTS' performance measures

        * BTS reports that it measures the overall degree to which products
          and services are useful and responsive to the needs of users by
          tracking two performance indicators-the number of congressional and
          governmental agency contacts regarding BTS information and the
          average number of daily unique visitors to the TranStats data Web
          site:13
             o BTS reports that the number of congressional and governmental
               agency contacts has increased each year, from 84 requests in
               2002, to 287 in 2005. BTS officials attribute this increase to
               their responsiveness to inquiries and they told us that the
               increase indicates that BTS is providing Congress and agencies
               with information that they need, and that they find BTS to be
               a reliable source.
             o BTS reports the average number of daily unique visits to the
               TranStats data site has increased from 565 in 2003, to 813 in
               2005. According to BTS officials, this indicates that the
               information is useful and that awareness among current and
               potential new BTS data users is increasing.
     o We note, however, that these indicators are simple counts and not
       measures of user satisfaction with BTS information.

13 Required by SAFETEA-LU, the TranStats data web site contains hundreds
of transportation-related databases and is made available to the public
via BTS' Web site. 34

DOT's Perspective on Key Issues Related to RITA

     o We met with senior DOT officials-including the Department's Chief of
       Staff, the Assistant Secretary for Budget and Programs and Chief
       Financial Officer, the RITA Administrator, and other officials-to
       discuss the factual information presented in these slides. These
       officials agreed with the information, providing clarifications, which
       we incorporated as appropriate.
     o These officials said that their vision is for RITA to play a key
       role-in concert with the operating administrations-in shaping the
       future of DOT's RD&T activities, although they believe that RITA will
       need significant additional resources, especially technical experts,
       to achieve this vision. They said that OST will rely heavily on input
       from RITA's Planning Council in making its decisions related to the
       RD&T budget requests from the operating administrations. They also
       said that OST will look to RITA to play a lead role in helping realign
       the Department's RD&T activities to focus more closely on key
       cross-modal transportation issues facing the nation, including
       congestion, alternative fuels, freight capacity, and other issues.

Next Steps

     o Prepare final slides, with a short summary of results, for publication
       as a GAO report.
     o Determine if committee staff want any additional work on RITA.

Appendix II: GAO Contact and Staff Acknowledgments

  GAO Contact

Mathew Scire, (202) 512-2834 or [email protected]

  Staff Acknowledgments
  
In addition to the contact named above, key contributors to this report
were Rita Grieco, Assistant Director; Ashley Alley; Carl Barden; Christine
Bonham; Elizabeth Curda; Michelle Dresben; Colin Fallon;
Richard Hung; Sara Ann Moessbauer; Susan Ragland; April Thompson; Deborah
Winters; and Dorothy Yee.

  GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts GAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To

have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

                             Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone:Voice:(202) 512-6000 TDD:(202) 512-2537 Fax:(202)
512-6061

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548

Public Affairs
  
Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

RECYCLED PAPER
*** End of document. ***