[Federal Register Volume 64, Number 181 (Monday, September 20, 1999)]
[Notices]
[Pages 50854-50861]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 99-24363]


-----------------------------------------------------------------------

DEPARTMENT OF TRANSPORTATION

Federal Highway Administration
Federal Transit Administration


Transportation Equity Act for the 21st Century; Guidelines for 
the Evaluation of Operational Tests and Deployment Projects for 
Intelligent Transportation Systems (ITS)

AGENCIES: Federal Highway Administration (FHWA), Federal Transit 
Administration (FTA), Department of Transportation (DOT).

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: This document provides implementation guidance for section 
5204(j)(1) of the Transportation Equity Act for the 21st Century (TEA-
21) 23 U.S.C 502 Note. Guidelines and requirements for the evaluation 
of

[[Page 50855]]

operational tests and deployment projects for ITS ensure the 
objectivity and independence of project evaluators to avoid any real or 
apparent conflict of interest or potential influence on the outcome by 
parties to such tests and projects. They also establish evaluation 
funding levels based on the size and scope of each test or project to 
ensure adequate evaluation. The ITS Joint Program Office (ITS JPO) 
plans to disseminate the TEA-21 Evaluation Guidelines to affected 
program offices within the DOT. Should it be deemed necessary to 
establish any requirements for the evaluation of operational tests and/
or deployment projects, these would be established pursuant to 
rulemaking to be issued in the future.

FOR FURTHER INFORMATION CONTACT: For technical information: Mr. Joe 
Peters, (202) 366-2202, ITS Joint Program Office, (HOIT-1) FHWA. For 
legal information: Mr. Wilbert Baccus, Office of the Chief Counsel, 
FHWA (HCC-32), (202) 366-0780; Ms. Linda Sorkin, Office of the Chief 
Counsel, FTA (TCC-24), (202) 366-1936. All are located at the U.S. 
Department of Transportation, 400 Seventh Street, SW., Washington, DC 
20590. Office hours are from 8 a.m. to 4:30 p.m., Monday through 
Friday, except Federal holidays.

SUPPLEMENTARY INFORMATION:

Electronic Access

    An electronic copy of the document may be downloaded using a modem 
and suitable communications software from the Government Printing 
Office's Electronic Bulletin Board Service at (202) 512-1661. Internet 
users may reach the Federal Register's home page at: http://
www.nara.gov/fedreg and the Government Printing Office's database at: 
http://www.access.gpo.gov/nara. The TEA-21 Evaluation Guidelines may 
also be accessed at the U.S. DOT's ITS home page, through the 
evaluation link, at http://www.its.dot.gov.

Background

    Section 5204(j)(1) of TEA-21, Pub. L. 105-178, 112 Stat. 107, 455 
(1998), states that the Secretary of Transportation (Secretary) shall 
issue guidelines and requirements for the evaluation of operational 
tests and deployment projects for the ITS. This document sets forth 
TEA-21 Guidelines that are required to include provisions to ensure 
objectivity and independence of evaluators so as to avoid real or 
apparent conflicts of interest on the outcome of project evaluations. 
The guidelines are further required to establish evaluation funding 
levels based on the size and scope of each test or project to ensure 
adequate project or operational test evaluations. Should it be deemed 
necessary to establish any requirements for the evaluation of 
operational tests and/or deployment projects, these would be 
established pursuant to rulemaking to be issued in the future.
    The objective of the TEA-21 Evaluation Guidelines is to fulfill the 
requirements imposed on the Secretary in the referenced section by 
accomplishing the following:
    1. Defining the different categories of projects carried out under 
subtitle C;
    2. Defining, in general terms, different types of evaluations to be 
conducted by projects in the categories defined;
    3. Establishing criteria to guide the selection of evaluations to 
be performed;
    4. Defining, in general terms, procedures for ensuring objectivity 
and independence of evaluating organizations;
    5. Defining the funding mechanism to provide project evaluation 
resources;
    6. Providing a general description of the procedures [and 
requirements] that project partnerships can expect when participating 
in the types of evaluations defined;
    7. Providing access through web site address to detailed evaluation 
procedures to include examples of specific evaluation plans, test 
plans, and reports.
    The TEA-21 Evaluation Guidelines are published in the Federal 
Register for informational purposes on our approach to satisfying the 
requirements prescribed in section 5204(j)(1). Specific questions on 
any of the material published in this notice should be directed to the 
appropriate contact person named in the caption FOR FURTHER INFORMATION 
CONTACT.

Authority: 23 U.S.C. 315; sec. 5204(j), Pub. L. 105-178, 112 Stat. 
455 (1998) 23 U.S.C. 502 Note.

    Issued on: September 13, 1999.
Kenneth R. Wykle,
Federal Highway Administrator.
Gordon J. Linton,
Federal Transit Administrator.
    The text of the TEA-21 Evaluation Guidelines is presented as 
follows:

United States Department of Transportation
Intelligent Transportation Systems
Evaluation Guidelines for the Transportation Equity Act for the 21st 
Century
Operational Tests and Deployment Projects

I. Introduction

A. Background

    The enactment of the Transportation Equity Act for the 21st Century 
(TEA-21) [Pub. L. 105-178, 112 Stat. 107 (1998)] has expanded the focus 
of the Intelligent Transportation Systems (ITS) program from one of 
research and operational tests to one that now includes deployment of 
ITS across the Nation. Subtitle C (Intelligent Transportation Systems 
Act of 1998) under title V of TEA-21 authorizes investment to 
accelerate the rate at which intelligent transportation systems are 
incorporated in the Nation's surface transportation network to improve 
transportation safety and efficiency, and to reduce costs and negative 
impacts on communities and the environment. The effective 
implementation of the deployment and integration activities specified 
in subtitle C will rely heavily on active and rigorous evaluations of 
those activities. Recognizing the evolving nature of the ITS program, 
Congress included a mandate in TEA-21 which requires the Secretary of 
Transportation (Secretary) to issue guidelines and requirements for the 
evaluation of operational tests and deployment projects carried out 
under the program. A prerequisite for the continuing support of 
decision makers addressing policy and investment issues will be a clear 
understanding of ITS system effectiveness. At the Federal level, the 
Government Performance and Results Act [Pub. L. 103-62, 107 Stat. 285 
(1993)] has established a formal process for program and budget 
planning. ITS benefits information will be needed for assessing the 
efficacy of Federal investment in ITS. At the project level, agencies 
and partners in public-private project initiatives require the means to 
monitor costs and system effectiveness on a continuing basis to support 
management of systems and operations.

B. Legislative Requirements

    The Congress has recognized the critical role of ITS evaluation in 
section 5204(j)(1)(A) of subtitle C which prescribes that:

Subparagraph (A):

    In General.--The Secretary shall issue guidelines and requirements 
for the evaluation of operational tests and deployment projects carried 
out under this subtitle.

Subparagraph (B):

    Objectivity and Independence.--The guidelines and requirements 
issued under Subparagraph (A) shall include provisions to ensure the 
objectivity and

[[Page 50856]]

independence of the evaluator so as to avoid any real or apparent 
conflict of interest or potential influence on the outcome by parties 
to any such test or deployment project or by any other formal 
evaluation carried out under this subtitle.

Subparagraph (C):

    Funding.--The guidelines and requirements issued under Subparagraph 
(A) shall establish evaluation funding levels based on the size and 
scope of each test or project to ensure adequate evaluation of the 
results of the test or project.

C. Objective

    The objective of this document is to fulfill the ``guidelines'' 
requirement imposed on the Secretary by TEA-21, title V, subtitle C, 
section 5204(j)(1)(A), (B), and (C) as referenced above.

D. Purposes of Evaluation Guidelines

    In order to satisfy the applicable title V subtitle C criterion 
imposed by TEA-21, guidelines must be issued which accomplish the 
following purposes:
    1. Define different categories of projects carried out under 
subtitle C;
    2. Define, in general terms, different types of evaluations to be 
conducted by projects in the categories defined;
    3. Establish criteria to guide the selection of evaluations to be 
performed;
    4. Define, in general terms, procedures for ensuring objectivity 
and independence of evaluating organizations;
    5. Define the funding mechanism to provide resources for 
evaluations;
    6. Provide a general description of the procedures [and 
requirements] that project partnerships can expect when participating 
in the types of evaluations defined;
    7. Provide access through web site address to detailed evaluation 
procedures including examples of specific evaluation plans and reports. 
Guidelines in this document do not address detailed evaluation 
procedures.

II. Terms of Reference

    The establishment of clearly understood guidelines for evaluation 
of projects funded with Federal ITS funding requires identification of 
the categories of projects specified in the law. Similarly, 
understanding of the Department's concept for conducting project 
evaluations requires differentiation between types of evaluation.

A. Project Categorization

    Projects to be carried out with Federal ITS funds are identified as 
follows:
    1. Operational Tests--Section 5207(c) specifies the conduct of 
operational tests of intelligent vehicles and intelligent 
infrastructure systems. Section 5207(c) further specifies that 
operational tests conducted under this section shall be designed for 
the collection of data to permit objective evaluation of the results of 
the tests, derivation of cost-benefit information that is useful to 
others contemplating deployment of similar systems, and the development 
and implementation of standards.
    2. ITS Deployment Program--The ITS Deployment Program provides 
funding for two major project categories specified in sections 5208 and 
5209. The two components are the ITS Integration Program and the 
Commercial Vehicle Intelligent Transportation Infrastructure Deployment 
Program, commonly known as the Commercial Vehicle Information Systems 
and Networks (CVISN) Program.
    a. Intelligent Transportation System Integration Program--Section 
5208(a) specifies projects to be carried out under the Intelligent 
Transportation System Integration Program in metropolitan and rural 
areas. These projects comprise one major component of the ITS 
Deployment Program defined above. The ITS Integration Program provides 
Federal funding for the integration of multi-modal ITS components in a 
variety of settings, including large regional areas (for example, 
Statewide, multi-State, or multi-city), metropolitan areas, non-
metropolitan areas, and rural areas. ITS integration projects should 
improve transportation efficiency; promote safety; enhance transit 
integration; improve paratransit/demand-responsive transit operations, 
including operations of health and human services providers; improve 
traffic flow, including the flow of intermodal freight at ports of 
entry; reduce emissions of air pollutants; improve traveler 
information; promote tourism; enhance alternative transportation modes; 
or support improved transportation systems operations, management and 
maintenance.
    As part of the Intelligent Transportation System Integration 
Program, section 5208(g), Corridor Deployment and Coordination, 
requires the Secretary to encourage multistate cooperative agreements, 
coalitions, or other arrangements intended to promote regional 
cooperation, planning, and shared project implementation for 
intelligent transportation system projects. There are two areas of 
implementation: Great Lakes ITS Implementation and Northeast ITS 
Implementation.
    (a) Great Lakes Implementation--Section 5208(g)(2)(A): The 
Secretary is required to make grants under this subsection to the State 
of Wisconsin to continue ITS activities in the corridor serving the 
Greater Milwaukee, Wisconsin; Chicago, Illinois; and Gary, Indiana, 
areas initiated under the Intermodal Surface Transportation Efficiency 
Act of 1991 (ISTEA) [Pub. L. 102-240, 105 Stat. 1914 (1991)] and other 
areas in the State.
    (b) Northeast ITS Implementation--Section 5208(g)(3)(A): The 
Secretary is required to make grants under this subsection to the 
states to continue ITS activities in the Interstate Route I-95 Corridor 
in the northeastern United States initiated under ISTEA.
    b. Commercial Vehicle Intelligent Transportation System 
Infrastructure Deployment--Section 5209 specifies that the Secretary 
shall carry out a comprehensive program to deploy intelligent 
transportation systems that improve safety and productivity of 
commercial vehicles and drivers, while reducing costs of commercial 
vehicle operations and regulatory requirements. The term ``Commercial 
Vehicle Information Systems and Networks'' is defined (section 5211) as 
the information systems and communications networks that support 
commercial vehicle operations. Funding in section 5209 is used to 
support CVISN projects which comprise the second major component of the 
ITS Deployment Program defined above. CVISN objectives focus on 
advancing technological capabilities and promoting the deployment of 
intelligent transportation system applications to commercial vehicle 
operations, including commercial vehicle, commercial driver, and 
carrier-specific information systems and networks. Priority areas for 
projects are those that encourage multi-state cooperation and corridor 
development; improve the safety of commercial vehicle operations; 
increase the efficiency of regulatory inspection processes to reduce 
administrative burdens; improve the efficiency of enforcement efforts; 
advance electronic processing of registration, driver licensing and 
fuel tax information, inspection and crash data; promote communication 
of information among the States; and, enhance safe passage of 
commercial vehicles across international borders.

B. Evaluation Categorization

    For the purpose of the guidelines, there are two categories of 
project evaluations.

[[Page 50857]]

    1. Self-Evaluations--In the interests of accepted sound management 
practice, it is expected that ITS Deployment Program participants 
should conduct locally executed and funded evaluations under the 
auspices of the project partners. These self-evaluations, also 
identified as local evaluations, incorporate certain minimum evaluation 
and reporting requirements. Cross-cutting assessments of these local 
evaluations will be conducted by the ITS Joint Program Office (ITS JPO) 
at Headquarters (HQ), U.S. DOT, and will include gathering data and 
dissemination of results.
    2. National Evaluations--National evaluations are formal, in-depth, 
independently conducted evaluations of operational tests of intelligent 
infrastructure systems and selected projects carried out under the ITS 
Deployment Program. In the case of selected ITS Deployment Program 
projects, these evaluations will supplement and expand on the 
activities of self-evaluations. The added resources available for 
national evaluations will facilitate assessment of quantitative 
performance measures and other aspects of project evaluation difficult 
to pursue with the potentially limited resources assigned for self-
evaluations. National evaluations will be conducted under the auspices 
of U.S. DOT, and will be closely monitored by a designated U.S. DOT 
representative. The U.S. DOT has established the Intelligent Vehicle 
Initiative (IVI) to improve the safety and efficiency of motor vehicle 
operations by reducing the probability of motor vehicle crashes. 
Evaluations of IVI field operational tests will be conducted under the 
auspices of the IVI Program Manager in accordance with procedures 
defined within the IVI Program and not associated with requirements for 
national evaluations discussed in these guidelines.
    Features and requirements of these categories are described below 
in these guidelines.

III. Concept for Project Selection and Evaluation Funding

A. National Evaluations

    1. Operational Tests--All operational tests of intelligent 
infrastructure systems carried out under section 5207 shall be 
administered formal, national evaluations under the auspices of the ITS 
JPO Program Assessment Coordinator.
    2. ITS Deployment Program--Projects carried out under the ITS 
Deployment Program will be reviewed as follows:
    (a) ITS Integration Program--Projects selected for funding under 
section 5208 shall be reviewed, and a limited number will be selected 
for national evaluations in accordance with appropriate criteria 
described in these guidelines.
     Corridor Development and Coordination--Projects resulting 
from grants made to the State of Wisconsin to continue activities in 
the corridor serving Milwaukee, Wisconsin; Chicago, Illinois; and Gary, 
Indiana, in accordance with section 5208(g)(2)(A) and projects 
resulting from grants made to the I-95 Northeast ITS Priority 
Corridors, in accordance with section 5208(g)(3)(A), may be reviewed 
for incorporation in a national evaluation as determined by the 
Department's evaluation needs in corridor settings.
    (b) Commercial Vehicle ITS Infrastructure Deployment--A limited 
number of CVISN deployment sites funded under section 5209 will be 
selected for national evaluations in accordance with appropriate 
criteria described in these guidelines.

B. Self-Evaluations

    All project partnerships are expected to perform locally conducted 
self-evaluations.

C. Funding for Project Evaluations

    In order to fulfill the mandate specified in section 5204(j)(1)(C) 
that these guidelines ``shall establish evaluation funding levels based 
on the size and scope of each test or project that ensure adequate 
evaluation of the results of the test or project,'' the following 
procedures apply:
    1. Operational Tests--Funding for evaluations of operational tests 
of intelligent infrastructure systems will be provided by the ITS JPO.
    2. ITS Deployment Program--(a) Projects funded under the 
Intelligent Transportation System Integration Program (section 5208) 
and the Commercial Vehicle Intelligent Transportation System 
Infrastructure Deployment Program (section 5209) selected for national 
evaluations will use a pooled funding mechanism. During each year 
authorized by TEA-21, two percent of the amount authorized for the ITS 
Deployment Program will be placed into a deployment evaluation fund. 
National evaluations for selected projects will be funded by this 
account.
    (b) All projects will fund locally conducted self-evaluations from 
project resources.

D. Selection Criteria for National Evaluation

1. Intelligent Transportation System Integration Program
    Projects carried out under section 5208 are designed to accelerate 
the integration and interoperability of intelligent transportation 
systems in metropolitan and rural areas. Projects resulting from 
corridor grants may also be reviewed for inclusion in a national 
evaluation. Projects selected for funding should support the goals of 
the program as defined in section 5208.
    Projects selected for funding will be reviewed by the ITS Program 
Assessment Working Group (ITS PAWG) chaired by the ITS JPO Program 
Assessment Coordinator. The ITS PAWG is comprised of representatives 
from the modal administrations at HQ, U.S. DOT, and field 
representatives from the Federal Highway Administration (FHWA) Resource 
Centers and Division Offices, to include the Office of Technology 
Evaluation and Deployment within the Office of Motor Carrier and 
Highway Safety. Federal Transit Administration field representation is 
included as well.
    The ITS PAWG will consider projects for national evaluation through 
successive iterations designed to narrow the list of candidate projects 
consistent with available funding and the ITS Program's most compelling 
information needs. While the specific considerations for project 
selection may vary in any given year of TEA-21 authorization, the 
criteria described below will guide the review process.
    a. Major consideration will be given to ITS areas for which 
insufficient knowledge was developed during the operational test 
experience under ISTEA authorization.
    Metropolitan ITS Integration Program projects will be assessed for 
their potential to provide benefits-related data in areas where 
evaluation data are still needed.
    b. Rural ITS Integration Program projects will be assessed for 
their potential to provide benefits-related data.
    It is anticipated that, over the life of TEA-21, the areas 
reflecting shortfalls in ITS Program evaluation needs will vary. The 
ITS Program reserves the flexibility to develop a working document with 
annual updates of these areas of shortfall which would be used to guide 
the selection process. These areas will be addressed in annual 
Departmental project guidance documents.
2. CVISN Deployment Projects
    The FHWA's State CVISN Level 1 deployment strategy consists of 
three key steps: Planning; Design; and, Implementation and Deployment.
    Step 1, Planning, includes participation in two ITS/Commercial 
Vehicle Operations (ITS/CVO) training

[[Page 50858]]

courses (Introduction to ITS/CVO and ITS/CVO Technical Project 
Management for Non-Technical Managers) and the development of an ITS/
CVO State business plan. This step is essential to promote ITS/CVO 
awareness and coalition building among the State agencies involved in 
CVO and with industry.
    The focus of Step 2, Design, is for the State to establish its 
CVISN project team, including at a minimum a CVISN project manager and 
a system architect. Once these individuals have been selected, a State 
can participate in the Understanding ITS/CVO Technology training course 
and in three CVISN workshops. These activities will assist the State in 
developing its CVISN Project Plan and Top-Level Design.
    Step 3 is the Implementation and Deployment of CVISN Level 1 
capabilities.
    Only CVISN sites selected for Step 3 funding will be considered for 
national evaluation. The three elements of Level 1 capabilities subject 
to national evaluation are:

 Safety Information Exchange,
 Credentials Administration, and
 Electronic Screening.

    During each year of TEA-21 authorization, the ITS PAWG will convene 
its CVO Subcommittee comprised of subject matter experts from FHWA's 
Office of Technology Evaluation and Deployment within the Office of 
Motor Carrier and Highway Safety. Based on the ITS/CVO Program's 
information needs about Level 1 capabilities in different operating 
environments, a limited number of State CVISN deployment sites engaged 
in Step 3 (Implementation and Deployment) activities will be selected 
for national evaluation.

IV. Selection of Evaluating Organizations

A. General

    Subtitle C, section 5204(j)(1)(B) requires that these guidelines 
include provisions to ensure objectivity and independence of the 
evaluator so as to avoid any real or apparent conflict of interest or 
potential influence on the outcome by parties to any such test or 
deployment project or by any other formal evaluation carried out under 
this subtitle.

B. Provisions for National Evaluations

    The required provisions are outlined as follows:
    1. Organizations demonstrating technical qualifications are 
eligible for selection to perform a national evaluation of designated 
operational tests or deployment projects.
    2. In the process of submitting necessary documentation 
demonstrating qualifications to perform a national evaluation, 
interested organizations should include a statement certifying 
commitment to the conduct of a completely objective and independent 
evaluation, and its commitment to ensuring any subcontracting 
organizations will adhere to that standard. It is expected that this 
would include a provision for voluntary recusal from evaluating 
selected aspects of, or technologies involved in, the project to be 
evaluated, if appropriate.
    3. For national evaluations conducted under the auspices of U.S. 
DOT, the ITS JPO Program Assessment Coordinator will convene selected 
members of the ITS PAWG to participate in the selection of an 
evaluating organization with consultations as follows:
    a. Operational Tests--For operational tests of intelligent 
infrastructure systems, the ITS JPO Program Assessment Coordinator, in 
consultation with project partners and the ITS PAWG, will select the 
evaluating organization. For IVI field operational tests, the selection 
of organizations will be conducted in accordance with procedures 
established by the IVI Program Manager.
    b. ITS Deployment Program--Participants in the evaluating 
organization selection process are as follows:
    (1) Intelligent Transportation System Integration Program--For 
projects in this category selected for national evaluations, the ITS 
JPO Program Assessment Coordinator in consultation with project 
partners will conduct a selection process. The selected evaluator will 
receive direction and oversight by a U.S. DOT representative, and will 
be required to document compliance with the ``OBJECTIVITY and 
INDEPENDENCE'' requirements imposed by section 5204(j)(1)(B) of 
subtitle C, and will forward authenticated certifications described in 
IV.B.2 of these guidelines to the designated U.S. DOT representative. 
This representative, in coordination with the ITS JPO Program 
Assessment Coordinator, will conduct an expeditious review of selected 
organizations prior to completion of contract arrangements.
    (2) CVISN Project Evaluations--The selection of an evaluation 
organization for the conduct of national evaluations of selected CVISN 
project sites will be led by the ITS JPO Program Assessment Coordinator 
and the ITS PAWG representative from the Office of Technology 
Evaluation and Deployment in the Office of Motor Carrier and Highway 
Safety who will convene a CVO subcommittee of the ITS PAWG. This 
subcommittee will be comprised of the ITS JPO Commercial Vehicle 
Operations Coordinator and subject matter experts on the staff of the 
Office of Motor Carrier and Highway Safety, HQ, FHWA.

C. Provisions for Self-Evaluations

    The guidelines for the conduct of self-evaluations in ITS 
Deployment Program projects include the following: (1) The evaluations 
should be conducted under the auspices of the project partners; (2) The 
partners should form evaluation organizations; and, (3) The process for 
forming evaluation organizations should strive to adhere to the 
objectivity and independence principles cited in section 5204(j)(1)(B) 
of subtitle C.

V. Evaluation--Definition, Process, and Guidelines

A. General--The purposes of this section are to accomplish the 
following:

     Define evaluation as a management process;
     In general terms, differentiate between procedures and 
expectations to be encountered in national evaluations and self-
evaluations;
     Convey reporting requirements.
    These guidelines are not intended to provide detailed procedures 
for the conduct of evaluations. Extensive documentation addressing 
detailed procedures, to include examples of evaluation plans and 
reports will be available at the ITS JPO web site. (See Section V.G.).

B. Definition of Evaluation

    Evaluation is the reasoned consideration of how well project goals 
and objectives are being achieved. The primary purpose of evaluation is 
to cause changes in the project so that it eventually meets or exceeds 
its goals and objectives. Evaluation is an essential ingredient to good 
project management. Evaluations can be qualitative and quantitative; 
however, the best evaluations employ the combination of qualitative and 
quantitative information that compare and contrast converging, non-
converging, and diverging evidence to result in a complete diagnosis. 
The most effective evaluations occur when goals and objectives are 
explicitly stated, are measurable, and are agreed to by all project 
partners.
    Evaluation should be considered as part of the project development 
process that iterates across stages of strategy

[[Page 50859]]

formulation, detailed planning, system design and implementation, data 
collection, data analysis, and reporting of results. Evaluations should 
be performed by a party who has had no vested interest or stake in the 
project itself.
    Independence of the evaluator does not mean uninvolvement with the 
project. Key roles of the evaluator requiring early evaluator 
involvement are: (1) Identification of key stakeholder partners; (2) 
eliciting from the partners a meaningful set of goals and objectives 
for the project and their relative priorities; (3) obtaining insight 
and consensus regarding which measures will best reflect the degree of 
success in achieving prioritized goals; and, (4) communicating changes 
in goals, objectives, and measures as the project evaluation matures. 
Data can be collected either by the partners in the project (as long as 
the independent evaluator maintains some oversight of the process) or 
by the independent evaluator, or by both. The data analysis phase must 
be performed completely independently of the partners; however, draft 
results should be shared to obtain partners' expert insights regarding 
possible flaws in assumptions or errors in analysis. The best 
evaluation results are those that can bring about positive and timely 
changes in system operations or maintenance. In such instances, 
documentation of results may not be as important as the positive 
changes influenced by the results. Nevertheless, documentation has 
tremendous importance to the national interest.
    A significant goal influencing the approach to ITS Program 
evaluation planning is the expansion of the knowledge base among 
transportation community professionals. To the extent that policy 
makers, planners, engineers, and other influence brokers become better 
informed about successful ITS deployment practices, project evaluations 
will make significant contributions. Effective information 
dissemination techniques will be critical. Information drawn from 
interim and final evaluation reports will be summarized in electronic 
media normally accessed by transportation professionals.
    Project evaluation reports will be crucial to the ITS Program 
during the years funded by TEA-21. As the tempo of project deployments 
accelerates, newly formed project partnerships preparing to undertake 
deployments will need any available experience-based information 
relevant to their projects. The need for timely information will 
emphasize the value of interim reports capturing valuable, recent, 
lessons learned.

C. General Evaluation Process

    The following are general steps successfully used to accomplish an 
ITS project evaluation for national or local evaluations:
    1. Form the Evaluation Team. Each of the project partners and 
stakeholders designates one member to participate on the evaluation 
team. The program manager should designate an evaluation team leader. 
In the interests of conducting an effective evaluation, this team 
should interact with the independent evaluator periodically throughout 
the project development and deployment. Experience has demonstrated 
that formation of this team early in the project is essential to 
facilitating evaluation planning along a ``no surprises'' path. 
Participation by every project stakeholder is particularly crucial 
during the development of the ``Evaluation Strategy.''
    2. Develop the Evaluation Strategy. This evaluation strategy 
document includes a description of the project to be evaluated and 
identifies the key stakeholders committed to the success of the 
project. It also relates the purpose of the project to the general goal 
areas of an ITS project.
    Projects deploying intelligent metropolitan or rural infrastructure 
are expected to allocate resources adequate for evaluating the impact 
(or impacts) their projects exert in certain major goal areas which can 
be pursued through deploying and integrating ITS technologies. ITS goal 
areas include:

 Traveler Safety
 Traveler Mobility
 Transportation System Efficiency
 Productivity of Transportation Providers
 Conservation of Energy and Protection of the Environment
 Others as may be appropriate to unique features of a project

    A major purpose of the evaluation strategy document is to focus 
partner attention on identifying which of the above goal areas has 
priority for their project. Partners assign ratings of importance to 
goal areas and evaluation priorities and resources are consequently 
aligned to the prioritized set. This rating process gives partners 
valuable insights regarding areas of agreement and disagreement and 
assists in reconciling differences and bolstering common causes.
    Each of these goal areas can be associated with outcomes of 
deployment that lend themselves to measurement. These outcomes 
resulting from project deployment are identified as measures and have 
been adopted as useful metrics. The association of goal areas and 
measures is depicted as follows:

------------------------------------------------------------------------
          Goal area                             Measure
------------------------------------------------------------------------
Safety.......................   Reduction in the Overall Rate of
                                Crashes.
                                Reduction in the Rate of Crashes
                                Resulting in Fatalities.
                                Reduction in the Rate of Crashes
                                Resulting in Injuries.
Mobility.....................   Reduction in Delay.
                                Reduction in Transit Time
                                Variability.
                                Improvement in Customer
                                Satisfaction.
Efficiency...................   Increases in Freeway and
                                Arterial Throughput or Effective
                                Capacity*.
Productivity.................   Cost Savings.
Energy and Environment.......   Decrease in Emissions Levels.
                                Decrease in Energy Consumption.
------------------------------------------------------------------------
*A discussion of the distinction between Throughput and Effective
  Capacity is included in the ITS Evaluation Resource Guide referred to
  in Section V.G. at the end of this document.

    The ``few good measures'' in the preceding table constitute the 
framework of benefits expected to result from deploying and integrating 
ITS technologies. While each project partnership will establish its 
unique evaluation goals, the measures serve to maintain the focus of 
goal setting on how the project can contribute to reaping the benefits 
of one or more of the measures. A sample Evaluation Strategy document 
is provided as part of the ITS Evaluation Resource Guide (see Section 
V.G. at the end of this document).

[[Page 50860]]

    3. Develop the Evaluation Plan. After the goals are identified and 
priorities are set by the partners, the evaluation plan should refine 
the evaluation approach by formulating hypotheses. Hypotheses are 
merely ``if-then'' statements about expected outcomes after the project 
is deployed. For example, a possible goal of coordinating 
jurisdictions' signal systems is improving safety by reducing rear-end 
crashes. If the evaluation strategy included this goal, the evaluation 
plan would formulate hypotheses that could be tested. In this case, one 
hypothesis might be, ``If jurisdictions coordinate signal timing, then 
rear-end collisions will be significantly reduced at intersections at 
jurisdictional boundaries.'' An even more aggressive hypothesis might 
suggest that such collisions would be reduced by ten percent. The 
evaluation plan identifies all such hypotheses and then outlines the 
number of different tests that might be needed to test all hypotheses.
    In addition to hypotheses regarding system and subsystem 
performance, the evaluation plan identifies qualitative studies that 
will be performed. The evaluation should address key components of the 
project, such as, (but not limited to):
     Implications of achieving consistency with the National 
ITS Architecture;
     Standards implementation;
     Consumer acceptance;
     Others as appropriate to local considerations;
     Institutional issues.
    Institutional issues require brief elaboration. An area of special 
emphasis in all evaluation endeavors should be the non-technical 
factors influencing project performance. ITS projects conducted under 
ISTEA were profoundly influenced by considerations such as procurement 
practices, contracting policy, organizational structure, and 
relationships among major participants such as prime contractors and 
their subcontractors. The transportation community stands to reap 
significant benefit from understanding how the varied range of non-
technical factors impacts directly on traditional project performance 
parameters, such as, cost, schedule, and final functionality.
    As these critical aspects of the project are addressed, the value 
of the evaluation increases in proportion to its ability to produce 
lessons learned that can improve project performance. Thus, the 
measures serve as the foundations of the evaluation as project 
implementation seeks the best mix of approaches to ensure achieving 
some level of benefits described by the measures.
    To the extent that projects define evaluation goals derived from 
one or more of the few good measures and document the impacts of 
deployment on transportation performance in their communities, they 
will have made significant contributions to the ITS Program and the 
Nation. A sample Evaluation Plan document is provided as part of the 
ITS Evaluation Resource Guide (see Section V.G. at the end of this 
document).
    4. Develop one or more Test Plans. A test plan will be needed for 
each test identified in the evaluation plan. A test plan lays out all 
of the details regarding how the test will be conducted. It identifies 
the number of evaluator personnel, equipment and supplies, procedures, 
schedule, and resources required to complete the test. A sample test 
plan is provided as part of the ITS Evaluation Resource Guide (see 
Section V.G. at the end of this document).
    5. Collect and analyze data and information. This step is the 
implementation of each test plan. It is in this phase where careful 
cooperation between partners and evaluators can save money. By early 
planning, it is possible to build into the ITS project capabilities for 
automatic data collection. Such data collection can be used by partners 
after the evaluation is completed to provide valuable feedback with 
regard to the performance of the system. Such feedback can help in 
detecting system failures and to improve system performance.
    6. Document strategy, plans, results, conclusions, and 
recommendations in a Final Report. A sample Final Report document is 
provided as part of the ITS Evaluation Resource Guide (see Section V.G. 
at the end of this document).

D. Self-Evaluation Process

1. ITS Integration Program Evaluation Procedures
    To the extent that resources allow, project partners should strive 
to follow the General Evaluation Process outlined in Section V.C. of 
this document. Partners should also utilize the ITS Evaluation Resource 
Guide described in Section V.G. at the end of this document.
    Projects conducting self-evaluation should respond to a minimum of 
two reporting requirements.
    a. In an attempt to satisfy significant and critical data needs in 
the ITS Program, projects should collect and document cost accounting 
data to include acquisition, life-cycle, and operations and maintenance 
costs capturing both start-up and sustaining cost factors. Reporting of 
cost data will be solicited annually for the duration of the deployed 
system's (or systems'') life cycle(s). Cost data collection guidelines 
are provided as part of the ITS Evaluation Resource Guide (see Section 
V.G. at the end of this document).
    b. Additionally, self-evaluations should result in one or more of 
the following efforts:
     Evaluating institutional issues associated with achieving 
cooperation among public sector agencies and documenting how they were 
overcome.
     Providing a brief lessons learned report on the technical 
and institutional issues encountered in integrating ITS components.
     Providing an evaluation report on the lessons learned in 
employing innovative financing or procurement and/or public-private 
partnering techniques.
     Producing a lessons learned report on the experiences, 
challenges, and approaches used in achieving consistency with the 
National ITS Architecture and/or implementation of ITS standards.
     Producing a case study on the planning process used to 
achieve integration into an approved plan and program developed under 
an area-wide (statewide and/or metropolitan) planning process which 
also complies with applicable State air quality implementation plans.
     Providing the appropriate metropolitan planning process 
with data generated by ITS technologies and services, and provide a 
report on plans or intentions for archiving the data and using it.
2. Commercial Vehicle Intelligent Transportation System Infrastructure 
Deployment Projects
    All projects participating in the Commercial Vehicle Intelligent 
Transportation System Infrastructure Deployment Program are expected to 
accomplish the following:

     Document lessons learned in the areas of:
    --institutional issues
    --technical challenges
    --employment of innovating financing
    --public-private partnering
    --achieving consistency with the National ITS Architecture and 
implementation of ITS Standards.
 Document benefits data.
 Collect cost data, especially with regard to operations and 
maintenance cost factors, and briefly address their implications for 
sustaining the project.

[[Page 50861]]

E. National Evaluations

1. Initial Procedures for Operational Tests of Intelligent 
Infrastructure
    Upon selection of the organization to conduct a national evaluation 
of a designated ITS operational test of intelligent infrastructure, the 
ITS JPO Program Assessment Coordinator will establish the initial 
communications between involved parties. The detailed procedures for 
the conduct of the evaluation and the scope of the evaluating 
organization's tasks will be defined in accordance with procedures 
established by the ITS JPO Program Assessment Coordinator.
2. Requirements for ITS Integration Program Projects
    During the annual project definition and proposal process, 
participants in the ITS Integration Program will be offered the 
opportunity to commit to cooperate with evaluators in the event of 
selection for a national evaluation. This commitment includes 
participation in evaluation planning and in-progress reviews to ensure 
a consensus-based, successfully implemented national evaluation as 
described in sections V.B. and V.C. of these guidelines.
3. Requirements for CVISN Deployment Projects
    During the annual project definition and proposal process, 
participants in the Commercial Vehicle Intelligent Transportation 
System Infrastructure Program will be offered the opportunity to commit 
to cooperate with evaluators in the event of selection for a national 
evaluation. This commitment includes participation in evaluation 
planning and in-progress reviews to ensure consensus-based, 
successfully implemented national evaluations.
4. Timing Considerations for ITS Deployment Program Projects Selected 
for National Evaluations
    Participants in ITS Deployment Program projects selected for 
national evaluations may experience a time delay between receipt of 
notification for project funding and, in the event of selection for 
national evaluation, notification of such selection. Upon notification 
of project funding approval, the project participants should proceed 
with the preparatory steps required for evaluation. The preparatory 
measures will lay the foundation for an effective self-evaluation. In 
the event of selection for a national evaluation, that process will 
build on this foundation. An evaluation team should be formed and an 
evaluation strategy, based upon the example in the ITS Evaluation 
Resource Guide, should be developed.
    It is anticipated that U.S. DOT notification of selection for 
national evaluation will be accomplished prior to a project's 
development of an evaluation plan. This will facilitate coordination 
between the independent evaluating organization and the project 
partners in proceeding with the development of a consensus-based 
evaluation plan.

F. Reporting Requirements

    This section prescribes reporting procedures for the categories of 
evaluations.
    1. Projects conducting self-evaluations in the ITS Integration 
Program are expected to produce: (1) an annual cost report based upon 
guidelines in the ITS Evaluation Resource Guide; and, (2) a final 
evaluation report. Projects conducting self-evaluations of the CVISN 
are expected to produce a final report. All project partnerships 
conducting self-evaluations are expected to submit two camera-ready 
reproducible copies and one electronic file to the ITS JPO Program 
Assessment Coordinator at: Intelligent Transportation Systems, Joint 
Program Office (HOIT-1), Attn: JPO Program Assessment Coordinator, U.S. 
Department of Transportation, 400 Seventh St., SW., Washington, D.C. 
20590.
    Copies which may be required for other addressees will be defined 
in annual guidance documents transmitting instructions.
    2. Reporting procedures in national evaluations will be defined in 
the appropriate documentation governing the contract entered into by 
the evaluating organization.

G. References

    In lieu of incorporating detailed procedural guidance for the 
conduct of evaluations in this document, an ITS Evaluation Resource 
Guide has been developed. This comprehensive resource for supporting 
evaluation planning is accessible at the ITS JPO web site (http://
www.its.dot.gov) through the Program Assessment/Evaluation Link.

[FR Doc. 99-24363 Filed 9-17-99; 8:45 am]
BILLING CODE 4910-22-P