Federal Motor Carrier Safety Administration: Education and
Outreach Programs Target Safety and Consumer Issues, but Gaps in
Planning and Evaluation Remain (19-DEC-05, GAO-06-103).
The Federal Motor Carrier Safety Administration (FMCSA) is
responsible for improving commercial vehicle safety and uses
education and outreach as part of its efforts. The House report
accompanying the fiscal year 2005 Department of Transportation
(DOT) appropriations bill asked GAO to report on FMCSA's
education and outreach programs to the House and Senate
Committees on Appropriations. GAO (1) describes FMCSA's education
and outreach programs and how they relate to FMCSA's goals (2)
identifies the extent to which FMCSA has evaluated its education
and outreach programs and (3) describes the extent to which
FMCSA's education and outreach programs are effective.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-103
ACCNO: A43537
TITLE: Federal Motor Carrier Safety Administration: Education
and Outreach Programs Target Safety and Consumer Issues, but Gaps
in Planning and Evaluation Remain
DATE: 12/19/2005
SUBJECT: Education program evaluation
Highway safety
Motor carriers
Motor vehicle safety
Program evaluation
Program management
Safety regulation
Strategic planning
Transportation safety
Safety Belt
Commercial Motor Vehicle Safety Belt
Partnership
Household Goods Program
New Entrant Program
Non-Entrant Outreach Program
Passenger Carrier Safety
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-103
* Report to Congressional Committees
* December 2005
* FEDERAL MOTOR CARRIER SAFETY ADMINISTRATION
* Education and Outreach Programs Target Safety and Consumer
Issues, but Gaps in Planning and Evaluation Remain
* Contents
* Results in Brief
* Background
* FMCSA Has Established Several Education and Outreach Efforts but
Needs Specific Links to Goals
* Education and Outreach Programs Address a Range of Safety
and Consumer Protection Concerns and Are Targeted to
Industry and the Public
* New Entrant Program
* Non-Entrant Program
* Commercial Motor Vehicle Safety Belt Program
* Motor Coach Outreach Program
* Household Goods Outreach Program
* Except for the New Entrant Program, Education and Outreach
Funding Represents a Small Portion of FMCSA's Budget
* FMCSA Has Not Clearly Articulated How Its Education and
Outreach Initiatives Will Support Broader Goals
* A Logic Model Provides a Framework for Linking Program
Activities to Desired Results and Goals
* Evaluations of Specific Education and Outreach Programs Impacts
Are Still in Preliminary Stages, but the Lack of a Plan to
Evaluate How New Entrant Safety Audits Improve Knowledge and
Attitudes Raises Concern
* New Entrant Program
* Non-Entrant Program
* Commercial Motor Vehicle Safety Belt Program
* Motor Coach Outreach Program
* Household Goods Outreach Program
* The Effectiveness of Education and Outreach Programs Is Unclear,
but Programs Seem to be Reasonably Designed According to Theories
of Behavior Change and Industry Groups
* Behavior Theory Indicates Education and Outreach Can Be
Effective in Increasing Awareness and Changing Behavior for
Programs that Do Not Have Regulations Governing the Behavior
of Their Target Audiences
* A Link to Enforcement Can Strengthen Education and Outreach
Programs' Ability to Change Behavior
* FMCSA Is Increasing Enforcement in the New Entrant Program
and Acknowledges the Importance of Enforcement to the
Non-Entrant and Safety Belt Programs
* New Entrant Program
* Safety Belt Program
* Non-Entrant Program
* Industry Associations Offer A Mostly Positive View of
FMCSA's Education and Outreach Efforts, but Public Advocacy
Groups Expressed Some Concerns
* Conclusions
* Recommendations for Executive Action
* Agency Comments
* Scope and Methodology
* FMCSA Education and Outreach Programs
* New Entrant Program
* Non-Entrant Program
* Commercial Safety Belt Program
* Motor Coach Outreach Program
* Household Goods Outreach Program
* GAO Contact and Staff Acknowledgments
Report to Congressional Committees
December 2005
FEDERAL MOTOR CARRIER SAFETY ADMINISTRATION
Education and Outreach Programs Target Safety and Consumer Issues, but
Gaps in Planning and Evaluation Remain
Contents
Tables
Figures
December 19, 2005Letter
The Honorable Joe Knollenberg Chairman The Honorable John W. Olver
Ranking Minority Member Subcommittee on Transportation, Treasury, Housing,
and Urban Development, the Judiciary, the District of Columbia, and
Independent Agencies Committee on Appropriations House of Representatives
The Honorable Christopher S. Bond Chairman The Honorable Patty Murray
Ranking Minority Member Subcommittee on Transportation, Treasury, the
Judiciary, Housing and Urban Development, and Related Agencies Committee
on Appropriations United States Senate
In 2003, large trucks represented 3 percent of registered vehicles in the
country, but nearly 12 percent of the people killed in motor vehicle
accidents died in crashes involving large trucks. The Federal Motor
Carrier Safety Administration (FMCSA), within the Department of
Transportation (DOT), is responsible for improving the safety of
commercial vehicle operations-which includes interstate truck and motor
coach (bus) companies-and has set a safety goal to reduce fatalities from
an estimated 2.81 per 100 million truck vehicle miles traveled in 1996 to
no more than 1.65 per 100 million truck vehicle miles traveled by the end
of 2008. Among the ways that FMCSA attempts to achieve this and other
goals is through education and outreach activities.1 For example, the
agency provides information to consumers about motor coach safety records
and what to consider when hiring a moving company. It also provides new
motor carriers with information on safety requirements and conducts safety
audits to review motor carrier compliance with these requirements. In some
cases, education and outreach efforts regarding safety audits may be
reinforced when followed-up with enforcement.
In recent years, we have raised concerns about FMCSA education and
outreach efforts. In 2003, for example, we reported that FMCSA's "Share
the Road Safely" program lacked a clear program strategy and included
activities that were only tenuously linked to program goals, and we
reported that FMCSA had not recently evaluated the program's
effectiveness.2 In addition, our 2001 report on oversight of the household
goods moving industry noted that DOT and FMCSA had made limited efforts to
provide consumer education that would enable the public to be more
informed about hiring a mover.3 Furthermore, the Congress has expressed
concern about how FMCSA's education and outreach programs are meeting
broader goals. In the conference report accompanying the DOT
appropriations bill for fiscal year 2005, FMCSA was asked to report by
April 2005 to the House and Senate Committees on Appropriations on
strategies linking outreach and education program initiatives to each
goal.
The House report accompanying the DOT appropriations bill for fiscal year
2005 asked us to monitor and evaluate FMCSA's education and outreach
programs and to report on the status of these programs to the House and
Senate Committees on Appropriations.4 We (1) describe the scope and nature
of FMCSA's education and outreach programs and how they relate to FMCSA's
goals, (2) identify the extent to which FMCSA has evaluated its education
and outreach programs, and (3) describe the extent to which FMCSA's
education and outreach programs are effective. As discussed with your
staff, we focused on five specific education and outreach programs: (1)
the New Entrant; (2) the Non-Entrant; (3) Commercial Safety-Belt Use; (4)
Motor Coach Selection Outreach; and (5) Household Goods Outreach programs
and did not review the "Share the Road Safely" program, which Congress
transferred from FMCSA to the National Highway Traffic Safety
Administration (NHTSA) in fiscal year 2004, with FMCSA retaining a
supporting role. The recent highway reauthorization legislation5
authorizes funding for this program to both FMCSA and NHTSA and asks us to
review the "Share the Road Safely" program by June 2006.
To describe the scope and nature of FMCSA's education and outreach
programs and FMCSA's intended results for these programs, we discussed
these programs with FMCSA officials at agency headquarters and two field
locations, and we reviewed program materials and documentation as well as
observed a safety audit for the new entrant program. We also analyzed both
the extent to which FMCSA describes how its programs lead to agency goals
and incorporates its education and outreach programs into its strategic
and program planning and performance budgeting. To identify approaches to
describing how agency programs contribute to agency goals, we reviewed our
previous work on government education programs that showed how a program
logic model-a model that links activities to goals by analyzing program
inputs, outputs, and outcomes-illustrates how education and outreach
activities can influence attitudes and behavior, and ultimately contribute
to agency goals. To identify what the agency was doing to evaluate these
programs, we discussed evaluations and evaluation plans with FMCSA
officials at agency headquarters and two field locations. We also reviewed
evaluation materials, including proposals and statements of work for
planned contracts to carry out programs and program evaluations. Finally,
to identify what is known about the effectiveness of the programs, we
conducted a literature review of media based campaigns and reviewed the
structure and evaluation of these campaigns related to improving safety as
well as behavioral theories used in designing the campaigns. We selected
two behavioral theories that were directly applicable to the programs
under this review, which are called the theory of reasoned action6 and the
general deterrence theory.7 We also discussed what is currently known
about the effectiveness of these programs with FMCSA officials and with
representatives of associations that serve the trucking, motor coach, and
household goods moving industries, as well as a law enforcement
association and public interest groups that are involved in motor carrier
safety. Our work was conducted in accordance with generally accepted
government auditing standards. Appendix I provides the details of our
scope and methodology.
Results in Brief
FMCSA has recently established several education and outreach programs
intended to promote motor carrier safety and consumer awareness. Although
FMCSA documents state that education activities are vital to FMCSA's goal
of reducing fatalities, injuries, and crashes, FMCSA has not described how
most of its education and outreach programs are linked to agency goals.
Four of the programs we reviewed accounted for a relatively small portion
of FMCSA's funding-1 percent of the agency's fiscal year 2005 funding.
However, the new entrant program accounted for almost 7.5 percent of
FMCSA's fiscal year 2005 funding. These education and outreach programs
are part of FMCSA's overall approach to encourage safer practices and
better decision making by communicating information to motor carriers,
commercial drivers, and the public. FMCSA's education and outreach
programs are as follows:
o The New Entrant program is designed to inform newly registered motor
carriers (new entrants) about motor carrier safety standards and
regulations to help them comply with FMCSA requirements. FMCSA
disseminates information on safety requirements in its Education and
Technical Assistance Package. In addition, FMCSA, state, or contractor
personnel visit new carriers within their first 18 months of operations
for "safety audits" to explain safety requirements and review
documentation. Although primarily informational, a carrier can fail its
safety audit if it cannot document sufficient compliance to pass at least
four of the six sections of the audit and, as a result, could lose its
operating license.
o A planned Non-Entrant program will focus on identifying carriers
engaging in interstate operations that have not registered with FMCSA as
required, making them aware of the requirements and getting them to
register. This program is still being developed but is expected to include
outreach efforts through truck sales and leasing firms.
o The Commercial Motor Vehicle Safety Belt program is aimed at increasing
safety-belt use among commercial drivers, which is lower than safety-belt
use by passenger vehicle drivers. FMCSA, in partnership with industry,
distributes brochures, posters, and bumper stickers describing the
importance of wearing safety belts. The partnership also distributes its
materials at trucking industry trade shows, through law enforcement
programs, and at truck stops.
o The Motor Coach Outreach program, officially known as Passenger Carrier
Safety, provides information to help organizations and individuals who
hire motor coach (bus) services make safe choices. The primary program
effort is a FMCSA Web site that makes information on motor coach
companies' safety history accessible to the public.
o The Household Goods Outreach program provides advice to help individuals
planning a move make informed decisions on selecting and hiring moving
companies through brochures, publications, and FMCSA's Web site.
Research indicates that the development of education and outreach programs
should include identifying how these programs are expected to change the
target audience's attitudes and behaviors.8 However, FMCSA has not clearly
described how most of its education and outreach programs link expected
changes in attitudes and behavior to broader goals, such as DOT's
strategic objective of reducing transportation-related fatalities.9 In the
conference report accompanying the DOT appropriations bill for fiscal year
2005, Congress requested FMCSA to clarify the link between the agency's
education and outreach programs and program goals. FMCSA's October 2005
response to Congress states that FMCSA considers education and outreach
programs vital in achieving overall DOT and FMCSA safety goals. However,
the agency did not describe how the programs are intended to influence
knowledge, attitudes, and behaviors that will support FMCSA's goals. FMCSA
officials told us that the agency uses logic models in its performance
budget to establish, at a high level, the link between its education and
outreach program and its goals. However, the link FMCSA asserts was not
evident to us for most of the education and outreach programs we reviewed.
For example, FMCSA's fiscal year 2006 performance budget does describe how
the Commercial Motor Vehicle Safety Belt program10 will highlight the
risks of not wearing a safety belt, which FMCSA officials believe should
improve drivers' attitudes toward wearing safety belts, and subsequently
meet the program goal of increasing safety-belt usage 10 percent by 2009.
However, the performance budget does not describe similar links between
other FMCSA education and outreach programs and how those programs help
achieve DOT and FMCSA safety and productivity objectives. This lack of
clarity in FMCSA's planning, budgeting, and program documents relating to
education and outreach programs can make it difficult for stakeholders,
including congressional oversight authorities, to see how program
activities that seek to change attitudes and behavior can relate to agency
goals and whether the programs are appropriately targeted. In contrast,
FMCSA's Research and Technology group has used a logic model to clearly
describe how program activities support agency goals in its strategic
plan. As FMCSA further articulates the links between its education and
outreach programs and its goals, it might consider using a tool such as
the one used by FMCSA's Research and Technology group to make those
relationships clear.
FMCSA has begun some evaluations of its education and outreach programs.
Since most of these programs are relatively new and most evaluations are
still being planned, little information on results is currently available.
Therefore, FMCSA has limited information on program effectiveness,
particularly for the New Entrant program. FMCSA's evaluation related
activities and gaps in these activities are described below:
o For the New Entrant program, FMCSA conducted a preliminary comparison of
crash rates for new entrants who had received safety audits during the
first part of 2003 with new entrants who had not received safety audits
and found little difference in the crash rates of the two groups. In
addition, FMCSA is planning to conduct an evaluation study beginning in
2008 to examine new entrants' safety performance. Although the program has
been in place for over 2 years, FMCSA has not evaluated and has not
developed plans to evaluate the New Entrant program, specifically whether
the information package or safety audits effectively communicate
information to new entrants, making it difficult to determine the
effectiveness of that program in improving new motor carriers' knowledge
of safety requirements and ultimately improving safety.
o For the planned Non-Entrant program, FMCSA is planning to hire a
contractor to identify the target population of non-entrants, measure if
the outreach approaches it implements are reaching the target audience
through focus group studies, and measure changes in knowledge and
attitudes and changes in behavior through surveys.
o For the Commercial Motor Vehicle Safety Belt program, FMCSA is
installing software to monitor Web site hits, planning focus groups with
motor carrier operators on how safety belt marketing material affected
their awareness of the importance of using safety belts, and planning to
annually measure safety-belt usage by commercial vehicle drivers.
o For the Motor Coach Outreach program, FMCSA uses a web survey to track
exposure to information provided. FMCSA also expects to evaluate the
program under a contract it plans to award, which will support surveys and
other evaluations of its education and outreach programs.
o For the Household Goods Outreach program, FMCSA has contracted with the
General Services Administration to measure the effectiveness of its
booklet on moving tips in changing consumer knowledge by surveying the
target audience. It also expects to evaluate the program under the planned
survey contract.11
It is difficult to determine the overall effectiveness of FMCSA's
education and outreach efforts at this time; however, on the basis of
behavior theory and our discussions with industry and public safety
groups, the design of FMCSA's programs appears to be reasonable for
contributing to commercial motor vehicle safety. Little information on
effectiveness is available, since FMCSA has not completed many evaluations
of its programs, and so it cannot be sure of the extent to which target
audiences have received the information and intend on changing their
behavior. However, two of FMCSA's education and outreach programs and
marketing materials we reviewed appear to follow theories and research
regarding media campaigns that are intended to influence decision making
about safety. The Household Goods Outreach and Motor Coach Outreach
programs-targeted toward consumers-focus on changing consumer attitudes by
providing information about potential consequences of their actions. For
example, the Motor Coach Outreach program conveys information on the
safety history of motor coach companies. As a result, after considering
this information consumers may change their behavior and select a motor
coach company with a good safety record. In contrast, research and
behavior theory described in several studies we reviewed suggest that
industry-the target audiences of the Safety Belt, New Entrant, and
Non-Entrant programs-may be more likely to change behaviors when exposed
to both education and enforcement. FMCSA's program documentation indicates
enforcement is part of the agency's efforts to increase safety-belt usage
and will be part of the agency's efforts to decrease the number of
non-entrants. In part because there are few sanctions for carriers who
fail certain portions of the safety audit, FMCSA is now increasing the
enforcement associated with its safety audits by making it more difficult
to pass the audit and requiring carriers to correct deficiencies.
Currently, a carrier can pass its safety audit even if it fails two of the
six sections of the audit. We found, for example, that over the past 2
years, about 40 percent of the carriers failed the "driver" section of
their safety audits, despite passing the audit overall.12 Finally, motor
carrier association officials whom we spoke with stated that, in their
view, FMCSA is doing several positive things in its education and outreach
activities. In addition, officials of one public safety group suggested
that FMCSA followed reasonable approaches in starting its education and
outreach efforts, so implementation in theory should be effective;
however, the officials would like to see more evaluations on program
effectiveness in the future to help FMCSA refine its programs. Another
safety group we spoke with emphasized FMCSA's need for stronger
enforcement in conjunction with education and outreach programs.
Our study found that for most of the FMCSA's education and outreach
programs we reviewed, it is unclear how the activities link to program and
agency goals. In addition, although FMCSA plans to conduct a study
beginning in 2008 of its New Entrant program, the agency has not evaluated
the effectiveness of the education and outreach of this program. To more
clearly describe and better evaluate FMCSA education and outreach
programs, we are recommending that the Secretary of Transportation direct
the Administrator of FMCSA to take two actions: (1) ensure that the agency
describes and documents how education and outreach program activities link
to and support broader program and agency goals in a planning, program, or
budget document that is available to the public and (2) evaluate the
effectiveness of the education and outreach component of the New Entrant
program, assessing the extent to which the Education and Technical
Assistance Package and safety audits are helping new carriers learn and
understand FMCSA requirements. We provided a draft of this report to DOT
for its review and comment. In responding to a draft of this report, DOT
and FMCSA officials, through the DOT liaison, provided oral comments.
Agency officials disagreed with GAO's characterization that FMCSA has not
linked its education and outreach programs with its overall goals. We
acknowledge the FMCSA comments in our report. Because we did not find this
link evident for most of the education and outreach programs, we retained
our recommendation with some clarification. The officials also provided
technical comments, which we incorporated as appropriate.
Background
FMCSA, established as a separate administration under DOT by the Motor
Carrier Safety Improvement Act of 1999 is responsible for improving the
safety of commercial vehicle operations on the nation's highways. FMCSA is
engaged in several programs and activities to carry out its mission,
including developing and enforcing Federal Motor Carrier Safety
Regulations, administering Motor Carrier Safety Assistance Program grants
to states, regulating interstate household goods movers, and performing
education and outreach. FMCSA operates through its headquarters in
Washington, D.C.; four regional service centers; and division offices
located in all 50 states, the District of Columbia, and Puerto Rico.
FMCSA undertakes education or outreach while carrying out many of its
functions-for example, in posting regulations pertaining to commercial
drivers' licensing or transporting hazardous materials on its Web site, or
in providing on-line access to motor carrier crash statistics. FMCSA is
also allocated funding for specific education and outreach programs. In
fiscal year 2005, FMCSA used education and outreach funding for activities
supporting its Safety Belt program targeted to commercial vehicle drivers,
outreach to the public on hiring motor coach services, and outreach to
consumers using household goods movers. Congress also allocated fiscal
year 2005 funding for FMCSA's program directed at recently registered new
entrant motor carriers and funding to establish an initiative directed to
non-entrants-which are carriers in interstate operations that have not
registered with DOT.
Under the Government Performance and Results Act of 1993 (GPRA), federal
programs should be designed with measurable goals that support the
agency's overall strategic goals. Congress enacted GPRA to shift agencies'
focus from simply monitoring activities undertaken to measuring the
results of these activities. Under GPRA, agencies develop multiyear
strategic plans, which are the starting point for their performance
measurement. Each strategic plan is to include a mission statement, a set
of outcome-related strategic goals, and a description of how the agency
intends to achieve these goals. To measure progress toward the strategic
goals, we have previously reported that the agency should also have a plan
for collecting data to measure and evaluate program performance.13 Without
measurable goals and evaluation, it is difficult to determine whether the
program is accomplishing its intended purpose and whether the resources
dedicated to the program efforts should be increased, used in other ways,
or applied elsewhere.
Research conducted on the effectiveness of media campaigns14 indicates
that the development of an education and outreach program should include
identifying how the program is expected to change the target audience's
attitudes and behaviors. There are several theories and models on
affecting behavior, and two in particular are the most relevant to the
education and outreach programs under this review. This is because the
theories are consistent with the structure and goals of the FMCSA programs
we reviewed. One behavioral model that researchers have developed-the
reasoned action theory-assumes that people will take a desired action if
they believe the action is beneficial, and they have the means to
undertake it. Another model that experts have developed-the general
deterrence theory-indicates that in some circumstances people will take a
desired action when they recognize that they may be punished for failing
to act. The relative roles of education and enforcement depend upon the
program's target audience, the desired change to be achieved, and other
factors.
Since media campaigns should identify how the program is expected to
change target audiences' attitudes and behaviors, program managers can use
a tool or framework to show how outreach activities are intended to
influence attitudes and behaviors, and thus achieve broad program goals.
Academic literature on program development and evaluation provides a
number of models. One tool we have previously used to review education
and outreach programs15 is called a logic model. A logic model links
program inputs and outputs to program outcomes (see fig. 3). Agencies have
used such a model to plan, evaluate, and adjust program activities. A
logic model can facilitate planning and evaluation by (1) demonstrating
accountability through focusing on measurable outcomes; (2) linking
activities to results to prevent mismatches between program activities and
outcomes; and (3) integrating planning, implementation, evaluation, and
reporting. In addition, a program logic model can help describe a
program's components and desired results and explain the strategy-or
logic-by which the program is expected to achieve its goals. The logic
model can help clarify the links between program components, focusing on
outcomes that are measured and under some degree of control. In some
cases, it may be difficult to show a direct link-a specific cause and
effect-between program activities and outcomes due to the influence of
external factors. For example, the Safety Belt program, although targeted
toward commercial motor vehicles operators, may have outcomes influenced
by the NHTSA's "Click It or Ticket" program targeted to passenger car
drivers. Commercial motor vehicle operators could be affected by this
program, or other safety-belt messages, and decide to wear safety belts
when driving commercially. Thus, the Safety Belt program's initiatives may
not have been the primary factor in determining a particular driver's
safety-belt use.
Finally, once program activities are established and linked to program
goals, organizations can use program evaluation to determine whether or
not a program is meeting its goals. Program evaluations are conducted
periodically to provide an overall assessment of how well a program is
achieving its expected results. We have reported on the value of program
evaluations to assess the impact of a particular program.16 This work
showed that results of evaluations can support decisions on resource
allocation and ways to improve program effectiveness.
FMCSA Has Established Several Education and Outreach Efforts but Needs
Specific Links to Goals
FMCSA has established several education and outreach programs, with
different purposes targeted to different audiences. Total funding for
these programs in fiscal year 2005 was $36.3 million, with the largest
share-$33.1 million-allotted to the New Entrant program. FMCSA's education
and outreach efforts are intended to address recognized problems by
communicating information to motor carriers, commercial drivers, and the
public to encourage safer practices and better decision making. FMCSA
activities under these programs range from distributing brochures and
posting information on Web sites, to site visits to newly registered motor
carriers under its New Entrant program. Although FMCSA officials consider
these programs to be linked to agency goals at a high level, this linkage
to broader agency goals is not explicit in the agency's planning and
budgeting documents. FMCSA officials state that they have used a logic
model in their performance budgets to illustrate the link between
education and outreach and agency goals at a high level. Although the
performance budget implies a link between the education and outreach
program and agency goals, it does not describe how the activities are
intended to support these goals. As a result, it is difficult for
stakeholders to see how education and outreach activities that seek
changes in attitudes and behavior will ultimately contribute to agency
goals. Also, without a clear link, program managers can not easily
determine if program activities are appropriately targeted or if they need
to refine their programs to meet agency goals. In contrast, we found,
however, that FMCSA uses a logic model in the strategic plan for its
Research and Technology group to demonstrate how its programs and
activities support agency goals. As FMCSA continues its efforts to develop
the links between its education and outreach activities and broader goals,
it might consider using such a logic model to make those relationships
clear.
Education and Outreach Programs Address a Range of Safety and Consumer
Protection Concerns and Are Targeted to Industry and the Public
We reviewed five education and outreach programs that FMCSA has initiated
in response to either congressional direction or concerns about motor
carrier operations. FMCSA's education and outreach programs target a
variety of audiences, including the motor carrier industry, commercial
vehicle drivers, and the public; and they involve a variety of approaches,
such as direct contact with carriers, media campaigns, distributing
printed materials, and establishing Web sites. FMCSA also works with
industry and law enforcement associations, other agencies, and
safety-oriented organizations in carrying out some of these education and
outreach efforts. Four of these programs are about motor carrier, driver,
or passenger safety, while one has a consumer information focus. Table 1
provides a brief overview of the target audience, general purpose of the
program, and program activities.
Table 1: Overview of FMCSA Education and Outreach Programs
Program Target audience Program purpose Activities
New Entrant New motor carriers Promoting o Web site on
that have compliance with motor carrier
registered with FMCSA safety regulations
FMCSA requirements,
during new o Education and
carriers' first 18 Technical
months of operation Assistance Package
o Personal contact
with new carriers
through safety
audits by state or
federal officials,
or contractors
Non-Entrant Motor carriers that Increasing o Planned outreach
(Planned) are operating registrations by through truck
interstate but have carriers that sales and leasing
not registered with should register firms
FMCSA with FMCSA
Commercial Commercial vehicle Increasing o Brochures,
Motor Vehicle drivers safety-belt use by posters, bumper
Safety Belt Use commercial vehicle stickers
drivers distributed
through trade
shows and law
enforcement
campaigns
o Safety belt
information on Web
site
Motor Coach Organizations such Enabling the public o Web site
Outreach as schools and to make more information on
individuals that informed choices selecting a motor
hire motor coach when selecting a coach company and
services motor coach company company safety
information
Household Goods Individuals that Enabling the public o Brochures
Outreach are planning to to be better distributed
move across state informed about through moving
lines selecting an companies and
interstate moving other means
company
o Web site
information on
selecting and
contracting with a
moving company
Source: GAO analysis of FMCSA information.
The five programs in this review are summarized below, and discussed in
greater detail in appendix II.
New Entrant Program
FMCSA intends the New Entrant program to inform newly registered motor
carriers (new entrants) about motor carrier safety standards and
regulations to help them gain compliance with FMCSA requirements and
improve truck safety and thus reduce crash rates.17 Studies have shown
that new carriers have higher safety violation and crash rates than more
experienced carriers. FMCSA uses its Education and Technical Assistance
Package to communicate safety requirements and other information to new
entrants when they register with DOT. This information package is also
available at FMCSA's Web site, along with carrier registration forms.18
The primary activity of the New Entrant program is safety audits conducted
through site visits to the new carriers. Safety audits are conducted by
FMCSA, state law enforcement, or contractor personnel, who review the
carrier's compliance with FMCSA requirements, including driver
qualifications, driver records of duty status, vehicle maintenance
records, and participation in a controlled substance and alcohol use
testing program. As of April 2005, FMCSA had conducted 52,000 safety
audits since the New Entrant program began in 2003.
Non-Entrant Program
FMCSA is developing a Non-Entrant program intended to inform start-up
motor carriers who have not registered with the agency-called
non-entrants-of the requirement to register with FMCSA. FMCSA officials
are concerned that carriers that have not registered with FMCSA may
represent an increased safety risk. FMCSA expects to target motor carriers
who have not registered as interstate carriers with FMCSA, by working
through truck sales and leasing firms to inform start-up motor carriers
about registration requirements and, ultimately, to ensure that the
carriers register as new entrants. FMCSA officials expect to award a
contract to develop the Non-Entrant program before the end of 2005.
Commercial Motor Vehicle Safety Belt Program
The Commercial Motor Vehicle Safety Belt program is intended to increase
safety-belt use by commercial truck drivers. FMCSA began this effort after
a study showed that only 48 percent of commercial truck drivers used
safety belts, compared with nearly 80 percent safety-belt use by passenger
car drivers. Working through the Commercial Motor Vehicle Safety Belt
Partnership-an organization of government, industry, law enforcement, and
safety associations-FMCSA facilitates and coordinates the development and
distribution of outreach materials, including posters, bumper stickers,
and brochures promoting safety-belt use. (See fig. 1 for an example of a
brochure describing the safety-belt partnership.) Working through the
Partnership allows FMCSA to leverage its resources and tap into
organizations that have their own communication links to the trucking
industry. FMCSA staff distribute materials through trucking industry shows
and events, and state law enforcement personnel distribute materials
during safety enforcement campaigns. FMCSA has prepared over 200,000
brochures and 50,000 bumper stickers for the 2005 "Be Ready, Be Buckled"
campaign. FMCSA also provides safety belt information through its Web
site.
Figure 1: Cover of Safety Belt Partnership Brochure
Motor Coach Outreach Program
The Motor Coach Outreach program provides consumers with information to
help them select and hire motor coach services. The program began
following an increase in motor coach accident fatalities. Currently,
FMCSA's primary effort is maintaining a Web site that allows individuals
to access advice on selecting a motor coach company and safety information
on specific motor coach companies. For the "Moving Kids Safely" campaign
in 2002 that was part of the Motor Coach Outreach program, FMCSA created
49,000 brochures; 1,200 posters; and 1,200 "toolkits" to provide advice to
school districts on hiring school bus services. FMCSA also reaches out to
the industry to inform motor coach companies of regulations and provide
safety advice through printed materials and the agency's motor coach
specialists.
Household Goods Outreach Program
The Household Goods Outreach program is intended to help consumers make
more informed choices in selecting an interstate moving company to avoid
unscrupulous movers. This program was begun in response to congressional
concern over increasing consumer complaints about fraudulent moving
companies. FMCSA distributes printed material through moving companies and
other means, such as its Web site and the General Services Administration
(GSA) consumer information catalog, to inform consumers of their rights
when dealing with moving companies. The material that FMCSA has developed
includes a new outreach theme: "Protect Your Memories, Your Money, Your
Move" and it includes brochures and presentations that can be delivered by
FMCSA staff. For example, as of October 2005, FMCSA officials reported
that they had printed a total of 50,000 copies of the two brochures the
agency distributes. FMCSA also maintains a hotline to receive complaints
from individuals about problems with movers and enters them into a
database. Although FMCSA does not get involved in resolving complaints,
the agency uses the database to target firms for investigation. Also, we
previously reported that FMCSA should make the complaint information
available to consumers.19 According to FMCSA officials the agency has not
yet done this because of privacy and other legal issues.
Except for the New Entrant Program, Education and Outreach Funding
Represents a Small Portion of FMCSA's Budget
The five education and outreach programs represent a small portion of
FMCSA's total funding. Fiscal year 2005 funding for the five programs
totaled $36.3 million, out of approximately $443.3 million to fund FMCSA
operations and grant programs. The New Entrant program accounted for $33.1
million of the funding for the five programs, while funding for the other
four programs combined was $3.2 million-less than 1 percent of the
agency's total fiscal year 2005 funding.
The first chart in figure 2 below depicts the New Entrant program, and the
combined funding for the other four programs, as their relative share of
total FMCSA funding in fiscal year 2005. The second chart depicts the
relative size of the four smaller education and outreach programs.
Figure 2: FMCSA Fiscal Year 2005 Education and Outreach Funding
Note: Data resulted from U.S. House of Representatives Report 108-792,
Making Appropriations for Foreign Operations, Export Financing, and
Related Programs for the Fiscal Year Ending September 30, 2005, and for
Other Purposes (Nov. 20, 2004): and discussions with FMCSA officials.
aCongressional language accompanying the fiscal year 2005 DOT
appropriations requested FMCSA to apply $50,000 to the "Safety is Good
Business" program. FMCSA officials informed us that this amount would be
added to the $1.0 million appropriated for the Non-Entrant program.
The funding available for education and outreach supports a diverse range
of activities, from the costs of state personnel conducting safety audits,
to the costs of printing and distributing brochures and other materials,
and the costs of contracts being awarded to develop and evaluate education
and outreach activities. The principal activities that FMCSA is
undertaking through the five programs are shown in table 2.
Table 2: How Fiscal Year 2005 Outreach and Education Funding Is Being Used
Program FY 2005 Funding Principal use of fundinga
New Entrant $33,100,000 $30,100,000 in grants to states to
conduct safety audits.
$3,000,000 for FMCSA to contract for
safety audits.
Commercial Motor $500,000 $440,000 interagency transfer to
Carrier Safety- Belt NHTSA for two studies addressing
Use safety belt use by commercial
drivers.
$24,400 for brochures, posters,
bumper stickers and other materials
prepared for "Be Ready, Be Buckled"
campaign.b
Motor Coach Outreach $150,000 Printing and distributing brochures,
Web site maintenance, and surveys of
Web-site users.
Household Goods $1,488,000 $900,000 for education, outreach and
evaluation support contracts.
Outreach
$570,000 for complaint database
development and maintenance.
Non-Entrant $1,050,000 $1,000,000 appropriated for a
Non-Entrant program plus $50,000
targeted for "Safety is Good
Business," will be used to contract
for development of a Non-Entrant
program.
Source: FMCSA information.
aBecause these examples illustrate the principal use of components of
these programs, in some cases they do not add to the total amounts spent
in the programs.
bThe Commercial Vehicle Safety Alliance (CVSA) provides additional
financial support for printing and distributing brochures under this
initiative.
FMCSA Has Not Clearly Articulated How Its Education and Outreach
Initiatives Will Support Broader Goals
Congress has expressed concern about how FMCSA's education and outreach
activities are meeting broader goals, such as individual program goals, or
DOT's strategic goals. The conference report accompanying the DOT
appropriations bill for fiscal year 2005, for example, asked that FMCSA
report by April 2005 to the House and Senate Committees on Appropriations
on strategies linking outreach and education program initiatives to each
goal.20 FMCSA sent its report on October 11, 2005. The report indicates
that FMCSA believes education and outreach programs support the overall
DOT safety strategic objective of "enhancing public health and safety by
elimination of transportation deaths and injuries." FMCSA also indicates
that every education activity undertaken "is vital to FMCSA's goal of
reducing fatalities, injuries, and crashes, and attaining a large truck
crash rate of no more than 1.65 fatalities per 100 million miles of truck
travel by the end of 2008."
Although the report was intended to address congressional concerns about
linking education and outreach to program goals, it is unclear
specifically how this will be accomplished, since the report does not
describe how these programs are intended to influence knowledge,
attitudes, and behaviors and thus support FMCSA's broader goals. Although
the report describes in detail the activities FMCSA is undertaking for
each of these programs, it does not describe how these activities are
intended to change attitudes and behaviors that will ultimately meet
FMCSA's goals.
In addition, although FMCSA's fiscal year 2006 performance budget has some
discussion of how activities support FMCSA and DOT safety and productivity
goals, except for the Commercial Motor Vehicle Safety Belt program, none
of the budget or program documentation we reviewed for FMCSA's education
and outreach programs provides detailed descriptions about how activities
link to goals. A greater level of detail in the description of how
education and outreach program activities link to goals would help the
public and other interested parties to see how program activities that
seek changes in attitudes and behavior will ultimately contribute to
agency goals. Furthermore, without this link, program managers cannot
easily determine if program activities are appropriately targeted or if
they need to refine their programs to meet agency goals.
FMCSA officials stated that information about education and outreach
programs is reported in DOT's strategic plan and in FMCSA's performance
budget documents.21 However, DOT's strategic plan does not specifically
address FMCSA's education and outreach programs and refers readers to the
performance budgets for more detailed, program-specific information.
FMCSA's performance budget documents identify strategic objectives, such
as saving lives and reducing injuries by preventing bus and truck crashes,
that link and contribute to objectives in DOT's strategic plan, such as
reducing transportation-related fatalities. It also identifies underlying
performance goals and indicators for each objective. For example, FMCSA's
strategic objective for safety is illustrated in table 3.
Table 3: Hierarchy of Objective, Goal, and Indicators for FMCSA's
Commercial Motor Vehicle Safety Objective
Strategic objective Save lives and reduce injuries by preventing truck and
bus crashes.
Performance goal Reduce the rate of large truck-related fatalities to
no more than 1.65 fatalities per 100 million truck
vehicle miles traveled by the end of 2008.
Leading indicators Fatalities in large truck crashes.
Intercity bus fatalities.
Injuries in large truck crashes.
Rate of large truck-related injuries.
Source: Performance Budget Estimates section of FMCSA's Fiscal Year 2006
Budget Submission to Congress.
The performance budget documents address education and outreach primarily
under this strategic objective for safety, by stating that "educating
carriers about the benefits of operating safely, and in compliance with
safety regulations, is advantageous to both the carrier and enforcement
community." The budget also states that the education programs are
designed to change "the knowledge, attitudes, and behaviors of commercial
motor carriers, commercial motor vehicle drivers, and passenger vehicle
drivers, driving in the vicinity of large trucks."
However, the performance budget does not provide specific information for
each program on how these expected attitude and behavior changes are
linked to broader goals. Of the five programs included in our review, the
description of the safety belt initiative provides the clearest
information articulating how the program is intended to affect the
knowledge, attitudes, and behaviors of its targeted audience, and
consequently to improve safety. FMCSA describes how education will
highlight the risks of not wearing a safety belt, which in turn is
intended to improve drivers' attitudes toward wearing safety belts, and
subsequently meet the program goal of increasing safety-belt usage 10
percent by 2009 and further, meet the strategic objective of reducing
fatalities.22 The performance budget also indicates that education will be
combined with traffic enforcement of safety-belt requirements to achieve
this outcome. In contrast, the performance budget discusses the New
Entrant program in terms of staffing, funding, and the number of safety
audits to be done in fiscal year 2006, without defining a goal for what
the New Entrant program will accomplish and how that accomplishment would
contribute to the strategic objective of reducing fatalities.
In addition, the links between the Motor Coach and Household Goods program
activities and broader strategic goals are difficult to discern.
Specifically, the performance budget document explains that the Motor
Coach program seeks to increase consumers' knowledge about the safety
records of carriers, with a potential side benefit of encouraging motor
carriers to maintain good safety records. FMCSA does not, however,
describe how affecting consumers' choice of a particular motor coach
company would support the overall strategic objective of reducing
transportation-related fatalities. FMCSA's performance budget relates its
Household Goods program to DOT's mobility and FMCSA's productivity
objectives. The mobility objective is "to advance accessible, efficient,
and
intermodal transportation for the movement of people and goods."23 The
productivity strategic objective is "to promote efficient and economical
motor carrier operations to sustain mobility and economic growth." The
performance budget document asserts links between these objectives and the
education and outreach program, but it does not explain how specific
activities-such as establishing its Web site on selecting moving companies
and creating brochures on consumer rights and responsibilities-support
either of these objectives.24
On the basis of our discussions with FMCSA officials, it appears the
agency is attempting to more clearly describe how education and outreach
activities link to the agency's goals. FMCSA acknowledged that there may
have been some disconnection between the strategic planning and budgeting
processes in the past, but managers have recently received training on
defining a program's desired outcome and then linking inputs-such as
funding-and outputs. Officials told us that as a result the agency's
proposed fiscal year 2007 performance budget will better link programs and
goals to budget requests. FMCSA's eventual goal is to link program outputs
to outcomes, as well as identifying measurable and verifiable goals to the
extent possible.
A Logic Model Provides a Framework for Linking Program Activities to
Desired Results and Goals
Although the relationships among program activities, changes in knowledge
and behaviors, and broader goals can sometimes seem intuitive, laying a
program out through a logic model can be useful for program managers and
stakeholders to demonstrate and understand these relationships. FMCSA
officials state that they use a logic model in its performance budget to
link education and outreach to agency objectives. However, the performance
budget implies a link between the education and outreach program budget
and agency goals without describing how FMCSA intends the program to
change behaviors, such as following federal safety regulations, to support
agency goals, as would be shown in a logic model. FMCSA has used a logic
model in another program area that provides a description of how a program
is intended to support agency goals. Specifically, FMCSA's Research and
Technology group used a logic model along with other planning tools to
demonstrate how its activities support FMCSA's goals in its 5-Year
Strategic Plan. The Research and Technology logic model includes inputs
such as staff, outputs such as research studies, outcomes such as improved
understanding of root causes of crashes, and impacts such as fewer
commercial motor vehicle crashes.25 FMCSA described the Research and
Technology Logic Model as a "mechanism to help define metrics for
performance, quality, and relevance of the program."26
As FMCSA endeavors to describe how its education and outreach program
activities affect the knowledge, attitudes, and behaviors of its target
audiences, and link these results to broader goals, it may wish to
consider a tool such as a logic model-as the agency used in its Research
and Technology strategic plan-to clearly outline those links. In previous
work on educational and outreach,27 we used a logic model to describe the
program components-inputs and outputs-and how these support the desired
results, thus explaining the strategy-or logic-by which programs are
expected to achieve their goals. Figure 3 graphically depicts this model
and its components.
Figure 3: Program Logic Model
The logic model illustrates how program activities such as distributing
informational pamphlets contribute to the results of a program, such as
increasing safety-belt usage, and to an ultimate goal, such as reducing
fatalities. In addition, program managers should consider the effect of
external factors when defining outcomes and recognize that the correlation
between activities and desired outcomes may not be as direct as one would
like. Without a logic model or similar tool to clearly describe the links
between education and outreach programs and broader goals, FMCSA may have
difficulty designing and evaluating effective programs.
Evaluations of Specific Education and Outreach Programs Impacts Are Still
in Preliminary Stages, but the Lack of a Plan to Evaluate How New Entrant
Safety Audits Improve Knowledge and Attitudes Raises Concern
FMCSA has begun some evaluations of its education and outreach activities,
including the New Entrant program, and plans to use contractors to
evaluate some programs. Surveys and studies are the primary means by which
FMCSA or its contractor will evaluate the programs. However, some gaps in
evaluation plans remain, especially for the largest of the programs we
reviewed. Specifically, FMCSA has not evaluated whether the Education and
Technical Assistance Package provided to new entrants and the safety
audits conducted under the New Entrant program effectively communicated
information to new entrants, making it difficult to determine the impact
of that program. A list of FMCSA's evaluations and planned evaluations are
provided in figure 4.
Figure 4: FMCSA's Evaluations and Planned Evaluations by Program
For each program, FMCSA's evaluation efforts are described as follows.
New Entrant Program
FMCSA evaluates the performance of the New Entrant program primarily by
tracking the number of safety audits conducted of new motor carriers. The
agency has a goal of conducting a certain number of safety audits per
year, safety audits of 80 percent of new entrants within the first 6
months they are in business, and 100 percent of all new entrants within
their first 18 months of operation. FMCSA sets employee performance goals
for division office managers for conducting a particular number of safety
audits of new entrants. According to FMCSA officials, agency personnel
conducted over 9,400 safety audits in fiscal year 2005, exceeding their
goal of 8,000 FMCSA completed safety audits. In addition, by April 2005,
52,000 safety audits had been conducted since the program began, and the
number conducted each quarter has been generally increasing. However,
FMCSA officials acknowledge that there is a safety audit backlog in some
states, and the goals for auditing new entrants in the first 6 months, and
the first 18 months, are not being met. The number of safety audits
reflects the number of new registrants that were personally informed about
the carrier requirements. Using registration data on the number of new
entrants, the agency can estimate the number of new entrants needing a
safety audit and agency progress toward the program goal. There are about
40,000 new entrants registered per year, but officials told us that about
40 percent drop out or discontinue business within the first year of
operations.
By tracking the number of safety audits conducted, FMCSA seeks to
determine the extent to which it has provided information to its targeted
audience of new drivers. FMCSA has no process to measure whether new
entrants' awareness of safe trucking practices was raised, which ideally
would be a consequence of an effective education and outreach effort.
Specifically, although FMCSA plans to evaluate the New Entrant program in
2008, it has no current plan to determine how FMCSA's safety requirement
information, such as its Education and Technical Assistance Package and
its safety audits, affect new entrants' (1) awareness of FMCSA
requirements, such as drug and alcohol testing requirements or (2) their
motivation to change behaviors that might lead to safer operations. The
New Entrant program manager indicated that since the program is new and
still evolving, the outreach campaign does not have much historical data
to evaluate.
Although the program has not been in existence very long, FMCSA officials
stated that the agency has conducted one study of the New Entrant program
and has plans to conduct another. One preliminary analysis by FMCSA
compared the crash rates of new motor carriers registering with FMCSA
during 2002 to the crash rates of over 1,000 new motor carriers that
registered with FMCSA and had a safety audit during the first 6 months of
2003. This analysis showed that there was little difference between the
crash rates of these two groups. FMCSA officials said that they are
planning a study to be conducted after planned changes to the safety
audit, discussed later in this report, have been in place for a period of
time. Officials told us that the study planned to begin in 2008 will
examine carriers' safety records over time, although they have not yet
developed an evaluation plan.
We have previously reported on the difficulties of assessing education and
outreach programs such as the New Entrant program. We have also reported,
however, that federal agencies are expected to demonstrate how their
activities contribute to achieving agency or governmentwide goals.
Additionally, the guidance that FMCSA uses when developing evaluation
efforts for education and outreach programs-a booklet entitled The Art of
Appropriate Evaluation28 -published by NHTSA-emphasizes that evaluations
should include measurements of changes in awareness, knowledge, and
behavior. If FMCSA's planned evaluation of the New Entrant program does
not consider whether the program is actually succeeding in effectively
conveying the educational material on safe trucking practices, FMCSA will
be unable to clearly determine the program's impact. Or, if safety audits
are having little impact, as suggested by FMCSA's preliminary analysis,
the agency will not be able to determine whether it is due to (1) the
educational material provided, (2) how the information is conveyed during
the safety audits, (3) the lack of follow-up on the safety audit results,
or (4) other factors. Additionally, it will be difficult to demonstrate to
Congress that this program is having its desired effect.
Non-Entrant Program
This program has not yet begun. FMCSA officials plan to award a contract
by the end of 2005 to develop, implement, and evaluate the Non-Entrant
program. The contractor is to test the effectiveness of the outreach
materials while they are being developed to determine if they will reach
the intended audience and effectively changed knowledge and attitudes.
Development of the materials will include surveying the targeted audience
and conducting outreach through truck leasing and sales firms. FMCSA's
program plan indicated that the contractor will be expected to assess
changes in knowledge, attitudes, and behaviors of its target audience.
Commercial Motor Vehicle Safety Belt Program
FMCSA officials stated that they plan to evaluate this initiative in
several ways once they have finalized its evaluation arrangements.
According to FMCSA officials, during program design, FMCSA tested its
marketing materials with target audiences, including truck drivers, and
found that the marketing material was appropriately targeted. Also, FMCSA
is installing software to track the number of Web-site hits on certain of
its Web pages that contain important educational elements of the Safety
Belt campaign. This system will enable FMCSA to determine how many times
its information has been viewed, but it will not enable FMCSA officials to
know whether that information was effectively conveyed to its target
audience or changed their attitudes about wearing safety belts.
Nonetheless, FMCSA officials believe that this software will enable
program managers to monitor and determine which Web-site pages are viewed,
and therefore are more effective, based on the number of hits and the
length of time at each site. Furthermore, FMCSA is considering focus
groups and surveys with carriers on how safety belt marketing material
affected their awareness of the importance of using safety belts.
In addition, FMCSA has an agreement with NHTSA for two efforts that could
measure progress toward the expected outcome of this initiative-a
10-percent increase in use of safety belts from 2006 through 2009. The
first effort is a study, expected to be completed in December 2005, which
updates a 2003 commercial motor vehicle study that reported that 48
percent of commercial drivers wear safety belts. The new study will update
information on the percentage of commercial drivers that wear safety belts
by replicating the methodology used in the 2003 study, according to FMCSA
officials. The second effort will investigate the feasibility of annually
collecting data on safety-belt use by commercial drivers through the
National Occupant Protection Usage Study (NOPUS) sponsored by NHTSA.
Although these efforts will measure safety-belt use, they will not
evaluate whether the education and outreach program in particular
contributed to changes in the number of individuals wearing safety belts.
Motor Coach Outreach Program
FMCSA has a Web-based, pop-up survey for its Web site which, although
focusing on consumer satisfaction in general, gives FMCSA some ideas about
Web-site usage and exposure to the information contained on the site. The
survey randomly selects visitors to the Motor Coach Web site and asks them
to answer a series of questions concerning the accuracy, quality,
convenience, and ability of finding needed information. The survey also
asks visitors about their overall satisfaction with the Web site. Although
these questions may be generally helpful to rate the Web site experience,
more specific questions on the Motor Coach program information have not
been incorporated. The pop-up survey has flexibility for use in other
FMCSA program Web pages, and FMCSA has plans to use Web-based usage
tracking as an additional tool in the future. However, with only a small
budget-$150,000 in fiscal year 2005-for the Motor Coach program, it is
reasonable to expect that FMCSA would balance the need for this type of
evaluation information with its other education and outreach program
priorities.
As of October 2005, FMCSA officials also had plans to award a contract for
contractor support to conduct surveys, focus groups, and interviews to
evaluate its education and outreach programs, including obtaining Office
of Management and Budget (OMB) clearance for these activities.29 FMCSA
officials told us that the motor coach program would be evaluated under
this contract.
Household Goods Outreach Program
FMCSA has identified at least two approaches to evaluate the effectiveness
of the outreach efforts for the household goods outreach program. One
effort is to survey recipients who have received the Ready to Move - Tips
for a Successful Interstate Move booklet from GSA. As of May 2005, 703
surveys had been distributed to determine whether the recipients believed
that the booklet would change their intentions about which household goods
carrier they might select. The survey asks questions regarding the
effectiveness of the booklet and its information content. Also, FMCSA
officials stated that the Household Goods program would be evaluated under
the survey support contract discussed in the preceding section on the
Motor Coach program.
An FMCSA official whom we spoke with indicated that the elderly may be
particularly targeted by unscrupulous movers. Recent data released by the
Census Bureau indicate that only about 40 percent of people age 55 years
or older have computers with Internet access in their homes.30 Although
FMCSA has an available hard-copy brochure, the Web-based outreach efforts
may not be fully reaching this population. FMCSA's planned evaluations of
the Household Goods program do not address the effectiveness of FMCSA's
Web-based outreach in reaching this population.
The Effectiveness of Education and Outreach Programs Is Unclear, but
Programs Seem to be Reasonably Designed According to Theories of Behavior
Change and Industry Groups
Most of FMCSA's evaluations of its education and outreach programs have
yet to be completed, and therefore little is currently known about the
effectiveness of the programs. FMCSA expects the contracts for the
evaluation programs to be finalized within months and will receive
information from the evaluations once the data are collected and analyzed.
Until that time, what is known about program effectiveness is largely
based on anecdotal information from stakeholders such as industry
associations, as well as a comparison of the education and outreach
programs to the design of other successful programs that have been
implemented. However, FMCSA's education and outreach programs and
marketing materials appear to follow theories and research on behavior
change. Finally, a public-safety group suggests FMCSA followed reasonable
approaches in starting its education and outreach efforts, however, it
would like to see more quantitative evaluation in the future to help FMCSA
refine its programs.
Behavior Theory Indicates Education and Outreach Can Be Effective in
Increasing Awareness and Changing Behavior for Programs that Do Not Have
Regulations Governing the Behavior of Their Target Audiences
Studies by us, other agencies, and academics show that education and
outreach programs, such as those undertaken by FMCSA, can be effective in
creating awareness of a problem or issue and influencing some change in
behavior. One theory-the theory of reasoned action-explains the
relationship between attitudes, beliefs, intentions, and behavior. In
short, people are more likely to change their behavior if they believe the
change is valued-or that not changing their behavior has negative
consequences-and if they have the necessary opportunity to change the
behavior. For example, one antismoking advertising campaign emphasized the
negative effects of smoking on a smoker's health. Smokers became aware of
the negative consequences, and 26 percent of respondents indicated that
they were more likely to quit smoking based on the awareness of the
negative consequences of smoking. Another study on the effects of sun
exposure showed that countering the mistaken belief that sun tanning had a
positive effect reduced the percentage of individuals who habitually lie
in the sun. These studies demonstrate that emphasizing the consequences of
a negative behavior to individuals can result in changes to attitudes and
behavior, without applying any specific sanctions.
Based on the theory of reasoned action, education and outreach can have a
positive impact through FMCSA's two programs that have a goal of
influencing consumers to make appropriate choices of motor carriers.
FMCSA's Motor Coach and Household Goods Outreach programs are targeted
toward consumers and do not have regulations governing the consumers'
behavior, thus, according to the theory of reasoned action, gaining
knowledge about the positive consequences of choosing a safe carrier will
affect the individuals' attitude toward this behavior. The change in
attitude will directly affect intention and the subsequent behavior. Thus,
education and outreach campaigns can provide an individual under these
circumstances with the information that is needed to make a more educated
choice. For example, the Motor Coach program follows this approach by
targeting consumers and conveying the negative consequences of choosing an
unsafe carrier. In the case of the Household Goods program, the campaign
conveys the positive consequences of choosing a reputable carrier in the
Households Goods program, although the elderly, a group potentially
susceptible to unscrupulous carriers, may require a different delivery
method other than FMCSA's Web site to receive this message.
A Link to Enforcement Can Strengthen Education and Outreach Programs'
Ability to Change Behavior
Independent studies as well as our prior work31 indicate that the level of
enforcement in safety related programs is positively correlated with
safety-belt use, especially when coupled with public-awareness efforts.
According to the general deterrence theory, individuals must be exposed to
law enforcement or receive information about legal ramifications of their
actions before they can be deterred. The perception or expectation of
certain sanctions results in attempts to avoid committing the offense.
Studies show this theory is an effective model for strengthening the
change in behavior resulting from safety programs. Thus, this link to
enforcement would apply to FMCSA's three education and outreach
programs-Safety Belt, Non-Entrant and New Entrant-that have existing
regulations that address their target audience's behavior.
One study, conducted by the National Safety Council, on the state of South
Carolina's safety-belt program, showed that while media campaigns
immediately increased awareness of the importance of safety belts, the
largest increase in safety belt-use occurred after officers enforced the
requirement to wear safety belts. Additionally, a 2002 study by NHTSA
included data from the state of Washington, which showed that the baseline
percentage of individuals wearing safety belts (81 percent) increased
slightly with advertising over a 4-week period to 82 percent. Ultimately
with the addition of enforcement efforts, the percentage of those who wear
safety belts increased to 90 percent within 2 months. The same study
showed similar results in Texas, where the effects of advertising alone
raised the safety belt percentage slightly, but the combination of
advertising with enforcement caused the number to increase another 6
percent, from the 80-percent baseline. These studies suggest that although
publicity may increase the awareness of the need to wear safety belts,
some people are unlikely to act on that awareness unless the enforcement
component of the campaign was closely integrated with the publicity
campaign. The results of these studies suggest that education and outreach
alone can increase awareness and encourage the intended behaviors for
FMCSA's education and outreach programs, but that enforcement can
strengthen the result of these efforts.
FMCSA Is Increasing Enforcement in the New Entrant Program and
Acknowledges the Importance of Enforcement to the Non-Entrant and Safety
Belt Programs
Three of FMCSA's programs we examined, the New Entrant, Non-Entrant and
Commercial Motor Vehicle Safety Belt programs have regulations, such as a
requirement for motor carrier drivers to wear safety belts, affecting the
target audiences' (motor carrier industry) behavior. This provides the
opportunity to use both enforcement and education in programs with the
goal of improving the safety behavior of the motor carrier industry. The
following sections describe how FMCSA has indicated it is using or will be
using enforcement along with education and outreach in these programs.
New Entrant Program
Enforcement has been only a minor component of the New Entrant program,
but FMCSA is currently strengthening the criteria for passing New Entrant
safety audits because of concerns of its effectiveness, based on analysis
and anecdotal information indicating that new entrants that have been
notified of the safety audit requirements may not be operating more
safely. In addition, even if a new entrant passes the safety audit, some
carriers are not yet following important safety requirements, such as
having a drug and alcohol testing program, at the time the safety audit
was conducted. This change should establish a better link to enforcement,
in keeping with behavioral research, in that it will be more difficult for
new entrants to retain their operating status unless they are following
the safety requirements at the time of the audit.
The safety audit has been considered an educational unit, with a strong
emphasis on training new entrants about the requirements to operate in
interstate commerce. For example, the federal register notice that
established the new entrant safety assurance process indicated that the
new entrant process is intended to "improve the safety performance of new
entrants by providing educational and technical assistance" and required
"minimum requirements to new entrant motor carriers to ensure that they
are knowledgeable about applicable federal motor carrier safety
standards." There are few sanctions for carriers who fail critical areas
of the safety audit, although a new entrant might be subject to a more
stringent and detailed review-a compliance review-under certain
circumstances.32 However, the compliance review process occurs less often
with new motor carriers than experienced carriers.
Although FMCSA's requirements indicate that the objectives of the safety
audit are to educate the carrier and determine areas where the carrier's
compliance might be deficient, in practical application almost all new
entrants have passed the safety audit, and little is done to follow-up to
ensure that new motor carriers correct deficiencies identified in the
safety audits. A new entrant can fail two of six sections and still pass
the audit. Safety audit sections include areas such as driver
qualifications, business operations, and maintenance records. Over a
2-year period ending in April 2005, there were 51,681 carriers that passed
the safety audit, and only 372 failed, producing a 99.3 percent passing
rate for new entrant motor carriers. Those who fail the audit must take
corrective action within 45 days if they are a passenger or hazardous
material carrier, or 60 days for all other carriers, or they will lose
operating status. Furthermore, for those carriers who passed the audit
despite failing multiple sections of it, FMCSA currently has no mechanism
to ensure the failed sections of the audit have been corrected before the
carrier achieves permanent status. Although FMCSA sends letters to new
entrants about the sections in which they are not in compliance with motor
carrier regulations; beyond the letter and agency follow-up to obtain
proof of insurance, the agency or its representatives do not determine if
the new entrants have in fact corrected deficiencies identified during the
safety audits.
Our analysis of critical factors within the safety audit questions
revealed that over the same 2-year period, about 40 percent of the
carriers failed the "driver" section of the audit, despite passing the
audit overall. This section includes critical questions, such as whether a
carrier's driver qualifications have been reviewed and whether the
carrier's drivers are covered by a drug and alcohol testing program.
Specifically, we found that about 60 percent of the new entrant carriers
that passed the safety audit had failed the requirement to check
qualifications of drivers prior to hiring them. Checking qualifications
means that before hiring a driver a motor carrier company obtains and
reviews information on an applicant's driving history, including accident
and ticket information. In addition, we found that for the most part,
between 27 and 29 percent of carriers who passed the safety audit had not
maintained maintenance files for requested vehicles. The lack of
improvement over the 2-year period illustrates the importance of
determining if the Education and Technical Assistance Package, which the
carriers received prior to the audit, should be changed to improve these
results.
FMCSA officials agreed that the safety audit may not be improving safety
behavior of new entrants, and the agency is already tightening these
requirements. Currently, there is no one question or section of the audit
that can cause a new entrant to fail. FMCSA officials informed us that the
new federal register notice laying out stricter pass requirements-expected
to be issued shortly-identify eight critical questions that individually
could lead to failing the safety audit, thus providing more incentive for
carriers to comply with new entrant requirements. For example, carriers
would automatically fail if they do not employ a drug and alcohol program,
have insurance, or have the proper maintenance requirements. Carriers who
fail the safety audit and do not correct the deficiencies could lose their
operating status and would be required to correct deficiencies before
their operating status is reinstated.
Safety Belt Program
Similarly to the New Entrant program, there has been limited enforcement
associated with the Safety Belt program to date. The finding that 48
percent of commercial drivers wear safety belts, compared with 80 percent
of passenger vehicle drivers, suggests that commercial motor vehicle
operators have thus far been less receptive to messages in the market
place regarding the importance of wearing safety belts while on duty.33 In
light of the studies we reviewed, an enforcement element might help
strengthen education and outreach programs' attempts to increase awareness
of the risk to drivers of not wearing safety belts and encourage
commercial drivers to wear them. It is reasonable to assume that
commercial motor vehicle operators, similar to private operators, would be
educated and informed from the materials presented in the campaign but
that some would not act on the information unless there was some risk of
sanction. The approved plan for FMCSA's safety belt initiatives includes a
third phase of implementation that calls for increased enforcement.
Although the plan does not specify how FMCSA will enforce federal
regulations regarding wearing safety belts and how FMCSA will work with
state agencies to carry out enforcement-as this program develops,
strengthening the enforcement link may increase the effectiveness of the
safety belt education and outreach efforts. FMCSA officials indicated that
this enforcement effort had begun and included fiscal year 2005 grants to
state agencies that can be used for safety belt education and enforcement.
Non-Entrant Program
FMCSA's program plan for the Non-Entrant program indicates it will have
some enforcement component as it is developed; however, specific
information on enforcement has not yet been developed. Literature on
safety programs and experience with similar programs indicates the
Non-Entrant program might benefit from enforcement after non-entrants are
exposed to the message about the importance of registering for a DOT
license when they buy or rent a vehicle for operation.
Industry Associations Offer A Mostly Positive View of FMCSA's Education
and Outreach Efforts, but Public Advocacy Groups Expressed Some Concerns
Overall, the industry associations' view is that FMCSA is doing several
positive things in its education and outreach activities. Officials of the
seven industry associations whom we spoke with had regular contact with
FMCSA officials and were familiar with the education and outreach programs
under this review. Many association officials cited direct contact with
FMCSA on industry issues on a regular basis and said that their
partnerships with FMCSA on education and outreach activities were
productive. Specifically, an official of one moving industry association
told us that it shared data with FMCSA regarding complaints against
carriers and often refers complainants to FMCSA for further information.
Officials representing a consortium of primarily state agencies involved
in motor carrier safety, to which FMCSA belongs, told us that the support
FMCSA was able to provide was important to the implementation of the
Safety Belt program. Lastly, some association officials said that FMCSA
was doing a good job with the resources it has, but others said FMCSA
could do more with its education and outreach efforts if the agency had
more resources.
However, one of the two public safety groups we contacted believed that
although the education and outreach programs were based on good ideas,
FMCSA needs to do more quantitative evaluations of these programs. Both
public safety groups also felt that FMCSA should have stronger
enforcement. For example, one group indicated that research has shown
enforcement to increase the effectiveness of education programs and thus,
FMCSA should focus more on compliance and enforcement. Finally, although
industry associations emphasized the positive professional relationships
they have with FMCSA officials, a public safety group indicated that it is
wary about FMCSA's tendency to manage programs in alliance with industry
partners, since the group believes it might affect the agency's
objectivity.
Conclusions
Uncertainty about how FMCSA's education and outreach activities link to
broader program and agency goals and about FMCSA's New Entrant program
have emerged from our study. FMCSA officials have stated that the agency's
education and outreach programs support department objectives and that
they have used a logic model to show this in the agency's performance
budgets. However, in our review of agency documents, we found that except
for the Safety Belt program, FMCSA has not established clear links between
the education and outreach programs and agency and departmental goals. For
example, the conference report accompanying the DOT appropriations bill
for fiscal year 2005 requested FMCSA report to the House and Senate
Committees on Appropriations "a goal, message, and coherent and explicit
program strategy that clearly and directly link FMCSA's outreach and
education program initiatives to each program's goals" because of its
concerns over how FMCSA's education and outreach activities support agency
goals. In the October 2005 response, FMCSA stated that its education and
outreach programs support DOT strategic objectives. The report, however,
did not clearly set out the links between changing attitudes and behaviors
that would describe how these programs contribute to objectives. Such a
step would be welcome, for without an explanation of that link, FMCSA
cannot transparently demonstrate to stakeholders, including congressional
oversight authorities that the programs do, in fact, contribute to these
broader goals. Furthermore, without a clear statement of how these
programs are intended to support these goals, it is difficult to design
evaluations to determine if these programs are performing as intended or
if changes are needed in program design. FMCSA has used a type of a logic
model for its Research and Technology program, making it clear to all
interested parties how its research activities help the agency achieve its
goals. While FMCSA officials say they have used a logic model for its
education and outreach program, this is not evident in agency budget and
program documents. The use of a logic model, similar to the one used by
the agency's Research and Technology group, in planning, budgeting or
program for education and outreach programs may be beneficial.
Second, the absence of a plan to evaluate how well the information
provided to motor carriers as part of the New Entrant program was
understood and if the information raised awareness and helped promote
safer behavior also emerges as a concern. An FMCSA official stated that
the New Entrant program is still new and has limited data to evaluate.
Further, changes to the program are planned to raise the threshold for
passing a safety audit. Thus, FMCSA is planning on starting an evaluation
of the New Entrant program in 2008. However, without an evaluation of the
effect of education on new entrants' knowledge about safety requirement,
FMCSA is losing an opportunity to obtain information about the
effectiveness of the educational information conveyed through the program.
In addition, FMCSA cannot know whether its safety audits, which composed
almost 7.5 percent of FMCSA's 2005 funding, are having the desired effect;
whether the Education and Technical Assistance Package and the information
conveyed during the safety audits might require some changes to improve
their effectiveness; and whether the safety audits themselves might be
achieving their purpose of improving motor carrier safety. FMCSA has two
opportunities upcoming to obtain information on the effectiveness of the
educational materials and safety audits. First, it could use its planned
contract for evaluation of education and outreach activities to develop
and conduct such an evaluation. Second, FMCSA could include this
evaluation in its plans to evaluate the New Entrant program beginning in
2008.
Recommendations for Executive Action
To better demonstrate how FMCSA education and outreach programs contribute
to achieving agency goals, we recommend that the Secretary of
Transportation direct the Administrator of FMCSA to take the following two
actions:
o Ensure that the agency describes and documents how education and
outreach program activities link to and support broader program and agency
goals in a planning, program, or budget document that is available to the
public, and
o Evaluate the effectiveness of the education and outreach of the New
Entrant program, assessing the extent to which the Education and Technical
Assistance Package and safety audits are helping new carriers learn and
understand FMCSA requirements.
Agency Comments
We provided a draft of this report to the Department of Transportation for
its review and comment. FMCSA officials, through the DOT liaison, provided
oral comments on a draft of this report. The officials did not agree with
GAO's finding that FMCSA has not linked its education efforts with its
overall goals. FMCSA officials state they have linked their education and
outreach programs to agency goals at a high level in the agency's
performance budgets. In addition, the officials state that they used a
logic model to make this link. We included FMCSA's view that it has linked
its education and outreach programs to agency goals in our report.
However, we did not find this link evident for most of the education and
outreach programs. For example, Congress requested FMCSA to report on the
link between its education and outreach program initiatives to each
program's goal. FMCSA's report to Congress described the program
activities; however, it did not describe the link between the activities
and FMCSA's goals. In addition, although FMCSA officials stated that they
have used a logic model in their performance budget to link education and
outreach programs to agency goals, the discussion implied a link, but does
not describe how FMCSA's education and outreach activities are intended to
change target audience's attitudes and behaviors that would contribute to
meeting agency goals. However, in FMCSA's fiscal year 2006 performance
budget, we did find and report on a relatively clear discussion of how the
Commercial Motor Vehicle Safety Belt program supported safety objectives
such as reduced fatalities.
In addition, DOT and FMCSA officials were concerned that our
recommendation on documenting the link between education programs and
agency goals was too prescriptive. We recommended that FMCSA expedite the
process of describing and documenting how its education and outreach
programs link to and support overall agency goals in strategic planning
and budget documents. DOT officials said that including this specific
information on relatively small dollar projects-such as the Motor Coach
program-was not appropriate in the DOT strategic plan. In addition, DOT
officials told us that it is not appropriate to include this level of
performance information in budget requests. Since the strategic plan and
the performance budget are not the only opportunities to detail program
linkage to broad agency goals, we clarified our recommendation to provide
FMCSA with more flexibility in identifying an appropriate planning,
program, or budgeting document in which to describe and document the link
between program activities and goals, so that external stakeholders,
including congressional oversight authorities, can understand how these
programs are supporting agency goals.
Second, agency officials suggested that it is too early to evaluate the
New Entrant program, as we recommended; and they have not developed
evaluation plans because they do not plan on beginning an evaluation until
2008. However, after over 2 years of implementation and 52,000 safety
audits, FMCSA is losing an opportunity to conduct an evaluation of the
educational component of the New Entrant program, including the
Educational and Technical Assistance Package for new entrants and the
information conveyed during safety audits to determine if new motor
carriers increase their knowledge of motor carrier requirements and change
their behavior to better comply with these requirements. The lack of such
an evaluation is a missed opportunity for FMCSA to demonstrate the
effectiveness of its educational information and safety audits, which
represented almost 7.5 percent of its funding in fiscal year 2005. In
addition, we continue to believe this is an important component of any
evaluation FMCSA develops to determine the effectiveness of the New
Entrant program. Therefore, we retained our recommendation to evaluate the
New Entrant program and conclude that there are two opportunities for
doing so-either through contracts for evaluations of education and
outreach programs or through the evaluation FMCSA is planning to start in
2008.
Finally, agency officials provided technical comments, which we
incorporated as appropriate.
We are sending copies of this report to the Honorable Norman Mineta,
Secretary of Transportation. We will also make copies available to others
upon request. In addition, the report will be available at no charge on
the GAO Web site at h ttp://www.gao.gov.
If you or your staff have any questions about this report, please contact
me at s iggerudk@gao.gov or (202) 512-2834. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on the
last page of this report. Major contributors to this report are listed in
appendix III.
Katherine Siggerud Director, Physical Infrastructure
Scope and Methodology Appendix I
To address our first objective and describe the scope and nature of the
Federal Motor Carrier Safety Administration's (FMCSA) education and
outreach programs, we reviewed program documentation and interviewed FMCSA
officials at agency headquarters, its Eastern Service Center in Maryland,
and its New Jersey division office. We selected this division office and
service center because they were significantly involved in all four of the
currently active education and outreach programs we reviewed, which are
the New Entrant, Commercial Safety Belt, Motor Coach Outreach, and
Household Goods Outreach programs. We obtained and reviewed information on
FMCSA's plans to initiate an additional program targeted at non-entrant
carriers.
Further, to determine how these programs relate to FMCSA's goals, we
reviewed its strategic planning and performance budgeting documents and
how these education and outreach programs were portrayed in the current
fiscal year 2006 documents. We also discussed the performance budgeting
process with an FMCSA senior budget analyst to identify the agency's plans
for the fiscal year 2007 and 2008 performance budgets. We also obtained
and reviewed the Department of Transportation's (DOT) strategic plan,
which FMCSA refers to in its performance budget. In addition, we obtained
the strategic plan for FMCSA's Research and Technology group. We reviewed
this plan to identify what one group within FMCSA uses for its strategic
and program planning. To identify other tools used for describing how
program activities relate to agency goals, we reviewed our previous work
and identified a program logic model, originally developed by the
University of Wisconsin Cooperative Extension, as a model that FMCSA could
use to clearly describe its education and outreach programs.
To address our second objective-identifying the extent to which FMCSA has
evaluated its education and outreach programs, we discussed the status of
these evaluation efforts with FMCSA officials and reviewed proposals and
statements of work for current and planned contracts to carry out program
evaluations. We also assessed the usefulness of FMCSA's ongoing or planned
program evaluations to determine if the ealuations would address how
program activities affected target audiences' attitudes, knowledge, and
behavior. We did not thoroughly assess the evaluation design.
We took several approaches to assess what is currently known about the
effectiveness of programs like FMCSA's education and outreach programs.
Finally, to identify what is known about the effectiveness of the
programs, we conducted a literature review of media based campaigns and
reviewed the structure and evaluation of these campaigns as well as
behavioral theories used in designing the campaigns. We selected two
behavioral theories that were directly applicable to the programs under
this review-the theory of reasoned action and the general deterrence
theory. We then directly compared the characteristics of the two theories
with the characteristics of FMCSA's programs. Additionally, through our
literature search, we also identified additional research that discusses
the importance of the relationship between education and enforcement in
changing behavior. We used this research to help assess the potential for
using enforcement to augment FMCSA's education and outreach efforts. We
obtained aggregated data from safety audit results from FMCSA's Motor
Carrier Management Information System (MCMIS) to examine results for
individual elements within a safety audit. We used these data to identify
the failure rates for key elements of the safety audit. We conducted a
reliability assessment of MCMIS data by (1) reviewing documentation
related to system development, (2) interviewing knowledgeable agency
officials, and (3) reviewing FMCSA's internal controls, and we determined
the data was sufficiently reliable for our use. Finally, we interviewed
officials from several motor carrier, law enforcement, and safety
organizations, including the American Trucking Associations; the National
Private Truck Council; the American Bus Association; the United Motorcoach
Association; the American Moving and Storage Association; the Commercial
Vehicle Safety Alliance; Advocates for Highway and Auto Safety; and Public
Citizen for their perspective of the effectiveness of FMCSA's education
and outreach efforts.
In accordance with congressional direction, we did not extend our review
to include "Share the Road Safely" which is targeted to passenger car
drivers. In fiscal year 2004, Congress transferred funding for the program
from FMCSA the National Highway Traffic Safety Administration (NHTSA).
Funding for this program is being returned to FMCSA in fiscal year 2006.
In fiscal years 2004 and 2005, Congress provided FMCSA with funding for
another outreach effort entitled "Safety Is Good Business." The SAFETEA-LU
requires GAO to review this program by June of 2006.
Our work was conducted in accordance with generally accepted government
auditing standards, from December 2004 through October 2005.
FMCSA Education and Outreach Programs Appendix II
The following provides a more detailed discussion of the five education
and outreach programs we reviewed.
New Entrant Program
The New Entrant program is intended to ensure that new motor carriers are
knowledgeable about federal motor carrier safety standards and applicable
hazardous material regulations. The program is focused on the safety
audits conducted through site visits to the new carriers' offices by
personnel from the state law enforcement agency, the FMCSA division
office, or a FMCSA contractor.
In 1998, a FMCSA commissioned study1 found that new entrants to the motor
carrier industry have a substantially higher level of safety regulation
violations than more experienced carriers. Other, earlier research had
also indicated that new entrant carriers had crash rates higher than more
experienced carriers.2 The Motor Carrier Safety Improvement Act, which
established FMCSA, also directed the Secretary of Transportation to
establish regulations specifying minimum requirements for applicant motor
carriers seeking federal interstate operating authority and to require
that new entrants undergo a safety audit within the first 18 months of
operations. Effective January 1, 2003, FMCSA established its New Entrant
program to improve the safety of new entrant motor carriers by providing
them with educational and technical assistance as they begin their
businesses. Under FMCSA's New Entrant program, any new motor carrier that
registers with DOT is designated a "new entrant" and will be subject to a
new entrant safety audit and increased roadside performance monitoring for
an 18-month period.
The safety audit's purpose is to provide educational and technical
assistance to the new entrant by reviewing the carrier's compliance with
FMCSA requirements, including driver qualifications, driver records of
duty status, vehicle maintenance records, accident registers, and
controlled substances and alcohol use and testing requirements. Education
and technical assistance information is also available to new entrants
through FMCSA's Web site. A carrier that passes the safety audit is
informed of any deficiencies and what it must do to be in compliance. A
carrier that fails the safety audit has 60 days3 to take necessary
corrective action, and ultimately if the deficiencies are not corrected,
its registration can be revoked. At the end of the 18-month period, if the
carrier has passed its safety audit, does not have an unsatisfactory
safety rating, and is not subject to an "out of service" order or any
notice to remedy safety management controls, the "new entrant" designation
will be removed from its registration, and it will be monitored like any
other carrier.
In most states, new entrant safety audits have been done by both state law
enforcement agency and FMCSA division office personnel. Contractors
conduct safety audits in four states (Florida, Maine, Oregon, and Wyoming)
that are presently doing few or none of their own. When the New Entrant
program began, FMCSA staff performed most of the safety audits, but states
now carry out the majority of these audits, as indicated by figure 5.
Figure 5: Number of New Entrant Safety Audits Completed by FMCSA and State
Agencies (first 7 months of fiscal year 2005)
Non-Entrant Program
FMCSA's Non-Entrant program will target motor carriers that operate
interstate but have not registered with DOT. The purpose of the program
will be to raise awareness of registration requirements, and to increase
the number of registrations among these non-entrant motor carriers. This
program has not yet begun, but FMCSA officials told us it will include
information disseminated through truck sales and leasing firms to
non-entrants.
The number of non-entrants is unknown, but a FMCSA official estimates it
may be in the tens of thousands. FMCSA officials whom we spoke with
believe that non-entrants may represent an even higher level of risk than
new entrants, because they do not get exposure to FMCSA safety information
through registration. The officials believe that, in general, motor
carriers that do not register with DOT are unaware of the requirement to
do so, rather than intentionally avoiding registering. FMCSA will be
awarding a contract for the development of the Non-Entrant program; the
contract is expected to include a requirement to identify the target
population of non-entrants, develop informational material, with follow-up
monitoring of non-entrants that are identified.
Commercial Safety Belt Program
FMCSA's Commercial Safety Belt Program is intended to improve upon the low
rate of safety-belt use by commercial truck drivers. In 2003, a study
commissioned by FMCSA reported that only 48 percent of commercial truck
drivers wear safety belts, compared with 80 percent of passenger car
drivers wearing safety belts. In December 2003, FMCSA established the
Commercial Motor Vehicle Safety Belt Partnership with 16 other
organizations, including trucking industry and law enforcement
associations, commercial drivers associations, and safety organizations,
with the goal of increasing commercial safety-belt use by commercial motor
vehicle drivers. With its safety belt partners, FMCSA produces materials
including posters, bumper stickers, and brochures for distribution at
trucking industry shows and events, and through law enforcement campaigns.
In 2004, as part of the Partnership, FMCSA helped distribute informational
brochures developed by the Commercial Vehicle Safety Alliance (CVSA).4 In
2005, FMCSA introduced and distributed new materials with the theme "Be
Ready. Be Buckled." for distribution at national and regional trucking
industry shows and events, and through the CVSA-sponsored Road Check 2005.
This was a 3-day truck and motor coach inspection effort in June 2005 that
involved over 9,900 inspectors, conducting more than 60,000 vehicle
inspections in the United States, Canada, and Mexico. Through the
Commercial Motor Vehicle Safety Belt Partnership, FMCSA also communicates
its message through information displayed at truck stops and satellite
radio programming oriented to truck drivers.
Motor Coach Outreach Program
FMCSA's objective for the motor coach industry is to reduce injuries and
save lives through increased compliance with safety regulations and
promotion of safe operations by motor coach owners and operators. FMCSA's
approach is two-fold-(1) reaching out to the industry to inform motor
coach companies of regulations and provide safety advice and (2) providing
consumers with information to help them in selecting and hiring motor
coach services. FMCSA's motor coach outreach activities include
communication with the carriers through printed material and participation
in trade shows, and Web-based information for the public. For the purposes
of our review, we focused on FMCSA's outreach to members of the public
that hire motor coach services.
Motor coaches have been one of the safest forms of commercial
transportation, but an increase in motor coach occupant fatalities since
2001 led FMCSA to increase its focus on motor coach companies. In 2002,
FMCSA developed "Moving Kids Safely" a toolkit of reference materials
targeted to school districts and others who contract for bus services. In
2004, FMCSA introduced a motor coach safety Web site within the FMCSA Web
site, which provides guidance to consumers on selecting a motor coach
company and allows consumers to access safety data on specific interstate
motor coach companies.
FMCSA's most direct outreach to motor coach companies is through the
agency's Technical Assistance Group (TAG), consisting of about 12 FMCSA
staff with motor coach expertise, primarily in FMCSA's field offices. TAG
personnel staff information booths and give presentations at industry
conventions and events, and serve as agency contact points for carriers.
Also, a FMCSA contractor developed a series of brochures aimed at motor
coach companies that describe guidelines and regulations for motor coach
companies (for instance licensing and insurance, hours of service,
inspection and maintenance, and specific requirements for operators of
school buses). In 2004 and 2005, FMCSA also conducted mass mailings to
motor coach companies, advising them of their reporting requirements under
the Americans with Disabilities Act.
Household Goods Outreach Program
Most households need to hire moving services infrequently, and therefore
consumers may be unaware of what their rights and protections are in
hiring a commercial mover. The overriding objective of FMCSA's Household
Goods program is to help consumers make better informed choices in
selecting and negotiating with a moving company, primarily through printed
material distributed through moving companies and other sources, as well
as providing information through its Web site. FMCSA also maintains a
complaint hotline to receive complaints about movers from individuals.
The ICC Termination Act of 1995 transferred federal responsibilities for
protecting consumers who use commercial moving companies for interstate
moves to DOT; and the Motor Carrier Safety Improvement Act granted FMCSA
continuing oversight responsibility over household goods movers. However,
the act did not provide FMCSA with the authority to intercede on behalf of
individuals seeking reimbursement of moving company charges or recovery of
their goods. FMCSA officials whom we spoke with acknowledge that when the
agency was established in January 2000, its mandate was commercial vehicle
accidents and fatalities and, as a result, the household goods industry
received little attention. Also, in 2001, we reported that complaints
about movers were on the rise and that DOT's lack of action had created a
vacuum that allowed unscrupulous carriers to take advantage of consumers.5
At that time, our report noted that FMCSA had just begun planning to
increase both consumer education and enforcement effort of household goods
carriers.
FMCSA distributes three consumer-oriented publications. Interstate movers
are required by law to provide a copy of one FMCSA publication- "Your
Rights and Responsibilities When You Move" to prospective customers. A
second brochure, "Ready to Move? Tips for a Successful Interstate Move" is
available through the General Services Administration's Pueblo, Colorado
Information Center. A third brochure "Protect your Memories...Protect
Yourself from Moving Fraud" and the "Your Rights and Responsibilities"
publications are available on FMCSA's Web site.
In March 2005, FMCSA awarded a contract for the development of a household
goods education and outreach program. FMCSA and the contractor have
developed a new outreach theme: "Protect Your Memories, Your Money, Your
Move"; and the contractor will be developing brochures, presentations that
can be delivered by FMCSA staff, and other materials. In June 2005, FMCSA
launched its redesigned household goods Web page, designed around the
"Protect Your Memories..." theme, which incorporates new content,
including a checklist for moving and information on protecting against
moving fraud. In November 2005, FMCSA officials informed us that they were
in discussions with the U.S. Postal Service to have the Postal Service Web
site provide a link to FMCSA's Web site when registering a change of
address with the Postal Service.
FMCSA officials told us they do not arbitrate complaints that the agency
receives, but they maintain a national consumer complaint database that is
used to identify problem movers for investigation by FMCSA field staff.
Our 2001 report on DOT and FMCSA oversight of the household goods industry
recommended that FMCSA make information on the number and general nature
of complaints against carriers be made available to the public.6 FMCSA
officials whom we spoke with said the agency intends to do this, but
Privacy Act and other issues need to be resolved beforehand.
GAO Contact and Staff Acknowledgments Appendix III
Katherine Siggerud, (202) 512-2834
In addition to the person named above, Cathy Colwell, Assistant Director;
Colin Fallon; Eric Fielding; Donald Kittler; Sara Ann Moessbauer; and
Elaine Vaurio also made major contributions to this report.
(542051)
www.gao.gov/cgi-bin/getrpt? GAO-06-103 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Katherine Siggerud at (202) 512-2834 or at
siggerudk@gao.gov.
Highlights of GAO-06-103 , a report to congressional committees
December 2005
FEDERAL MOTOR CARRIER SAFETY ADMINISTRATION
Education and Outreach Programs Target Safety and Consumer Issues, but
Gaps in Planning and Evaluation Remain
The Federal Motor Carrier Safety Administration (FMCSA) is responsible for
improving commercial vehicle safety and uses education and outreach as
part of its efforts. The House report accompanying the fiscal year 2005
Department of Transportation (DOT) appropriations bill asked GAO to report
on FMCSA's education and outreach programs to the House and Senate
Committees on Appropriations. GAO (1) describes FMCSA's education and
outreach programs and how they relate to FMCSA's goals (2) identifies the
extent to which FMCSA has evaluated its education and outreach programs
and (3) describes the extent to which FMCSA's education and outreach
programs are effective.
What GAO Recommends
GAO recommends that the Secretary of Transportation direct the FMCSA
Administrator to describe the link between education and outreach programs
and agency strategic objectives and evaluate the extent to which
educational information and safety audits are helping new carriers learn
FMCSA requirements. GAO provided a draft of this report to DOT for its
review and comment. FMCSA officials commented on the link between its
education and outreach programs and its overall goals. Based on FMCSA's
comments, GAO acknowledged FMCSA's comments and clarified the
recommendations.
FMCSA's education and outreach programs-New Entrant, Non-Entrant, Motor
Coach, Safety Belt, and Household Goods- target different audiences,
including the motor carrier industry, commercial vehicle drivers, and the
public.Total funding for these programs in fiscal year 2005 was $36.3
million; the largest share (about $33 million) went to the New Entrant
program, which is designed to inform newly registered motor carriers (new
entrants) about motor carrier safety standards and regulations to help
them gain compliance with FMCSA requirements. FMCSA uses many approaches,
such as direct contact with carriers, media campaigns, distributing
printed materials, and establishing Web sites to provide information to
target audiences. FMCSA has not described how its education and outreach
program activities link expected changes in attitudes and behavior to
broader goals, such as DOT's strategic objective of reducing
transportation-related fatalities. FMCSA officials state that the
education and outreach activities and programs link to agency goals at a
high level, but this was not evident from our review, with the exception
of the Safety Belt program. FMCSA has used a logic model as a tool in
other programs to show the relationship between program activities and
broader goals.
FMCSA has begun some evaluations of its education and outreach programs,
and plans other evaluations of these programs. However, although FMCSA's
New Entrant program has existed for over 2 years, FMCSA has no plans to
evaluate its New Entrant program until 2008. Thus FMCSA has no information
on whether information on its safety requirements, provided through the
Education and Technical Assistance package or during New Entrant safety
audits-targeted toward truckers newly entering the industry-effectively
communicate information to new entrants. This lack of evaluation makes it
difficult to determine the impact the education portion of the New Entrant
program has on commercial motor vehicle safety.
Since FMCSA currently has little information on how its programs have
affected attitudes and behavior, it is difficult to determine the
effectiveness of FMCSA's effort. However, the designs of two programs
appear to follow theories and research regarding media campaigns, which
are intended to influence decision making about safety. Research and
behavior theory suggest that for some types of programs-such as DOT's
Click It or Ticket program, which is designed to increase safety-belt use
by passenger car drivers-enforcement linked to education can improve
results, and FMCSA has indicated it is linking some education and outreach
programs to enforcement efforts, where appropriate. Finally, motor carrier
association officials whom we spoke with stated that, in their view, FMCSA
is doing some positive things in its education and outreach activities. A
public safety group stated that FMCSA followed reasonable approaches in
starting its education and outreach efforts; however, they would like to
see more information on program effectiveness to help FMCSA refine the
programs.
*** End of document. ***