Motor Carrier Safety: A Statistical Approach Will Better Identify
Commercial Carriers That Pose High Crash Risks Than Does the
Current Federal Approach (11-JUN-07, GAO-07-585).
The Federal Motor Carrier Safety Administration (FMCSA) has the
primary federal responsibility for reducing crashes involving
large trucks and buses that operate in interstate commerce. FMCSA
decides which motor carriers to review for compliance with its
safety regulations primarily by using an automated, data-driven
analysis model called SafeStat. SafeStat uses data on crashes and
other data to assign carriers priorities for compliance reviews.
GAO assessed (1) the extent to which changes to the SafeStat
model could improve its ability to identify carriers that pose
high crash risks and (2) how the quality of the data used affects
SafeStat's performance. To carry out its work, GAO analyzed how
SafeStat identified high-risk carriers in 2004 and compared these
results with crash data through 2005.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-585
ACCNO: A70566
TITLE: Motor Carrier Safety: A Statistical Approach Will Better
Identify Commercial Carriers That Pose High Crash Risks Than Does
the Current Federal Approach
DATE: 06/11/2007
SUBJECT: Accident prevention
Accidents
Automated risk assessment
Comparative analysis
Data collection
Data integrity
Evaluation methods
Motor carriers
Motor vehicle safety
Motor vehicles
Program evaluation
Risk factors
Risk management
Safety regulation
Safety standards
Systems evaluation
Traffic accidents
FMCSA Safety Status Measurement System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-585
* [1]Results in Brief
* [2]Background
* [3]A Statistical Approach Would Better Identify Carriers That P
* [4]Regression Models Identify Carriers That Pose High Crash Ris
* [5]FMCSA Can Apply a Regression Model Approach in the Short Ter
* [6]Modifications of SafeStat Did Not Improve Crash Identificati
* [7]Despite Quality Problems, FMCSA's Crash Data Can Be Used to
* [8]Late Reporting Had a Small Effect on SafeStat's Ability to I
* [9]Incomplete Data from States Potentially Limit SafeStat's Ide
* [10]Inaccurate Data Potentially Limit SafeStat's Ability to Iden
* [11]FMCSA Has Undertaken Efforts to Improve Crash Data Quality
* [12]Conclusion
* [13]Recommendation for Executive Action
* [14]Agency Comments and Our Evaluation
* [15]Appendix I: Results of Other Assessments of the SafeStat Mod
* [16]Assessments of SafeStat's Predictive Capability
* [17]Predictive Capability of SafeStat Compared with Random
Selec
* [18]Application of Regression Models to Safety Data
* [19]Relationship of Carrier Financial Data and Safety Risk
* [20]Relationship of Commercial Driver License Convictions
and Cr
* [21]Impact of Data Quality on SafeStat's Predictive Capability
* [22]Late Reporting of Crash Data
* [23]Incomplete and Inaccurate Reporting of Crash Data
* [24]Appendix II: Scope and Methodology
* [25]Appendix III: Additional Results from Our Statistical Analys
* [26]Overview of Regression Analyses
* [27]Technical Explanation of the Negative Binomial Regression Mo
* [28]Evaluation of Regression Models' Performance
* [29]Order by Mail or Phone
Report to Congressional Requesters
United States Government Accountability Office
GAO
June 2007
MOTOR CARRIER SAFETY
A Statistical Approach Will Better Identify Commercial Carriers That Pose
High Crash Risks Than Does the Current Federal Approach
GAO-07-585
Contents
Letter 1
Results in Brief 3
Background 6
A Statistical Approach Would Better Identify Carriers That Pose High Crash
Risks Than Does FMCSA's Current Approach 13
Despite Quality Problems, FMCSA's Crash Data Can Be Used to Compare
Methods for Identifying Carriers That Pose High Crash Risks 22
Conclusion 28
Recommendation for Executive Action 28
Agency Comments and Our Evaluation 28
Appendix I Results of Other Assessments of the SafeStat Model's Ability to
Identify Motor Carriers That Pose High Crash Risks 32
Assessments of SafeStat's Predictive Capability 32
Impact of Data Quality on SafeStat's Predictive Capability 36
Appendix II Scope and Methodology 40
Appendix III Additional Results from Our Statistical Analyses of the
SafeStat Model 43
Overview of Regression Analyses 43
Technical Explanation of the Negative Binomial Regression Model 45
Tables
Table 1: SafeStat Categories and Their Priority for Compliance Reviews 11
Table 2: Size Distribution of Carriers Receiving a SafeStat Rating of A
through G 13
Table 3: Results for SafeStat Model and Regression Models 49
Figures
Figure 1: Commercial Motor Vehicle Fatality Rate, 1975 to 2005 7
Figure 2: Percentage of Crashes Submitted to MCMIS within 90 Days of
Occurrence 24
Abbreviations
FMCSA Federal Motor Carrier Safety Administration
MCMIS Motor Carrier Management Information System
SafeStat Motor Carrier Safety Status Measurement System
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
June 11, 2007
The Honorable James L. Oberstar
Chairman
The Honorable John L. Mica
Ranking Republican Member
Committee on Transportation and Infrastructure
House of Representatives
The Honorable James L. Oberstar
Chairman
The Honorable John L. Mica
Ranking Republican Member
Committee on Transportation and Infrastructure
House of Representatives
The Honorable Peter A. DeFazio
ChairmanThe Honorable John J. Duncan
Ranking Republican Member
Subcommittee on Highways and Transit
Committee on Transportation and Infrastructure
House of Representatives
The Honorable Peter A. DeFazio
Chairman
The Honorable John J. Duncan
Ranking Republican Member
Subcommittee on Highways and Transit
Committee on Transportation and Infrastructure
House of Representatives
The Honorable Thomas E. Petri
House of Representatives
The Honorable Thomas E. Petri
House of Representatives
The Federal Motor Carrier Safety Administration (FMCSA) within the U.S.
Department of Transportation has the primary federal responsibility for
reducing crashes, deaths, and injuries involving large trucks and buses
operating in interstate commerce. While it carries out a number of
activities toward this end, an important tool at its disposal is the
compliance review--a detailed inspection of a motor carrier's operations
at its place of business. FMCSA decides which carriers to inspect
primarily by using an automated, data-driven analysis system called the
Motor Carrier Safety Status Measurement System (SafeStat). SafeStat uses
data on crashes, vehicle and driver violations, and other information to
develop numerical scores for carriers, and then SafeStat assigns each
carrier a priority to receive a compliance review. The Federal Motor
Carrier Safety Administration (FMCSA) within the U.S. Department of
Transportation has the primary federal responsibility for reducing
crashes, deaths, and injuries involving large trucks and buses operating
in interstate commerce. While it carries out a number of activities toward
this end, an important tool at its disposal is the compliance review--a
detailed inspection of a motor carrier's operations at its place of
business. FMCSA decides which carriers to inspect primarily by using an
automated, data-driven analysis system called the Motor Carrier Safety
Status Measurement System (SafeStat). SafeStat uses data on crashes,
vehicle and driver violations, and other information to develop numerical
scores for carriers, and then SafeStat assigns each carrier a priority to
receive a compliance review.
Following an incident in which a bus company, with many driver violations
and a low priority for compliance review from the SafeStat model, suffered
a fire on one of its buses that resulted in 23 deaths, you were interested
in whether SafeStat could better identify commercial motor carriers at
risk for crashes. To address your interest, we assessed (1) the extent to
which changes to the SafeStat model could improve its ability to identify
these carriers and (2) how the quality of the data used affects SafeStat's
performance. These two topics are the main focus of this Following an
incident in which a bus company, with many driver violations and a low
priority for compliance review from the SafeStat model, suffered a fire on
one of its buses that resulted in 23 deaths, you were interested in
whether SafeStat could better identify commercial motor carriers at risk
for crashes. To address your interest, we assessed (1) the extent to which
changes to the SafeStat model could improve its ability to identify these
carriers and (2) how the quality of the data used affects SafeStat's
performance. These two topics are the main focus of this report. We also
examined the findings of other studies on how SafeStat's ability to
identify carriers at risk for crashes can be improved. (See app. I.)
To determine whether statistical approaches could be used to improve
FMCSA's ability to identify carriers that pose high crash risks, we tested
a number of regression models and compared their performance with
SafeStat's results from June 2004. We chose 2004 because it allowed us to
examine actual crash data for the 18-month period following June 2004 to
determine the degree to which SafeStat successfully identified carriers
that proved to be of high risk for crashes. It also allowed us to include
crashes that occurred within the 18 months after June 2004 but had not yet
been reported to FMCSA by December 2005. Using regression models, we
compared the predictive performance of these statistical approaches to
SafeStat's performance to determine which method best identified carriers
that pose high crash risks. We also calculated crash rates from a series
of random samples of all carriers to determine if the SafeStat model did a
better job than random selection in identifying motor carriers that pose
high crash risks. To assess whether changes could be made to the SafeStat
model to improve its identification of carriers that pose high crash
risks, we tested changes to selected portions of the SafeStat model and
investigated the effect of changing decision rules used to construct the
four safety evaluation areas.^1
To assess the extent to which data quality affects SafeStat's ability to
identify carriers that pose high crash risks, we carried out a series of
analyses and surveyed the literature to identify findings from other
studies. To address timeliness, we measured the number of days it took
states to report crashes. We also added late-reported crashes to FMCSA's
June 2004 data and recalculated SafeStat scores to determine the effect of
late-reported crashes on carriers' rankings. For completeness, we
attempted to match crash records in FMCSA's Motor Carrier Management
Information System (MCMIS) crash master file to motor carriers listed in
the MCMIS census file and reviewed studies on state reporting. To address
accuracy, we reviewed a report that tested the accuracy of electronic data
on a sample of paper records and studies that identified the impact of
incorrectly reported crashes in individual states on MCMIS data quality.
While there are known problems with the quality of the crash data reported
to FMCSA for use in SafeStat, we determined that the data were of
sufficient quality for our use, which was to compare the ability of
regression models to identify carriers that pose high crash risks to the
current approach, which is largely derived through professional judgment.
We conducted our work in accordance with generally accepted government
auditing standards from May 2006 through May 2007. Appendix II provides
further information on our scope and methodology.
^1There are four safety evaluation areas--accident, driver, vehicle, and
safety management. They are used by the SafeStat model to assess a
carrier's safety. See the background section for a description of these
four areas. SafeStat is built on a number of expert judgments rather than
using statistical approaches, such as a regression model.
Shortly, we expect to issue a related report that examines how FMCSA
identifies and takes action against carriers that are egregious safety
violators. In addition, that report examines how thoroughly and
consistently FMCSA conducts compliance reviews.
Results in Brief
While SafeStat does a better job of identifying motor carriers that pose
high crash risks than does a random selection, regression models we
applied do an even better job. SafeStat works about twice as well as
(about 83 percent better than) selecting carriers randomly and, therefore,
has value for improving safety. SafeStat is built on a number of expert
judgments. For example, SafeStat's designers used their judgment and
experience to weight more recent crashes involving a motor carrier twice
as much as less recent crashes on the premise that more recent crashes
were stronger indicators that a carrier may have crashes in the future.
Using similar reasoning, fatal crashes were weighted more heavily than
less serious crashes. We found that if a negative binomial regression
model was used instead, FMCSA could increase its ability to identify
carriers that pose high crash risks by about 9 percent over SafeStat.^2
Moreover, according to our analysis, this 9 percent improvement would
enable FMCSA to identify carriers with twice as many crashes in the
following 18 months as those carriers identified under its current
approach.^3 Carriers identified by the negative binomial regression model
as posing a high crash risk experienced 9,500 more crashes than those
identified by the SafeStat model over an 18 month follow-up period. The
primary use of SafeStat is to identify and prioritize carriers for FMCSA
and state safety compliance reviews. FMCSA measures the ability of
SafeStat to perform this role by comparing the crash rate of carriers
identified as posing a high crash risk with the crash rate of other
carriers. In our view, using a negative binomial regression model would
further FMCSA's mission of reducing crashes through the more effective
targeting of safety improvement and enforcement programs to the set of
carriers that pose the greatest crash risk. Applying a regression model
would be easy to adapt to the existing SafeStat model and, in our opinion,
would be beneficial even if FMCSA makes major revisions to its compliance
and enforcement approach in the coming years under its Comprehensive
Safety Analysis 2010 initiative.^4
2Negative binomial regression is often used to model count data (e.g.,
crashes). The results from this regression model can be interpreted as the
estimated mean number of crashes per carrier.
^3The 9 percent improvement is in crash rate per 1,000 vehicles over an
18-month period.
Crash data reported by the states from December 2001 through June 2004
have problems in terms of timeliness, accuracy, and completeness that
potentially hinder FMCSA's ability to identify high risk carriers.
Regarding timeliness, we found that including late-reported data had a
small impact on SafeStat--including late-reported data added a net of 299
(or 6 percent) more carriers to the original 4,989 carriers that the
SafeStat model ranked as highest risk in June 2004.^5 The timeliness of
crash reporting has shown steady and marked improvement: the percentage of
crashes reported by states within 90 days of occurrence jumped from 32
percent in fiscal year 2000 to 89 percent in fiscal year 2006. Regarding
completeness, data for about 21 percent of the crashes (about 39,000 of
184,000) exhibited problems that hampered linking crashes to motor
carriers. Thirteen percent of the crashes (about 24,000) involving
interstate carriers reported to FMCSA from December 2001 through June 2004
are missing the unique identifier that FMCSA assigns to each carrier when
the agency authorizes the carrier to engage in interstate commerce.
Crashes without a unique identifier to link to a company are excluded from
use in SafeStat. An additional 8 percent of the crashes (about 15,000)
that were reported had an identification number that could not be matched
to a motor carrier in the FMCSA database that contains census information
on motor carriers. Linking crashes to carriers is important because the
current SafeStat model treats crashes as the most important factor in
assessing motor carrier crash risk. Crash information is also the crucial
factor in the regression models that we employed. Regarding accuracy, a
series of University of Michigan Transportation Research Institute's
reports on crash reporting shows that, among the 14 states studied,
incorrect reporting of crash data is widespread. For example, in recent
reports, the researchers found that, in 2005, Ohio incorrectly reported
1,094 (22 percent) of the 5,037 cases it reported, and Louisiana
incorrectly reported 137 (5 percent) of the 2,699 cases it reported. In
Ohio, most of the 1,094 crashes did not qualify because they did not meet
the crash severity threshold.^6 We were not able to quantify the actual
effect of the incomplete or inaccurate data on SafeStat's ability to
identify carriers that pose high crash risks, because it would have
required us to gather crash records at the state level--an effort that was
impractical. FMCSA has acted to improve the quality of SafeStat's data by
completing a comprehensive plan for data quality improvement, implementing
an approach to correct inaccurate data, and providing grants to states for
improving data quality, among other things. We could not quantify the
effects of FMCSA's efforts to improve the completeness or accuracy of the
data for the same reason as mentioned above.
^4The goal of this initiative is to develop an optimal operational model
that will allow FMCSA to focus its resources on improving the safety
performance of high-risk operators.
^5We applied the SafeStat model to retrospective data. Because of changes
to the MCMIS crash file over the past 2 years, our number does not
correspond exactly to the number of carriers identified by FMCSA as high
risk on June 25, 2004. Had all crash data been reported within 90 days of
when the crashes occurred, 182 of the carriers identified by SafeStat as
highest risk would have been excluded (because other carriers had higher
crash risks), and 481 carriers that were not originally designated as
posing high crash risks would have scored high enough to be considered
high risk, resulting in a net addition of 299 carriers.
This report contains a recommendation to the Secretary of Transportation
aimed at applying a negative binomial regression model to the four
SafeStat safety evaluation areas that would result in better
identification of commercial motor carriers that pose high crash risks.
Because FMCSA has initiated efforts to improve the quality of SafeStat's
data, we are not making a recommendation in this area.
In commenting on a draft of this report, the department agreed that it
would be reasonable to consider the use of the negative binomial
regression model in order to better target compliance reviews to carriers
posing high crash risks, but expressed some concerns about placing more
emphasis on crash information and less on other factors, such as driver,
vehicle, or safety management issues. In addition, FMCSA noted that, while
it has devoted considerable efforts to improving the quality of crash data
submitted by states, the negative binomial regression model is more
sensitive than SafeStat to problems with the crash data.
^6A reportable crash is one that meets both a vehicle and a crash severity
threshold. Generally, for a crash to be reported, it must involve a truck
with a gross vehicle weight rating of over 10,000 pounds; a bus with
seating for at least nine people, including the driver; or a vehicle
displaying a hazardous materials placard. Reportable accidents involve a
fatality, an injury requiring transport to a medical facility for
immediate medical attention, or towing required because the vehicle
sustained disabling damage.
Background
The interstate commercial motor carrier industry, primarily the trucking
industry, is an important part of the nation's economy. Trucks transport
over 11 billion tons of goods annually, or about 60 percent of the total
domestic tonnage shipped.^7 Buses also play an important role,
transporting an estimated 631 million passengers annually. There are
approximately 711,000 commercial motor carriers registered in MCMIS,^8
about 9 million trucks and buses, and more than 10 million drivers. Most
motor carriers are small; about 51 percent operate one vehicle, and
another 31 percent operate two to four vehicles. Carrier operations vary
widely in size, however, and some of the largest motor carriers operate
upwards of 50,000 vehicles. Carriers continually enter and exit the
industry. Since 1998, the industry has increased in size by an average of
about 29,000 interstate carriers per year.
In the United States, commercial motor carriers account for less than 5
percent of all highway crashes, but these crashes result in about 13
percent of all highway deaths, or about 5,500 of the approximately 43,000
highway fatalities that occur nationwide annually. In addition, about
106,000 of the approximately 2.7 million highway injuries per year involve
motor carriers. The fatality rate for trucks has generally been decreasing
over the past 30 years, but this decrease has leveled off, and the rate
has been fairly stable since the mid-1990s. The fatality rate for buses
has improved slightly from 1975 to 2005 but has more annual variability
than the fatality rate for trucks due to a much smaller total vehicle
miles traveled. (See fig. 1.)
^7This figure is for 2002, the most recent date for which data is
available.
^8This includes an unidentified number of carriers that are registered but
are no longer in business.
Figure 1: Commercial Motor Vehicle Fatality Rate, 1975 to 2005
Congress created FMCSA through the Motor Carrier Safety Improvement Act of
1999 to reduce crashes, injuries, and fatalities involving commercial
motor vehicles. To accomplish this mission, FMCSA carries out a number of
enforcement, education, and outreach activities. FMCSA uses enforcement as
its primary approach for reducing the number of crashes, fatalities, and
injuries involving trucks and buses. Some of FMCSA's enforcement programs
include compliance reviews, which are on-site reviews of carriers' records
and operations to determine compliance with regulations; safety audits of
new interstate carriers; and roadside inspections of drivers and vehicles.
FMCSA's education and outreach programs are intended to promote motor
carrier safety and consumer awareness. One of the programs is the New
Entrant program, which is designed to inform newly registered motor
carriers about motor carrier safety standards and regulations to help them
comply with FMCSA's requirements. Other programs are designed to identify
unregistered carriers and get them to register, promote increased safety
belt use among commercial drivers, and inform organizations and
individuals that hire buses how to make safe choices. FMCSA plans to make
major revisions to its compliance and enforcement approach under an
initiative called Comprehensive Safety Analysis 2010.
Compliance reviews are an important enforcement tool because they allow
FMCSA to take an in-depth look at carriers that have been identified as
posing high crash risks because of high crash rates or poor safety
performance records. Motor carriers may be identified as high risk from
SafeStat or through calls to FMCSA's complaint hotline. Carriers are given
a satisfactory, conditional, or unsatisfactory safety rating. A
conditional rating means the carrier is allowed to continue operating, but
FMCSA may schedule a follow-up compliance review to ensure that problems
noted in the first compliance review are addressed. An unsatisfactory
rating must be addressed or the carrier is placed out of service, meaning
it is no longer allowed to do business, and the carrier may face legal
enforcement actions undertaken by FMCSA. Compliance reviews can take
several days to complete, depending on the size of the carrier, and may
result in enforcement actions being taken against a carrier.
FMCSA uses both its own inspectors and state inspectors to carry out its
enforcement activities. In total, about 750 staff are available to perform
compliance reviews, and more than 10,000 staff do vehicle and driver
inspections at weigh stations and other points. Together, FMCSA and its
state partners perform about 16,000 compliance reviews a year, which cover
about 2 percent of the nation's 711,000 carriers.^9
Because the number of inspectors is small compared with the size of the
motor carrier industry, FMCSA prioritizes carriers for compliance reviews.
To do so, it uses SafeStat to identify carriers that pose high crash
risks. SafeStat is a model that uses information gathered from crashes,
roadside inspections, traffic violations, compliance reviews, and
enforcement cases to determine a motor carrier's safety performance
relative to that of other motor carriers that have similar exposure in
these areas. A carrier's score is calculated on the basis of its
performance in four safety evaluation areas:
o Accident safety evaluation area: The accident safety evaluation
area reflects a carrier's crash history relative to other motor
carriers' histories. The safety evaluation area is based on
state-reported crash data, vehicle data from MCMIS, and data on
reportable crashes and annual vehicle miles traveled from the most
recent compliance review. A carrier must have two or more
reportable crashes within the last 30 months to have the potential
to receive a deficient value and thus be made a priority for a
compliance review.
^9FMCSA completed 15,626 compliance reviews in 2006. The number of
companies reviewed was less because some carriers received more than 1
compliance review.
o Driver safety evaluation area: The driver safety evaluation area
reflects a carrier's driver-related safety performance and
compliance relative to other motor carriers. The driver safety
evaluation area is based on violations cited in roadside
inspections that have been performed within the last 30 months and
compliance reviews that have occurred within the last 18 months,
together with the number of drivers listed in MCMIS. A carrier
must have three or more driver inspections, three or more moving
violations, or at least one acute or critical violation of driver
regulations^10 from a compliance review to have the potential to
receive a deficient value and thus be made a priority for a
compliance review.
o Vehicle safety evaluation area: The vehicle safety evaluation
area reflects a carrier's vehicle-related safety performance and
compliance relative to other motor carriers. The vehicle safety
evaluation area is based on violations identified during vehicle
roadside inspections that have occurred within the last 30 months
or vehicle-related acute and critical violations of regulations
discovered during compliance reviews that have occurred within the
last 18 months. A carrier must have either three or more vehicle
inspections or at least one acute or critical violation of vehicle
regulations from a compliance review to have the potential to
receive a deficient value and thus be made a priority for a
compliance review.
o Safety management safety evaluation area: The safety management
safety evaluation area reflects a carrier's safety management
relative to other motor carriers. It is based on the results of
violations cited in closed enforcement cases in the past 6 years
or violations of regulations related to hazardous materials and
safety management discovered during a compliance review performed
within the last 18 months. A carrier must have had at least one
enforcement case initiated and closed or at least two enforcement
cases closed within the past 6 years, or at least one acute,
critical, or severe violation of hazardous material or safety
management regulations^11 identified during a compliance review
within the last 18 months to have the potential to receive a
deficient value and thus be made a priority for a compliance
review.
^10Acute violations are violations so severe that FMCSA requires immediate
corrective actions by a motor carrier regardless of the carrier's overall
safety status. An example of an acute violation is a carrier's failing to
implement an alcohol or drug testing program for drivers. Critical
violations are serious, but less severe than acute violations, and most
often point to gaps in carriers' management or operational controls. For
example, a carrier may not maintain records of driver medical
certificates.
A motor carrier's score is based on its relative ranking,
indicated as a value, in each of the four safety evaluation areas.
For example, if a carrier receives a value of 75 in the accident
safety evaluation area, then 75 percent of all carriers with
sufficient data for evaluation performed better in that safety
evaluation area, while 25 percent performed worse. The calculation
used to determine a motor carrier's SafeStat score is as follows:
SafeStat Score = (2.0x accident value) + (1.5x driver value)
+ vehicle value + safety management value
As shown in the formula, the accident and driver safety evaluation
areas have 2.0 and 1.5 times the weight, respectively, of the
vehicle and safety management safety evaluation areas. Safety
evaluation area values less than 75 are ignored in the formula
used to determine the SafeStat score. For example, a carrier with
values of 74 for all four safety evaluation areas has a SafeStat
score of 0. FMCSA assigned more weight to these safety evaluation
areas because, according to FMCSA, crashes and driver violations
correlate relatively better with future crash risk. In addition,
more weight is assigned to fatal crashes and to crashes that
occurred within the last 18 months. In consultation with state
transportation officials, insurance industry representatives,
safety advocates, and the motor carrier industry, FMCSA used its
expert judgment and professional knowledge to assign these
weights, rather than determining them through a statistical
approach, such as regression modeling.
FMCSA assigns carriers categories ranging from A to H according to
their performance in each of the safety evaluation areas. A
carrier is considered to be deficient in a safety evaluation area
if it receives a value of 75 or higher in that particular safety
evaluation area. Although a carrier may receive a value in any of
the four safety evaluation areas, the carrier receives a SafeStat
score only if it is deficient in one or more safety evaluation
areas. Carriers that are deficient in two or more safety
evaluation areas and have a SafeStat score of 225 or more are
considered to pose high crash risks and are placed in category A
or B. (See table 1.) Carriers that are deficient in two safety
evaluation areas but have a SafeStat score of less than 225 are
placed in category C and receive a medium priority for compliance
reviews. Carriers that are deficient in only one of the safety
evaluation areas are placed in category D, E, F, or G. Carriers
that are not deficient in any of the safety evaluation areas do
not receive a SafeStat score and are placed in category H.
^11Severe violations are violations of hazardous materials regulations.
Level I violations require immediate corrective actions. An example of a
level I violation is offering or accepting a hazardous material for
transportation in an unauthorized vehicle. Level II violations indicate a
breakdown in the management or operational controls of the facility. An
example of a level II violation is failing to train hazardous materials
employees as required.
Table 1: SafeStat Categories and Their Priority for Compliance Reviews
Priority for compliance
Category Condition review
A Deficient in all four safety evaluation High
areas
or
Deficient in three safety evaluation
areas that result in a weighted SafeStat
score of 350 or more
B Deficient in three safety evaluation High
areas that result in a weighted SafeStat
score of less than 350
or
Deficient in two safety evaluation areas
that result in a weighted SafeStat score
of 225 or more
C Deficient in two safety evaluation areas Medium
that result in a weighted SafeStat score
of less than 225
D Deficient in the accident safety Low
evaluation area (accident safety
evaluation area value between 75-100)
E Deficient in the driver safety evaluation Low
area (driver safety evaluation area value
between 75-100)
F Deficient in the vehicle safety Low
evaluation area (vehicle safety
evaluation area value between 75-100)
G Deficient in the safety management safety Low
evaluation area (safety management safety
evaluation area value between 75-100)
H Not deficient in any of the safety Low
evaluation areas (value below 75 in each
of the safety evaluation areas)
Source: GAO summary of FMCSA data.
Of the 622,000 motor carriers listed in MCMIS as having one or more
vehicles in June 2004, about 140,000, or 23 percent, received a SafeStat
category A through H. There are several reasons why a small proportion of
carriers receive a score. First, approximately 305,900, or about 42
percent, of the carriers have crash, vehicle inspection, driver
inspection, or enforcement data of any kind. SafeStat relies on these data
to calculate a motor carrier's score, so carriers without such data are
not rated by SafeStat. It is likely that some of the carriers listed in
MCMIS are no longer in business, but it is also possible that these
carriers had no crashes, inspections, or compliance reviews in the
30-month period prior to June 2004. Second, a carrier must meet the
minimum requirements to be assigned a value in a given safety evaluation
area.^12 If, for example, a carrier had only one reportable crash within
the last 30 months, then the carrier would not be assigned an accident
safety evaluation area value. Of the 305,900 carriers that have any safety
data in SafeStat, 140,000 met the SafeStat minimum requirements in one or
more safety evaluation areas. Of these 140,000 carriers, 45,000 were rated
in categories A through G. The other carriers were placed in category H
because they were not considered deficient, meaning they did not receive a
value of 75 or more in any of the safety evaluation areas.
The design of SafeStat and its data sufficiency requirements increase the
likelihood that larger motor carriers will be deficient in one of the
safety evaluation areas, in other words, rated in categories A through G,
than are small carriers. About 51 percent of all carriers listed in MCMIS
operate one vehicle, and about 3 percent of them received a SafeStat
rating in categories A through G. (See table 2.) In contrast, fewer than 1
percent of the carriers listed in MCMIS have more than 100 vehicles, and
nearly 25 percent of them received a SafeStat rating in categories A
through G.
^12Minimum requirements in this context mean that the carrier has enough
safety data to receive a rating. Usually, the safety data are associated
with adverse safety events. However it is possible for a carrier to have
enough roadside inspections, even if none of the inspections resulted in
violations, to qualify for a driver and vehicle safety evaluation area
score.
Table 2: Size Distribution of Carriers Receiving a SafeStat Rating of A
through G
Carrier size Number of carriers within size category
(number of Number of carriers receiving A through G SafeStat rating
vehicles) (percentage^a) (percentage of carriers in size category)
1 317,037 (51%) 8,697 (3%)
>1 to 4 191,739 (31%) 14,430 (8%)
>4 to 10 66,422 (11%) 10,595 (16%)
>10 to 25 28,780 (5%) 6,504 (23%)
>25 to 100 14,148 (2%) 3,550 (25%)
>100 3,903 (1%) 909 (23%)
Source: GAO analysis of FMCSA data.
Note: The table only includes those carriers listed as having one or more
vehicles.
^aPercentages do not equal 100 because of rounding.
A Statistical Approach Would Better Identify Carriers That Pose High Crash Risks
Than Does FMCSA's Current Approach
We found that FMCSA could improve SafeStat's ability to identify carriers
that pose high crash risks if it applied a statistical approach, called a
negative binomial regression model, to the four SafeStat safety evaluation
areas instead of its current approach. Through this change, FMCSA could
more efficiently target compliance reviews to the set of carriers that
pose the greatest crash risk. Applying a negative binomial regression
model would improve the identification of high risk carriers over
SafeStat's performance by about 9 percent,^13 compared with the current
approach, which incorporates safety data weighted in accordance with the
professional judgment and experience of SafeStat's designers. Moreover,
according to our analysis, this 9 percent improvement would enable FMCSA
to identify carriers with almost twice as many crashes in the following 18
months as those carriers identified under its current approach. Targeting
these high-risk carriers would result in FMCSA giving compliance reviews
to carriers that experienced both a higher crash rate and, in conjunction
with the higher crash rate, 9,500 more crashes over an 18-month period
than those identified by the SafeStat model. Applying a negative binomial
regression model approach to the SafeStat safety evaluation areas would be
easy to implement and, in our opinion, would be consistent with other
FMCSA uses for SafeStat beyond identifying carriers that pose high risks
for crashes. In addition, adopting a negative binomial regression model
approach would be beneficial even if FMCSA makes major revisions to its
compliance and enforcement approach in the coming years under its
Comprehensive Safety Analysis 2010 initiative. Overall, other changes to
the SafeStat model that we explored, such as modifying decision rules used
in the construction of the safety evaluation areas, did not improve the
model's overall performance.
^13The 9 percent improvement is in the crash rate per 1,000 vehicles over
an 18-month period.
Regression Models Identify Carriers That Pose High Crash Risks Better Than
Expert Judgment
Although SafeStat is nearly twice as effective as (83 percent better than)
random selection in identifying carriers that pose high crash risks^14
and, therefore, has value for improving safety, we found that FMCSA could
improve SafeStat's ability to identify such carriers by about 9 percent if
it applied a negative binomial regression model approach to its analysis
of motor carrier safety data. The use of a regression model does not
entail assigning the letter categories currently assigned by the SafeStat
model. Rather, the model predicts carriers' crash risks, sorts the
carriers according to their risk level, and assigns a high priority for a
compliance review to the highest risk carriers. The improvement in
identification of high-risk carriers, which we observed with the negative
binomial regression model, is consistent with results obtained in an
earlier analysis of MCMIS data performed by a team of researchers at Oak
Ridge National Laboratory.^15
To compare the effectiveness of regression models and SafeStat in
identifying carriers that pose high crash risks, we applied several
regression models to the four safety evaluation areas (accident, driver,
vehicle, and safety management) used by the SafeStat model. We
recalculated SafeStat's June 2004 accident safety evaluation area values
because the data FMCSA provided on the number of crashes for each carrier
differed in 2006 from the data used in the model in 2004.^16 Using our
accident safety evaluation area value and the original driver, vehicle,
and safety management safety evaluation area values from June 2004, we
selected the 4,989 carriers that our regression models identified as the
highest crash risks,^17 calculated the crash rate per 1,000 vehicles for
these carriers over the next 18 months, and compared this rate with the
crash rate per 1,000 vehicles for the 4,989 carriers identified by the
SafeStat model as posing high crash risks (categories A and B).
^14Applying the SafeStat model to June 2004 data identifies 4,989 carriers
as high risk (categories A or B). Using 10,000 randomly selected samples
of 4,989 carriers and considering the crashes that these carriers had
between June 2004 and December 2005, we found that the crash rate per
1,000 vehicles in the ensuing 18 months was 83 percent higher among the
carriers identified by the SafeStat model than among the randomly selected
carriers.
^15Ken Campbell, Rich Schmoyer, and Ho-Ling Hwang, Review of the Motor
Carrier Safety Status Measurement System (SAFESTAT), Oak Ridge National
Laboratory, Final Report, October 2004. See appendix I for a more detailed
discussion of the findings from this report.
All of the regression models that we estimated were at least as effective
as SafeStat in identifying motor carriers that posed high crash risks.
(See app. III for these results.) Of these, the negative binomial
regression approach gave the best results and proved 9 percent more
effective than SafeStat, as measured by future crashes per 1,000 vehicles.
The set of carriers in SafeStat categories A and B had a crash rate of 102
per 1,000 vehicles for the 18 months after June 2004 while the set of
high-risk carriers identified by the negative binomial regression model
had 111 crashes per 1,000 vehicles. Even though this 9 percent improvement
rate seems modest, it translates into nearly twice as many "future
crashes" identified. Specifically, the negative binomial regression model
identified carriers that had nearly twice as many crashes (from July 2004
to December 2005) as the carriers identified by SafeStat--19,580 crashes
compared with 10,076.^18
SafeStat (categories A and B) and our negative binomial regression model
identified many of the same carriers--1,924 of the 4,989 (39 percent)--as
posing high crash risks. However, our model also identified a number of
high-risk carriers that SafeStat did not identify, and vice versa. For
example, our model identified 2,244 carriers as posing high crash risks,
while SafeStat placed these carriers in category D (the accident area),
assigning them a lower priority for compliance reviews. One reason for
this difference is the decision rules that SafeStat employs. Under
SafeStat, carriers must perform worse than 75 percent of all carriers to
be considered deficient in any safety evaluation area. The regression
approach identifies the carriers with the highest crash risks regardless
of how they compare with their peers in individual areas. For example, we
identified as posing high crash risks 482 carriers that SafeStat did not
consider at all for compliance reviews because the carriers had not
performed worse than 75 percent of their peers in any of the four safety
evaluation areas.
^16This occurs because data were added, deleted, or modified as more
information became known over time. See appendix III for a more detailed
discussion.
^17The threshold could be increased or decreased to align with the
resources that FMCSA and its state partners have available to perform
compliance reviews. As discussed earlier, FMCSA and its state partners
select carriers for these reviews because they pose high crash risks and
for other reasons.
^18The carriers identified as high risk by SafeStat had a total of 98,619
vehicles while those identified by the negative binomial regression model
had 175,820 vehicles. The identification of larger sized companies on
average by the negative binomial regression model is how a 9 percent
increase in the crash rate translated into 9,500 additional crashes.
FMCSA Can Apply a Regression Model Approach in the Short Term, Even Though It Is
Planning to Overhaul SafeStat
In the short term, FMCSA could easily implement a regression model
approach for SafeStat.^19 All the information required as input for the
negative binomial regression model is already entered into SafeStat. In
addition, a standard statistical package can be used to apply the negative
binomial approach to the four SafeStat safety evaluation areas. Like
SafeStat, the negative binomial regression model would be run every month
to produce a list of motor carriers that pose high crash risks, and these
carriers would then be assigned priorities for a compliance review. As
with SafeStat, the results of the negative binomial model would change
slightly each month with the addition of new safety data to MCMIS.
In discussing the concept of adopting a negative binomial regression model
approach with FMCSA officials, they were interested in understanding how
the use of the negative binomial regression model results could be used to
identify and improve the safety of those carriers that pose the greatest
crash risks (much as the SafeStat categories of A and B do now) and how it
could employ the proposed approach for current uses beyond identifying
carriers that pose high crash risks. These uses include providing an
understandable public display to shippers, insurers, and others who are
interested in the safety of carriers; selecting carriers for roadside
inspections; and trying to gain carriers' compliance with driver and
vehicle safety rules, when these carriers may not have crashes, consistent
with agency efforts.
o Identifying and improving the safety of carriers that pose high
crash risks. The negative binomial regression model approach would
produce a rank order listing of carriers by crash risk and by the
predicted number of crashes. For compliance reviews, FMCSA could
choose those carriers with the greatest number of predicted
crashes. FMCSA would choose the number of carriers to review based
on the resources available to it, much as it currently does.
^19FMCSA can use the current safety evaluation area values in SafeStat and
the number of state-reported crashes for each carrier in the 30 preceding
months in the negative binomial regression model.
Regarding improving the safety of carriers that pose high crash
risks, FMCSA currently enrolls carriers that receive a SafeStat
category of A, B, or C in the Motor Carrier Safety Improvement
Program. This program aims to improve the safety of high-risk
carriers through (1) a repetitive cycle of identification, data
gathering, and assessment and (2) progressively harsher treatments
applied to carriers that do not improve their safety. The use of a
negative binomial regression model would not affect the structure
or workings of this program, other than to better identify
carriers that pose high crash risks. As discussed above, FMCSA
would use the regression model's results to identify the highest
risk carriers and then intervene using its existing approaches
(such as issuing warning letters, conducting follow-up compliance
reviews, or levying civil penalties) as treatment.
o Providing an understandable display to the public. FMCSA could
choose to provide a rank order listing of carriers together with
the associated number of predicted crashes or it could look for
natural breaks in the predicted number of crashes and associate a
category--such as "category A" to these carriers.
o Selecting carriers for roadside inspections. Safety rankings
from the SafeStat model are also used in FMCSA's Inspection
Selection System to prioritize carriers for roadside driver and
vehicle inspections. The negative binomial regression model
optimizes the identification of carriers by crash risk using
safety evaluation area information. The negative binomial
regression model approach that we describe in this report retains
SafeStat's basic design with four safety management areas (driver,
vehicle, accident, and safety management). Therefore, FMCSA could
use the negative binomial regression model results to identify
carriers that pose a high crash risk, the results from the driver
and vehicle safety evaluation areas, or both, to target carriers
or vehicles for roadside driver and vehicle inspections.
o Furthering agency efforts to gain compliance with driver and
vehicle safety rules for carriers that do not experience crashes
(or a sufficient number of crashes to pose a high risk for
crashes). FMCSA was interested in understanding how, if at all,
the negative binomial regression model approach would affect
gaining compliance against carriers that may routinely violate
safety rules (such as drivers' hours of service requirements), but
where these violations do not lead to crashes. As discussed above,
the negative binomial regression model approach retains SafeStat's
four safety evaluation areas. Where it differs, is that it assigns
different weights to those areas based on a statistical procedure,
rather than having the weights assigned by expert judgment. As a
result, FMCSA would still be able to identify carriers with many
driver, vehicle, and safety management violations.
Other opportunities also exist for FMCSA to improve the ability of
regression models to identify carriers that pose high crash risks.
In 2005, a FMCSA compliance review work group reported a positive
correlation between driver hours of service violations and crash
rates.^20 Because FMCSA can link violations of specific regulatory
provisions, including those limiting driver hours of service, to
the crash experience of the carriers involved, it has the
opportunity to improve the violation severity weighting used in
constructing the driver and vehicle safety evaluation areas. FMCSA
has detailed violation data from roadside inspections and can
statistically analyze these data to find other strong
relationships with carriers' crash risks. Changes made to the
safety evaluation area methodology to strengthen the association
with crash risk will improve the ability of the negative binomial
regression model to identify carriers that pose high crash risks.
FMCSA has expressed doubts in the past when analysts have proposed
switching to a regression model approach. For example, Oak Ridge
National Laboratory advocated using a regression model approach in
place of SafeStat in 2004, but FMCSA was reluctant to move away
from its expert judgment model because it believed that the
regression model approach would place undue weight on the accident
safety evaluation area in determining priorities for compliance
reviews,^21 thereby diminishing the incentive for motor carriers
to comply with the many safety regulations that feed into the
driver, vehicle, and safety management safety evaluation areas. In
FMCSA's view, carriers would be less likely to comply with these
regulations because violations in the driver, vehicle, and safety
management areas would be less likely to lead to compliance
reviews under a regression model approach that placed a heavy
emphasis on crashes. Our view is that adopting a negative binomial
regression model approach would better identify carriers that pose
high crash risks and would thus further FMCSA's primary mission of
ensuring safe operating practices among commercial interstate
motor carriers.
^20Federal Motor Carrier Safety Administration Compliance Review
Workgroup, Phase II Final Report: Proposed Operational Model for FMCSA
Compliance and Safety Programs Report, February 2005.
^21Oak Ridge National Laboratory statistically measured the weights for
the safety evaluation areas and estimated the accident safety evaluation
area should have a weight of 57 in the SafeStat model formula. This
compares with the present weight of 2 that SafeStat gives the accident
safety evaluation area. Ken Campbell, Rich Schmoyer, and Ho-Ling Hwang,
Review of the Motor Carrier Safety Status Measurement System (SAFESTAT),
Oak Ridge National Laboratory, Final Report, October 2004.
Over the longer term, FMCSA is considering a complete overhaul of
its safety fitness determinations with its Comprehensive Safety
Analysis 2010 initiative. This planned comprehensive review and
analysis of the agency's compliance and enforcement programs may
result in a new operational model for identifying drivers and
carriers that pose safety problems and for intervening to address
those problems.^22 FMCSA expects to deploy the results of this
initiative in 2010. In our opinion, given the relative ease of
adopting the regression modeling approach discussed in this
report,^23 and the immediate benefits that can be achieved, there
is no reason to wait for FMCSA to complete its initiative, even if
the initiative results in major revisions to the SafeStat model.
Modifications of SafeStat Did Not Improve Crash Identification
Besides investigating whether the use of regression models could
improve SafeStat's ability to identify carriers that pose high
crash risks, we explored whether the existing model could be
improved by changing several of its decision rules. Overall, these
changes did not enhance the model's ability to identify carriers
that pose high crash risks. As long as FMCSA continues to estimate
the safety evaluation area values with its present methodology,
the rules we investigated help make the identification of
high-risk motor carriers more efficient for both SafeStat and the
negative binomial regression model.
Because the SafeStat model is composed of many components, we
selected three decision rules for analysis. We chose these three
rules because they are important pillars of the SafeStat model's
methodology for constructing the safety evaluation areas and
because we could complete our analysis of them during the time we
had to perform our work. A fuller exploration of areas with high
potential to improve the identification of carriers that pose high
crash risks would be a long-term effort, and FMCSA plans to
address this work as part of the Comprehensive Safety Analysis
2010 initiative.
^22We expect to issue a report shortly that provides additional discussion
of FMCSA's initiative to identify and take action against carriers that
are egregious safety violators.
^23Revisions to SafeStat are exempt from notice and comment under the
Administrative Procedure Act if they relate to FMCSA's internal practices
and procedures.
o Removing comparison groups. As part of its methodology for
calculating the accident, driver, and vehicle safety evaluation
area values, SafeStat divides carriers into comparison groups. For
example, in the driver safety evaluation area, SafeStat groups
carriers by the number of moving violations they have, placing
them in one of four groups (3 to 9, 10 to 28, 29 to 94, and 95 or
more).^24 SafeStat uses the comparison groups to control for the
size of the carrier. We removed all the comparison groups in each
of the three safety evaluation areas, recalculated their values,
and compared the number of crashes in which the carriers were
involved and their crash rates, for each of the SafeStat
categories A through H, with the SafeStat results in which
comparison groups were retained.
o Removing minimum event requirements. SafeStat imposes minimum
event requirements. For example, as noted, SafeStat does not
consider a carrier's moving violations if, in the aggregate, its
drivers had fewer than three moving violations over a 30-month
period. FMCSA does not calculate a safety evaluation area value
for carriers with fewer than three events in an attempt to control
for carriers that have infrequent, rather than possibly systemic,
safety problems.^25 We removed the requirement to have a minimum
number of events (such as moving violations, crashes, and
inspections), recalculated the three safety evaluation values, and
compared the number of crashes in which the carriers were involved
and their crash rates, for each of the SafeStat categories A
through H, with the SafeStat results in which minimum event
requirements were retained.
^24SafeStat does not consider carriers with fewer than three moving
violations.
^25Carriers with one or zero state-reported crashes do not receive an
accident safety evaluation area score unless the recordable accident
indicator is available from a recent compliance review. Carriers with two
or fewer driver inspections and two or fewer moving violations do not
receive a driver safety evaluation area score unless the driver review
indicator is available from a recent compliance review. Carriers with two
or fewer vehicle inspections do not receive a vehicle safety evaluation
area score unless the vehicle review indicator is available from a recent
compliance review. In the data we reviewed, almost 2 percent of the
carriers had undergone a compliance review within the 18 months prior to
the SafeStat run on June 25, 2004.
o Removing time and severity weights. The SafeStat formula weights
more recent events and more severe events more heavily than less
recent or less severe events in the accident, driver, and vehicle
safety evaluation areas. For example, the results of vehicle
roadside inspections performed within the latest 6 months receive
three times the weight of inspections performed 2 years ago.
Similarly, crashes involving deaths or injuries receive twice as
much weight as those that resulted in property damage only. We
removed the time and severity weights for the three safety
evaluation areas, recalculated these values, and compared the
number of crashes in which the carriers were involved and their
crash rates, for each of the SafeStat categories A through H, with
the SafeStat results in which time and severity weights were
retained.
o Simultaneous changes to comparison group, event, and time
severity requirements. Finally, we simultaneously removed
comparison groups, minimum event requirements, and time and
severity weights and compared the number of crashes in which the
carriers were involved and their crash rates, for each of the
SafeStat categories A through H, with the SafeStat results in
which comparison groups, minimum event requirements, and time and
severity weights were retained.
The results of each of our individual analyses and of making all
changes simultaneously produced one of two outcomes, neither of
which was considered more desirable. Relaxing the minimum data
requirements greatly increased the number of carriers identified
as high risk without increasing the overall number of predicted
crashes over the subsequent 18 months, thus reducing the
effectiveness of the SafeStat model. Removing comparison groups
and removing time and severity weights had the effect of reducing
the future crashes per 1,000 vehicles among those carriers
identified as high risk, also reducing the effectiveness of the
SafeStat model. As a result, we are not reporting on these results
in detail. Trying to modify the decision rules used in SafeStat
did highlight the balance that FMCSA has to strike between
maximizing the identification of companies with the largest number
of crashes (usually larger carriers) and those carriers with the
greatest safety risk (which can be of any size).
Despite Quality Problems, FMCSA's Crash Data Can Be Used to Compare
Methods for Identifying Carriers That Pose High Crash Risks
The quality of crash data is a long-standing problem that
potentially hindered FMCSA's ability to accurately identify
carriers that pose high crash risks.^26 Despite the problems of
late-reported crashes and incomplete and inaccurate data on
crashes during the period we studied, we determined that the data
were of sufficient quality for our use, which was to assess how
the application of regression models might improve the ability to
identify high-risk carriers over the current approach--not to
determine absolute measures of crash risk. Our reasoning is based
on the fact that we used the same data set to compare the results
of the SafeStat model and the regression models. Limitations in
the data would apply equally to both results. FMCSA has recently
undertaken a number of efforts to improve crash data quality.
Late Reporting Had a Small Effect on SafeStat's Ability to Identify
High-Risk Carriers
FMCSA's guidance provides that states report all crashes to MCMIS
within 90 days of their occurrence. Late reporting can cause
SafeStat to miss some of the carriers that should have received a
SafeStat score. Alternatively, since SafeStat's scoring involves a
relative ranking of carriers, a carrier may receive a SafeStat
score and have to undergo a compliance review because crash data
for a higher risk carrier were reported late and not included in
the calculation.
Late reporting affected SafeStat's ability to identify all
high-risk carriers to a small degree--about 6 percent---for the
period that we studied. Late reporting of crashes by states
affected the safety rankings of more than 600 carriers, both
positively and negatively. When SafeStat analyzed the 2004 data,
which did not include the late-reported crashes, it identified
4,989 motor carriers as highest risk, meaning they received a
category A or B ranking. With the addition of late-reported
crashes, 481 carriers moved into the highest risk category, and
182 carriers dropped out of the highest risk category, resulting
in a net increase of 299 carriers (6 percent) in the highest risk
category. After the late-reported crashes were added, 481 carriers
that originally received a category C, D, E, F, or G SafeStat
rating received an A or B rating. These carriers would not
originally have been given a high priority for a compliance review
because the SafeStat calculation did not take into account all of
their crashes. On the other hand, a small number of carriers would
have received a lower priority if the late-reported crashes had
been included in their score. Specifically, 182 carriers - or
fewer than 4 percent of those ranked, fell from the A or B
category into the C, D, E, F, or G category once the late-reported
crashes were included.^27 These carriers would not have been
considered high priority for a compliance review if all crashes
had been reported on time. This does not have a big effect on the
overall motor carrier population, however, as only 4 percent of
carriers originally identified as highest risk were negatively
affected by late reporting.
^26For another assessment of data quality, see Office of Inspector
General, Improvements Needed in the Motor Carrier Safety Status
Measurement System, U.S. Department of Transportation, Report MH-2004-034,
2004.
The timeliness of crash reporting has shown steady and marked
improvement. The median number of days it took states to report
crashes to MCMIS dropped from 225 days in calendar year 2001 to 57
days in 2005 (the latest data available at the time of our
analysis).^28 In addition, the percentage of crashes reported by
states within 90 days of occurrence has jumped from 32 percent in
fiscal year 2000 to 89 percent in fiscal year 2006. (See fig. 2.)
^27These 182 carriers were no longer in the worst 25 percent for the
accident safety evaluation area after the addition of the late-reported
crashes.
^28Part of the improvement in timeliness of reporting for the most recent
year is that an unknown number of crashes that occurred in 2005 had still
not been reported as of June 2006, the date we obtained these data.
Figure 2: Percentage of Crashes Submitted to MCMIS within 90 Days
of Occurrence
Incomplete Data from States Potentially Limit SafeStat's
Identification of All Carriers That Pose High Crash Risks
FMCSA uses a motor carrier identification number, which is unique
to each carrier, as the primary means of linking inspections,
crashes, and compliance reviews to motor carriers. Approximately
184,000 (76 percent) of the 244,000 crashes reported to MCMIS
between December 2001 and June 2004 involved interstate carriers.
Of these 184,000 crashes, nearly 24,000 (13 percent) were missing
this identification number. As a result, FMCSA could not match
these crashes to motor carriers or use them in SafeStat. In
addition, the carrier identification number could not be matched
to a number listed in MCMIS for 15,000 (8 percent) other crashes
that involved interstate carriers. Missing data or being unable to
match data for nearly one quarter of the crashes during the period
of our review potentially has a large impact on a motor carrier's
SafeStat score because SafeStat treats crashes as the most
important source of information for assessing motor carrier crash
risk. Theoretically, information exists to match crash records to
motor carriers by other means, but such matching would require too
much manual work to be practicable.
We were not able to quantify the actual effect of either the
missing data or the data that could not be matched for MCMIS
overall. To do so would have required us to gather crash records
at the state level--an effort that was impractical. For the same
reason, we cannot quantify the effects of FMCSA's efforts to
improve the completeness of the data (discussed later). However, a
series of reports by the University of Michigan Transportation
Research Institute sheds some light on the completeness of the
data submitted to MCMIS by the states.^29 One of the goals of the
research was to determine the states' crash reporting rates.
Reporting rates varied greatly among the 14 states studied,
ranging from 9 percent in New Mexico in 2003 to 87 percent in
Nebraska in 2005. It is not possible to draw wide-scale
conclusions about whether state reporting rates are improving over
time because only two of the states--Missouri and Ohio---were
studied in multiple years. However, in these two states, the
reporting rate did improve. Missouri experienced a large
improvement in its reporting rate, with 61 percent of eligible
crashes reported in 2001, and 83 percent reported in 2005. Ohio's
improvement was more modest, increasing from 39 percent in 2000 to
43 percent in 2005.
The University of Michigan Transportation Research Institute's
reports also identified a number of factors that may affect
states' reporting rates. One of the main factors affecting
reporting rates is the reporting officer's understanding of crash
reporting requirements. The studies note that reporting rates are
generally lower for less serious crashes and for crashes involving
smaller vehicles, which may indicate that there is some confusion
about which crashes are reportable. Some states, such as Missouri,
aid the officer by explicitly listing reporting criteria on the
police accident reporting form, while other states, such as
Washington, leave it up to the officer to complete certain
sections of the form if the crash is reportable, but the form
includes no guidance on reportable crashes. Yet other states, such
as North Carolina and Illinois, have taken this task out of
officers' hands and include all reporting elements on the police
accident reporting form. Reportable crashes are then selected
centrally by the state, and the required data are transmitted to
MCMIS.
^29The University of Michigan Transportation Research Institute's reports
on state crash reporting can be found at http://www.umtri.umich.edu. State
reports issued by the University of Michigan Transportation Research
Institute cover California, Florida, Illinois, Iowa, Louisiana, Maryland,
Michigan, Missouri, Nebraska, New Jersey, New Mexico, North Carolina,
Ohio, and Washington. We included all of these reports in our review.
Inaccurate Data Potentially Limit SafeStat's Ability to Identify
Carriers That Pose High Crash Risks
Inaccurate data, such as reporting a nonqualifying crash to FMCSA,
potentially has a large impact on a motor carrier's SafeStat score
because SafeStat treats crashes as the most important source of
information for assessing motor carrier crash risk. For the same
reasons as discussed in the preceding section, we were neither
able to quantify these effects nor determine how data accuracy has
improved for MCMIS overall.
The University of Michigan Transportation Research Institute's
reports on crash reporting show that, among the 14 states studied,
incorrect reporting of crash data is widespread. In recent
reports, the researchers found that, in 2005, Ohio incorrectly
reported 1,094 (22 percent) of the 5,037 cases, and Louisiana
incorrectly reported 137 (5 percent) of the 2,699 cases. In Ohio,
most of the incorrectly reported crashes did not qualify because
they did not meet the crash severity threshold. In contrast, most
of the incorrectly reported crashes in Louisiana did not qualify
because they did not involve vehicles eligible for reporting.
Other states studied by the institute had similar problems with
reporting crashes that did not meet the criteria for reporting to
MCMIS. These additional crashes could cause some carriers to
exceed the minimum number of crashes required to receive a
SafeStat rating and result in SafeStat's mistakenly identifying
carriers as posing high crash risks. Because each report focuses
on reporting in one state in a particular year, it is not possible
to identify the number of cases that have been incorrectly
reported nationwide and, therefore, it is not possible to
determine the impact of inaccurate reporting on SafeStat's
calculations.
As noted in the University of Michigan Transportation Research
Institute's reports, states may be unintentionally submitting
incorrect data to MCMIS because of difficulties in determining
whether a crash meets the reporting criteria. For example, in
Missouri, pickups are systematically excluded from MCMIS crash
reporting, which may cause the state to miss reportable crashes.
However, some pickups may have vehicle weights above the reporting
threshold, making crashes involving them eligible for reporting.
There is no way for the state to determine which crashes involving
pickups qualify for reporting without examining the
characteristics of each vehicle. In this case, the number of
omissions is likely to be relatively small, but this example
demonstrates the difficulty states may face when identifying
reportable crashes.
In addition, in some states, the information contained in the
police accident report may not be sufficient for the state to
determine if a crash meets the accident severity threshold. It is
generally straightforward to determine whether a fatality occurred
as a result of a crash, but it may be difficult to determine
whether an injured person was transported for medical attention or
a vehicle was towed because of disabling damage. In some states,
such as Illinois and New Jersey, an officer can indicate on the
form if a vehicle was towed by checking a box, but there is no way
to identify whether the reason for towing was disabling damage. It
is likely that such uncertainty results in overreporting because
some vehicles may be towed for other reasons.
FMCSA Has Undertaken Efforts to Improve Crash Data Quality
FMCSA has taken steps to try and improve the quality of crash data
reporting. As we noted in November 2005, FMCSA has undertaken two
major efforts to help states improve the quality of crash data.^30
One program, the Safety Data Improvement Program, has provided
funding to states to implement or expand activities designed to
improve the completeness, timeliness, accuracy, and consistency of
their data. FMCSA has also used a data quality rating system to
rate and display ratings for states' crash and inspection data
quality. Due to its public nature, this map serves as an incentive
for states to make improvements in their data quality.
To further improve these programs, FMCSA has made additional
grants available to states and implemented our recommendations to
(1) establish specific guidelines for assessing states' requests
for funding to support data improvement in order to better assess
and prioritize the requests and (2) increase the usefulness of its
state data quality map as a tool for monitoring and measuring
commercial motor vehicle crash data by ensuring that the map
adequately reflects the condition of the states' commercial motor
vehicle crash data.
In February 2004, FMCSA implemented Data Q's, an online system
that allows for challenging and correcting erroneous crash or
inspection data. Users of this system include motor carriers, the
general public, state officials, and FMCSA. In addition, in
response to a recent recommendation by the Department of
Transportation Inspector General, FMCSA is planning to conduct a
number of evaluations of the effectiveness of a training course on
crash data collection that it will be providing to states by
September 2008.
^30GAO, Highway Safety: Further Opportunities Exist to Improve Data on
Crashes Involving Commercial Motor Vehicles, [36]GAO-06-102 (Washington,
D.C.: Nov. 18, 2005).
While the quality of crash reporting is sufficient for use in
identifying motor carriers that pose high crash risks and has
started to improve, commercial motor vehicle crash data continue
to have some problems with timeliness, completeness, and accuracy.
These problems have been well-documented in several studies, and
FMCSA is taking steps to address the problems through studies of
each state's crash reporting system and grants to states to fund
improvements. As a result, we are not making any recommendations
in this area.
Conclusion
Interstate commerce involving large trucks and buses has been
growing substantially, and this growth is expected to continue.
While the number of fatalities per million vehicle miles traveled
has generally decreased over the last 30 years, the fatality rate
has leveled off and remained fairly steady since the mid-1990s.
FMCSA could more effectively address fatalities due to crashes
involving a commercial motor vehicle if it better targeted
compliance reviews to those carriers that pose the greatest crash
risks. Using a negative binomial regression model would further
FMCSA's mission of reducing crashes through the more effective
targeting of compliance reviews to the set of carriers that pose
the greatest crash risks. In light of possible changes to FMCSA's
safety fitness determinations resulting from its Comprehensive
Safety Analysis 2010 initiative, we are not suggesting that FMCSA
undertake a complete and thorough investigation of SafeStat.
Rather, we are advocating that FMCSA apply a statistical approach
that employs the negative binomial regression model rather than
relying on the current SafeStat formula that was determined
through expert judgment. In our view, the substitution of a
statistically based approach would likely yield a markedly better
ability to identify carriers that pose high crash risks with
relatively little time or effort on FMCSA's part.
Recommendation for Executive Action
We recommend that the Secretary of Transportation direct the
Administrator of FMCSA to apply a negative binomial regression
model, such as the one discussed in this report, to enhance the
current SafeStat methodology.
Agency Comments and Our Evaluation
We provided a draft of this report to the Department of
Transportation for its review and comment. In response,
departmental officials, including FMCSA's Director of the Office
of Enforcement and Compliance and Director of the Office of
Research and Analysis, noted that our report provided useful
insights and offered a potential avenue for further improving the
effectiveness of FMCSA's efforts to reduce crashes involving motor
carriers. The agency indicated that it is already working to
improve upon SafeStat as part of its Comprehensive Safety Analysis
2010 initiative. FMCSA agreed that it would be useful for it to
consider whether there are both short and longer term measures
that would incorporate the type of analysis identified in our
report, as an adjunct to the SafeStat model, in order to better
target compliance reviews so as to make the best use of FMCSA's
resources to reduce crashes.
The agency expressed some concerns with the negative binomial
regression analysis, noting that its intent is to effectively
target its compliance activities based on a broader range of
factors than is considered in the negative binomial regression
analysis approach described in our draft report, which increases
reliance on past crashes as a predictor of future crashes while
apparently de-emphasizing known driver, vehicle, or safety
management compliance issues. FMCSA told us that it incorporates a
broad range of information including driver behavior, vehicle
condition, and safety management in an attempt to capture and
enable the agency to act on accident precursors in order to reduce
crashes.
FMCSA is correct in concluding that the use of the negative
binomial regression approach could tilt enforcement heavily toward
carriers that have experienced crashes and away from other aspects
of its problem areas, such as violation of vehicle safety
standards, that are intended to prevent crashes. That is because
the present SafeStat model does not statistically assign weights
to the accident, driver, vehicle, and safety management areas. In
addition, the negative binomial regression approach fully
considers information on the results of driver and vehicle
inspection data and safety management data. We used the same data
that FMCSA used, with some adjustments as new information became
available. While we found that the driver, vehicle, and safety
management evaluation area scores are correlated with the future
crash risk of a carrier, the accident evaluation area correlates
the most with future crash risk. We recognize that FMCSA selects
carriers for compliance reviews for multiple reasons, such as to
respond to complaints, and we would expect that it would retain
this flexibility if it adopted the negative binomial regression
approach.
FMCSA also indicated that greater reliance on crash data increases
emphasis on the least reliable available data set, and one that is
out of the organization's direct control--crash reporting. While
our draft report found that crash reporting has improved, and that
late reporting has not significantly impaired FMCSA's use of the
SafeStat model, FMCSA noted that the reliance on previous crashes
in the negative binomial regression analysis described in our
draft report could result in greater sensitivity to the crash data
quality issues.
As FMCSA noted in its comments, our results showed that the effect
of late-reported data was minimal. Also, as mentioned in our draft
report and in this final report, it was not practical to determine
the effect, if any, on SafeStat rankings of correcting inaccurate
data or adding incomplete data. Since June 2004, FMCSA has devoted
considerable efforts to improving the quality of the crash data it
receives from the states. States are now tracked quarterly for the
completeness, timeliness, and accuracy of their crash reporting.
As FMCSA continues its efforts to have states improve these data,
any sensitivity of results to crash data quality issues for the
negative binomial regression approach should diminish.
We are sending copies of this report to congressional committees
and subcommittees with responsibility for surface transportation
safety issues; the Secretary of Transportation; the Administrator,
FMCSA; and the Director, Office of Management and Budget. We also
will make copies available to others upon request. In addition,
this report will be available at no charge on the GAO Web site at
http://www.gao.gov.
If you have any questions about this report, please either contact
Sidney H. Schwartz at (202) 512-7387 or Susan A. Fleming at (202)
512-2834. Alternatively, they may be reached at [email protected]
or [email protected]. Contact points for our Offices of
Congressional
Relations and Public Affairs may be found on the last page of this
report. Staff who made key contributions to this report are Carl
Barden, Elizabeth Eisenstadt, Laurie Hamilton, Lisa Mirel,
Stephanie Purcell, and James Sidney H. Schwartz.
Sidney H. Schwartz
Director
Applied Research annd Methods
Susan A. Fleming
Director
Physical Infrastructure Issues
Appendix I: Results of Other Assessments of the SafeStat Model's
Ability to Identify Motor Carriers That Pose High Crash Risks
Several studies by the Volpe National Transportation Systems
Center (Volpe), the Department of Transportation's Office of
Inspector General, the Oak Ridge National Laboratory (Oak Ridge),
and others have assessed the predictive capability of the Motor
Carrier Safety Status Measurement System (SafeStat) model and the
data used by that model. In general, those studies that assessed
the predictive power of SafeStat offered suggestions to increase
that power, and those studies that assessed data quality found
weaknesses in the data that the Federal Motor Carrier Safety
Administration (FMCSA) relies upon.
Assessments of SafeStat's Predictive Capability
The studies we reviewed covered topics such as comparing SafeStat
with random selection to determine which does a better job of
selecting carriers that pose high crash risks, assessing whether
statistical approaches could improve that selection, and analyzing
whether carrier financial positions or driver convictions are
associated with crash risk.
Predictive Capability of SafeStat Compared with Random Selection
In studies of the SafeStat model published in 2004 and 1998,^1
Volpe analyzed retrospective data to determine how many crashes
the carriers in SafeStat categories A and B experienced over the
following 18 months. The 2004 study used the carrier rankings
generated by the SafeStat model on March 24, 2001. Volpe then
compared the SafeStat carrier safety ratings with state-reported
data on crashes that occurred between March 25, 2001, and
September 24, 2002, to assess the model's performance. For each
carrier, Volpe calculated a total number of crashes, weighted for
time and severity, and then estimated a crash rate per 1,000
vehicles for comparing carriers in SafeStat categories A and B
with the carriers in other SafeStat categories. The 1998 Volpe
study used a similar methodology. Each study used a constrained
subset of carriers rather than the full list contained in the
Motor Carrier Management Information System (MCMIS).^2 Both
studies found that the crash rate for the carriers in SafeStat
categories A and B was substantially higher than the other
carriers during the 18 months after the respective SafeStat run.
On the basis of this finding, Volpe concluded that the SafeStat
model worked.
^1David Madsen and Donald Wright, Volpe National Transportation Systems
Center, An Effectiveness Analysis of SafeStat (Motor Carrier Safety Status
Measurement System), Paper No. 990448, November 1998 and John A. Volpe
National Transportation Systems Center, Motor Carrier Safety Assessment
Division, SafeStat Effectiveness Study Update, March 2004.
^2Volpe included only carriers with two or more crashes and/or three or
more inspections during the preceding 30 months, and/or an enforcement
action within the past 6 years, and/or a compliance review within the
previous 18 months. This is consistent with the SafeStat minimum event
requirements.
In response to a recommendation by the Department of
Transportation's Office of Inspector General,^3 FMCSA contracted
with Oak Ridge to independently review the SafeStat model. Oak
Ridge assessed the SafeStat model's performance and used the same
data set (for March 24, 2001), provided by Volpe, that Volpe had
used in its 2004 evaluation. Perhaps not surprisingly, Oak Ridge
obtained a similar result for the weighted crash rate of carriers
in SafeStat categories A and B over the 18-month follow-up period.
As with the Volpe study, the Oak Ridge study was constrained
because it was based on a limited data set rather than the entire
MCMIS data set.
Application of Regression Models to Safety Data
While SafeStat does better than simple random selection in
identifying carriers that pose high crash risks, other methods can
also be used to achieve this outcome. Oak Ridge extended Volpe's
analysis by applying regression models to identify carriers that
pose high crash risks. Specifically, Oak Ridge applied a Poisson
regression model and a negative binomial model using the safety
evaluation area values as independent variables to a weighted
count of crashes that occurred in the 30 months before March 24,
2001. (For more information on statistical analyses, see app.
III.)
In addition, Oak Ridge applied the empirical Bayes method to the
negative binomial regression model and assessed the variability of
carrier crash counts by estimating confidence intervals. Oak Ridge
found that the negative binomial model worked well in identifying
carriers that pose high crash risks. However, the data set Oak
Ridge used did not include any carriers with one reported crash in
the 30 months before March 24, 2001. Because data included only
carriers with zero or two or more reported crashes, the
distribution of crashes was truncated.
Since the Oak Ridge regression model analysis did not cover
carriers with safety evaluation area data and one reported crash,
the findings from the study are limited in their generalizability.
However, other analyses of crashes at intersections and on road
segments have also found that the negative binomial regression
model works well.^4 In addition, our analysis using a more recent
and more comprehensive data set supports the finding that the
negative binomial regression model performs better than the
SafeStat model.
^3Office of Inspector General, Improvements Needed, 2004.
^4Ezra Hauer, Douglas Harwood, and Michael Griffith, The Empirical Bayes
Method for Estimating Safety: A Tutorial. Transportation Research Record
1784, National Academies Press, 2002, 126-131.
The studies carried out by other authors advocate the use of the
empirical Bayes method in conjunction with a negative binomial
regression model to estimate crash risk. Oak Ridge also applied
this model to identify motor carriers that pose high crash risks.
We applied this method to the 2004 SafeStat data and found that
the empirical Bayes method best identified the carriers with the
largest number of crashes in the 18 months after June 25, 2004.
However, the crash rate per 1,000 vehicles was much lower than
that for carriers in SafeStat categories A and B. We analyzed this
result further and found that although the empirical Bayes method
best identifies future crashes, it is not as effective as the
SafeStat model or the negative binomial regression model in
identifying carriers with the highest future crash rates. The
carriers identified with the empirical Bayes method were
invariably the largest carriers. This result is not especially
useful from a regulatory perspective. Companies operating a large
number of vehicles often have more crashes over a period of time
than smaller companies. However, this does not mean that the
larger company is necessarily violating more safety regulations or
is less safe than the smaller company. For this reason, we do not
advocate the use of the empirical Bayes method in conjunction with
the negative binomial regression model as long as the method used
to calculate the safety evaluation area values remains unchanged.
If changes are made in how carriers are rated for safety, this
method may in the future offer more promise than the negative
binomial regression model alone.
Relationship of Carrier Financial Data and Safety Risk
Conducted on behalf of FMCSA, a study by Corsi, Barnard, and
Gibney in 2002 examined how a carrier's financial performance data
correlate with the carrier's score on a compliance review.^5 The
authors selected those motor carriers from MCMIS in December 2000
that had complete data for the accident, driver, vehicle, and
safety management safety evaluation areas. Using these data, the
authors then matched a total of 700 carriers to company financial
statements in the annual report database of the American Trucking
Associations.^6 The authors created a binary response variable for
whether the carrier received a satisfactory or an unsatisfactory
outcome on the compliance review. The authors then assessed how
this result correlated with financial measures derived from the
company financial statements. In general, the study found that
indicators of poor financial condition correlated with an
increased safety risk.
^5Thomas Corsi, Richard Barnard, and James Gibney, Motor Carrier Industry
Profile: Linkages Between Financial and Safety Performance Among Carriers
in Major Industry Segments, Robert H. Smith School of Business at the
University of Maryland, October 2002.
^6The American Trucking Associations is a membership organization with a
mission to serve and represent the interests of the trucking industry.
Two practical considerations limit the applicability of the
findings from this study to SafeStat. First, the 700 carriers in
the study sample are not necessarily representative of the motor
carriers that FMCSA oversees. Only about 2 percent of the carriers
evaluated by the SafeStat model in June 2004 had a value for the
safety management safety evaluation area. Of these carriers, not
all had complete data for the other three safety evaluation areas.
Second, FMCSA does not receive annual financial statements from
all motor carriers.^7 For these reasons, we did not consider using
carrier financial data in our analysis of the SafeStat data.
Relationship of Commercial Driver License Convictions and Crash Risk
A series of studies by Lantz and others examined the effect of
incorporating conviction data from the state-run commercial driver
license data system into the calculation of a driver conviction
measure.^8 The studies found that the driver conviction measure is
weakly correlated with the crash per vehicle rate.^9 However, the
studies did not incorporate the proposed driver conviction measure
into one of the existing safety evaluation areas and use the
updated measure to estimate new SafeStat scores for carriers.
While the use of commercial driver license conviction data may
have potential for future incorporation into a model for
identifying carriers that pose high crash risks, there is no
assessment of its impact at this time.
^7The Annual Report Form M is required only for class 1 or class 2
carriers that have revenue exceeding $3 million for 3 consecutive years.
^8Brenda Lantz and David Goettee, An Analysis of Commercial Vehicle Driver
Traffic Conviction Data to Identify Higher Safety Risk Motor Carriers,
Upper Great Plains Transportation Institute and FMCSA, 2004. Brenda Lantz,
Development and Implementation of a Driver Safety History Indicator into
the Roadside Inspection Selection System, FMCSA, April 2006.
^9Correlation = 0.085. (See FMCSA, Development and Implementation of a
Driver Safety History Indicator into the Roadside Inspection Selection
System, April 2006, 14).
Impact of Data Quality on SafeStat's Predictive Capability
The 2004 Office of Inspector General report, the 2004 Oak Ridge
study, and reports by the University of Michigan Transportation
Research Institute on state crash reporting all examined the
impact of data quality on SafeStat's ability to identify carriers
that pose high crash risks. These studies looked at issues such as
late reporting and incomplete or inaccurate reporting of crash
data and found weaknesses.
Late Reporting of Crash Data
To determine whether states promptly report SafeStat data, the
Office of Inspector General conducted a two-stage statistical
sample in which it selected 10 states for review and then selected
crash and inspection reports from those states for examination. It
sampled 392 crash records and 400 inspection records from July
through December 2002. In 2 of the 10 states selected,
Pennsylvania and New Mexico, no crash records were available for
the sample period, so it selected samples from earlier periods.
The Office of Inspector General also discussed reporting issues
with state and FMCSA officials and obtained crash records from
selected motor carriers. In addition, the Office of Inspector
General used the coefficient of variation to analyze data
consistency and trends in reporting timeliness across geographic
regions.^10 Our review of the study indicates that it was based on
sound audit methodology.
The study found that, as of November 2002, states submitted crash
reports in fiscal year 2002 an average of 103 days after the crash
occurred and that states varied widely in the timeliness of their
crash data reporting. (FMCSA requires that states report crashes
no more than 90 days after they occur.) In addition, the study
found that 20 percent of the crashes that occurred in fiscal year
2002 were entered into MCMIS 6 months or more after the crash
occurred. On the basis of this information, the Office of
Inspector General concluded that the calculation of the accident
safety evaluation area value was affected by the location of the
carrier's operations but did not estimate the degree of this
effect.
^10The Office of Inspector General used MCMIS data to estimate a standard
deviation for days to report a crash and then divided the standard
deviation by the average number of days. This number was multiplied by 100
to derive the coefficient of variation. The obtained value of about 77
indicates substantial variability relative to the average number of days
to report a crash.
We also assessed the extent of late reporting. We measured how
many days, on average, it took each state to report crashes to
MCMIS in each calendar year and found that the amount of time
taken to report crashes declined from 2000 to 2005. Our findings
were similar in nature to the Office of Inspector General's
findings. However, our results are broader because they are based
on all crash data rather than a sample. In addition, since our
work is more recent, it reflects more current conditions. We both
came to the conclusion, although to varying degrees, that late
reporting of crash data by states negatively affects SafeStat's
identification of carriers that pose high crash risks.
Oak Ridge also examined the impact of late reporting. Using data
provided by Volpe, Oak Ridge looked at the difference between the
date a crash occurred and the date it was entered into MCMIS. The
researchers found that after 497 days, 90 percent of the reported
crashes were entered into MCMIS.
The Oak Ridge study also reran the SafeStat model for March 2001
with the addition of crash data from March 2003 to see how more
complete data changed SafeStat scores. The study found that the
addition of late-reported data increased the number of carriers in
the high-risk group by 18 percent. This late reporting affected
the rankings of 8 percent of all the carriers ranked by SafeStat
in March 2001. Of these affected carriers, 3 percent moved to a
lower SafeStat category, and 5 percent moved to a higher category.
Including the late-reported crash data available in March 2003 for
the period from September 1998 through March 2001 resulted in a 35
percent increase in the available crash data.
We performed the same analysis as the Oak Ridge study and obtained
similar results. We used SafeStat data from June 2004, which
include carrier safety data from December 2001 through June 2004.
Using FMCSA's master crash file from June 2006, we found that,
with the addition of late-reported crashes, 481 carriers moved
into the highest risk category, and 182 carriers dropped out of
the highest risk category resulting in a net increase of 299
carriers (6 percent) being added to the highest risk category.
The University of Michigan Transportation Research Institute
issued a series of reports examining crash reporting rates in 14
states. These reports looked at late reporting as a potential
source of low crash reporting rates but did not specifically
examine the extent of late reporting or the impact of late
reporting on SafeStat scores. The institute looked at reporting
rates in each of the states by month to determine if reporting
rates were lower in the latter part of the year because of late
reporting. It found that reporting rates were lower in the latter
part of the year in 6 of the 14 states studied. This issue was not
a focus of our efforts, so we did not conduct a similar analysis.
Incomplete and Inaccurate Reporting of Crash Data
The Office of Inspector General's study found several instances of
incomplete or inaccurate data on crashes and carriers. The study
reviewed MCMIS reporting for all states and found that 6 of them
did not report any crashes to FMCSA in the 6-month period from
July through December 2002. In addition, the study found that
MCMIS listed about 11 percent of carriers as having no vehicles,
and 15 percent as having no drivers. Finally, from a sample of
crash records, the study estimated that 13 percent of the crash
reports and 7 percent of the inspection reports in MCMIS contained
errors that would affect SafeStat results. In particular, the
study concluded that the database identified the wrong motor
carrier as having been involved in a crash or as having received a
violation in 11 percent of the erroneous records.
The University of Michigan Transportation Research Institute also
examined the accuracy of states' crash data reporting. To
determine if crashes were reported accurately, the institute
compared information contained in the individual states' police
accident reporting files with crash data reported to MCMIS. Some
states, such as Ohio, had enough information captured in the
police accident file to determine if individual crashes were
eligible for reporting, and, therefore, the institute was able to
use these data in its analyses. In other states, not enough
information was available to make a determination, and the
institute had to project results on the basis of other states'
experience. The institute also carried out a number of analyses,
such as comparing reporting rates for different reporting
jurisdictions, in an attempt to identify reporting trends in the
individual states.
The institute identified several problems with the accuracy of
states' crash reporting. All 14 states that it studied reported
ineligible crashes to MCMIS. These crashes were ineligible because
they either involved vehicles not eligible for reporting or they
did not meet the crash severity threshold. In total, the 14 states
reported nearly 5,800 ineligible crashes to MCMIS out of almost
68,000 crashes reported (9 percent). The states also failed to
report a number of eligible crashes: the 14 states studied
reported from 9 percent to 87 percent of eligible crashes.
Our review of the institute's methodology indicates that its
findings are based on sound methodology and that its analyses were
very thorough. However, its studies are limited to the 14 states
studied and to the particular year studied. (Not all studies
covered the same year.) These states' experience may or may not be
representative of the experiences of the entire country, and there
is no way to determine if the reporting for this year is
representative of the state's reporting activities over a number
of years or if the results were unique to that particular year.
The exceptions to this are the studies for Missouri, which covered
calendar years 2001 and 2005, and Ohio, which covered calendar
years 2000 and 2005.
We did not attempt to assess the extent of inaccurate reporting in
individual states, but we did find examples of inaccurate data
reporting. To analyze the completeness of reporting, we attempted
to match all crash records in the MCMIS master crash file for
crashes occurring between December 26, 2001, and June 25, 2004, to
the list of motor carriers in the MCMIS census file. We found that
Department of Transportation numbers were missing for 30 percent
of the crashes that were reported, and the number did not match a
Department of Transportation number listed in MCMIS for 8 percent
of reported crashes. We also compared the number of crashes in
MCMIS with the number in the General Estimates System produced by
the National Highway Traffic Safety Administration and found
evidence of underreporting of crashes to MCMIS.^11
^11The General Estimates System collects all types of information from all
types of crashes. It is based on a nationally representative probability
sample from the estimated 6.4 million police-reported crashes that occur
annually. While the crash eligibility definitions are not strictly
comparable, the number of crashes reported to MCMIS is below the lower
bound for the 95 percent confidence interval around the estimated total
number of crashes for large trucks in 2004.
Appendix II: Scope and Methodology
To determine whether statistical approaches could be used to
improve FMCSA's ability to identify carriers that pose high crash
risks, we tested a variety of regression models and compared their
results with results from the existing SafeStat model. The models
we tested, using MCMIS data used by SafeStat in June 2004 to
identify carriers that pose high crash risks, include the Poisson,
negative binomial, zero-inflated negative binomial, zero-inflated
Poisson, and empirical Bayes. We chose these regression models
because crash totals for a company represent count outcomes, and
these statistical models are appropriate for use with count data.
In addition, we explored logistic regression to assess the odds of
having a crash. Based on the results of the statistical models, we
ranked the predicted means (or predicted probabilities in the
logistic regression) to see which carriers would be at risk during
the 18-month period after June 2004. We selected June 2004 because
this date enabled us to examine MCMIS data on actual crashes that
occurred in the 18-month period from July 2004 through December
2005.^1 We used these data to determine the degree to which
SafeStat identified carriers that proved to pose high crash risks.
We then compared the predictive performance of the regression
models with the performance of SafeStat to determine which method
best identified carriers that pose high crash risks. Using a
series of simple random samples,^2 we also calculated the crash
rates of all carriers listed in the main SafeStat summary results
table in MCMIS for comparison with the crash rates of carriers
identified by SafeStat as high risk. We did this analysis to
determine whether the SafeStat model did a better job than random
selection of identifying motor carriers that pose high crash
risks.
In addition, we tested changes to selected portions of the
SafeStat model to see whether improvements could be made in the
identification of high-risk motor carriers. In one analysis, we
modified the calculation of the safety evaluation area values and
compared the number of high-risk motor carriers identified with
the number identified by the unmodified safety evaluation areas.
For example, we included carriers with only one crash in the
calculation of the accident safety evaluation area whereas the
unmodified SafeStat model includes only carriers with two or more
crashes. We also investigated the effect of removing the time and
severity weights from the indexes used to construct the accident,
driver, and vehicle safety evaluation areas. We then compared the
result of using the modified and unmodified safety evaluation area
values to determine if this modification improved the model's
ability to identify future crash risks.
^1We obtained crash data for this period that were reported to FMCSA
through June 2006. This allowed us to obtain data on late-reported crashes
for the July 2004 through December 2005 period.
^2We drew 10,000 simple random samples of 4,989 carriers (the number of
carriers that SafeStat identified as being at highest risk for crashes
when we recalculated it) from the list of all carriers in the MCMIS master
file used by SafeStat on June 25, 2004, and for each sample we calculated
how many crashes the selected carriers reported to MCMIS between June 26,
2004, and December 25, 2005.
To assess the extent to which the timeliness, completeness, and
accuracy of MCMIS and state-reported crash data affect SafeStat's
performance, we carried out a series of analyses with the MCMIS
crash master file and MCMIS census file, as well as surveying the
literature to assess findings on MCMIS data quality from other
studies. To assess the effect of timeliness, we first measured how
many days on average it was taking each state to report crashes to
FMCSA by year for calendar years 2000 through 2005. We also
recalculated SafeStat scores from the model's June 25, 2004, run
to include crashes that had occurred more than 90 days before that
date but had not been reported to FMCSA by that date. We compared
the number and rankings of carriers from the original SafeStat
results with those obtained by adding in data for the
late-reported crashes. In addition, we reviewed the University of
Michigan Transportation Research Institute's studies of state
crash reporting to MCMIS to identify the impact of late reporting
in individual states on MCMIS data quality.
To assess the effect of completeness, we attempted to match all
crash records in the MCMIS crash file for crashes occurring from
December 2001 through June 2004 to the list of motor carriers in
the MCMIS census file. In addition, we reviewed the University of
Michigan Transportation Research Institute's studies of state
crash reporting to MCMIS to identify the impact of incomplete
crash reporting in individual states on MCMIS data quality.
To assess the effect of accuracy, we reviewed a report by the
Office of Inspector General that tested the accuracy of electronic
data by comparing records selected in the sample with source paper
documents. In addition, we reviewed the University of Michigan
Transportation Research Institute's studies of state crash
reporting to MCMIS to identify the impact of incorrectly reported
crashes in individual states on MCMIS data quality.
While the limitations in the data adversely affect the ability of
any method to identify carriers that pose high crash risks, we
determined that the data were of sufficient quality for our use,
which was to assess how the application of regression models might
improve the ability to identify high-risk carriers over the
current approach--not to determine absolute measures of crash
risk. Our reasoning is based on the fact that we used the same
data set to compare the results of the SafeStat model and the
regression models. Limitations in the data would apply equally to
both results. Methods to identify carriers that pose high crash
risk will perform more efficiently once the known problems with
the quality of state-reported crash data are addressed.
To understand what other researchers have found about how well
SafeStat identifies motor carriers that pose high crash risks, we
identified studies through a general literature review and by
asking stakeholders and study authors to identify high-quality
studies. Studies included in our review were (1) the 2004 study of
SafeStat done by Oak Ridge National Laboratory, (2) the SafeStat
effectiveness studies done by the Department of Transportation
Office of Inspector General and Volpe Institute, (3) the
University of Michigan Transportation Research Institute's studies
of state crash reporting to FMCSA, and (4) the 2006 Department of
Transportation Office of Inspector General's audit of data for new
entrant carriers.^3 We assessed the methodology used in each study
and identified which findings are supported by rigorous analysis.
We accomplished this by relying on information presented in the
studies and, where possible, by discussing the studies with the
authors. When the studies' methodologies and analyses appeared
reasonable, we used those findings in our analysis of SafeStat. We
discussed with FMCSA and industry and safety stakeholders the
SafeStat methodology issues and data quality issues raised by
these studies. We also discussed the aptness of the respective
methodological approaches with FMCSA. Finally, we reviewed FMCSA
documentation on how SafeStat is constructed and assessments of
SafeStat conducted by FMCSA.
^3Campbell, Schmoyer, and Hwang, Review of The Motor Carrier Safety Status
Measurement System (SAFESTAT), 2004; U.S. DOT Office of Inspector General,
Improvements Needed In the Motor Carrier Safety Status Measurement System,
2004; Madsen and Wright, U.S. DOT-Volpe National Transportation Systems
Center, An Effectiveness Analysis of SafeStat, 1998; John A. Volpe
National Transportation Systems Center, SafeStat Effectiveness Study
Update, 2004. University of Michigan Transportation Research Institute
MCMIS State Reports; U.S. DOT Office of Inspector General, Significant
Improvements in Motor Carrier Safety Program Since 1999 Act But Loopholes
For Repeat Violators Need Closing, 2006.
Appendix III: Additional Results from Our Statistical Analyses
of the SafeStat Model
This appendix contains technical descriptions and other
information related to our statistical analyses.
Overview of Regression Analyses
To study how well statistical methods identify carriers that pose
high crash risks, we carried out a series of regression analyses.
The safety evaluation area values for the accident, driver,
vehicle, and safety management areas served as the independent
variables to predict crash risks.^1 We used the state-reported
crash data in MCMIS for crashes that occurred during the 30 months
preceding June 25, 2004, as the dependent variable in each model.
We used the results of the SafeStat model run from June 25, 2004,
to benchmark the performance of the regression models with the
crash records for the identified high-risk carriers over the
succeeding 18 months.
We matched the state-reported crashes that occurred from December
26, 2001, through June 25, 2004, to the carriers listed in
SafeStat.^2 We checked our match of crashes for carriers with
those carriers used by FMCSA in June 2004 and found that the
reported numbers had changed for about 10,700 carriers in the
intervening 2 years. We found this difference even though we used
only crashes that occurred from December 26, 2001, through June
25, 2004, and were reported to FMCSA before June 25, 2004. Because
of this difference in matched crashes, we recalculated the
accident safety evaluation area using our match of the crashes.
This is discussed later in more detail.
Using our recalculation of the accident safety evaluation area
values and the original driver, vehicle, and safety management
safety evaluation area values for the carriers, we fit a Poisson
regression model and a negative binomial regression model to the
crash counts. Both of these models are statistically appropriate
for use when modeling counts that are positive and integer valued.
The two models differ in their assumptions about the mean and
variance. Whereas the Poisson model assumes that the mean and the
variance are equal, the negative binomial model assumes the mean
is not equal to the variance. The crash data in MCMIS fit the
assumptions of the negative binomial distribution better than
those of the Poisson.^3
^1In addition to the safety evaluation area scores, we included indicator
variables to flag any missing safety evaluation area scores.
^2We used the carrier's Department of Transportation number recorded in
the crash record to match to the carrier's Department of Transportation
number listed in the SafeStat summary table.
^3We checked this by estimating the mean and variance of the crashes for
the population of all carriers and determined that they were significantly
different.
We also tried to estimate zero-inflated Poisson and zero-inflated
negative binomial models with the SafeStat data. These models are
appropriate when the count values include many zeros, as is the
case with the values in this data set (because many carriers do
not have crash records). However, we could not estimate the
parameters for these models with the MCMIS data. We also
considered using logistic regression to model the carrier's odds
of experiencing a crash. However, the parameter estimates of the
four safety evaluation area values could not be estimated, so we
did not use the results of this model.^4
Finally, we used the results from the negative binomial model to
assess the expected carrier crash counts using the empirical Bayes
estimate. In safety applications, the empirical Bayes method^5 is
used to increase the precision of estimates and correct for the
regression-to-mean bias.^6 In this application, the empirical
Bayes method calculates a weighted average of the rate of crashes
for a carrier from the prior 30 months with the predicted mean
number of crashes from the negative binomial regression. This
method optimizes the identification of carriers with the highest
number of future crashes. This optimization of total crashes,
however, resulted in the identification of primarily the largest
companies. The crash rate (crashes per 1,000 vehicles per 18
months) was not as high for this group as for the carriers placed
by the SafeStat model in its A and B categories.
^4The coefficients in the model could not be reliably estimated (the
maximum likelihood of the model did not converge).
^5Hauer, Harwood, Council, and Griffith, Estimating Safety by the
Empirical Bayes Method: A Tutorial, 2001.
^6In the context of crashes, we wish to "treat" the most dangerous
companies with a compliance review to make them safer. But, crashes are
distributed with a fair degree of randomness. A company selected for a
compliance review may have suffered an atypical random grouping of
accidents in the preceding months. With or without a compliance review, it
is likely that the random grouping will not exist next year, and the crash
figures will improve. Statistical methods seek to control for this
regression-to-mean bias in order to better identify the effect of a
compliance review on a company's safety.
Technical Explanation of the Negative Binomial Regression Model
This section provides the technical details for the negative
binomial regression model fit to the SafeStat data. This section
also explains how we handled incomplete safety evaluation area
data for carriers in the regression model analyses.
The basic negative binomial probability distribution function for
count data is expressed as
for . The term represents the dispersion parameter, nbgre. It is
not assumed to equal one, as in the Poisson distribution. The
represents the crash count for the motor carrier, and the
represents the observed safety evaluation areas. To formulate the
negative binomial regression model and control for differences in
exposure to events among the carriers, we can express the
functional relationship between the safety evaluation areas and
the mean number of crashes as 0=iyi0>kykx
With complete data for a motor carrier, where none of the safety
evaluation area values are equal to missing, the regression model
of interest is as follows:
This equation models the log of the expected mean number of
crashes for each motor carrier using the four safety evaluation
area values, but most commercial motor companies listed in MCMIS
do not have values for all four safety evaluation areas.^7 To
account for this, it is necessary to define four indicator
variables. Let
;
;
;
.
The indicator variables will be used as main effects in the
negative binomial regression model to indicate cases for which
information is available. The effect of the safety evaluation area
will be measured by the interaction of the indicator function with
the safety evaluation area value. This gives us the following
model specification:
log(
With this parameterization, the estimate for the mean rate of
crashes for a carrier with no safety evaluation area information
is . For a carrier with information for just the accident safety
evaluation area, the estimate for the mean number of crashes is .
Note that the effect for each safety evaluation area will include
a coefficient times the safety evaluation area value for the
carrier plus an offset to the )exp(0b)exp(2ACCbbb++intercept for
the indicator term (the coefficient for the indicator function).
^7A carrier has to have two or more reported crashes in the past 30 months
to receive an accident safety evaluation area value. A carrier has to have
three or more roadside inspections to receive a driver or vehicle safety
evaluation area value. A driver has to have had a compliance review in the
past 18 months to receive a safety management safety evaluation area
value. There are other ways a carrier can receive a value for one of these
four safety evaluation areas, refer to the description of each one
provided in the Background.
We used a similar parameterization to formulate the Poisson
regression model.
Evaluation of Regression Models' Performance
We estimated regression models using the same data FMCSA used in
its application of the SafeStat model on June 25, 2004, with one
exception for the accident safety evaluation area. For that area,
we used our own match of crashes to carriers for December 26,
2001, through June 25, 2004. The MCMIS data we received in June
2006 produced different totals in the match of crashes to carriers
for about 10,700 carriers. MCMIS data change over time because
crash data are added, deleted, or changed as more information
about these crashes is obtained. The discrepancies in matching
arose even though we used the identical time interval and counted
crashes only when the record indicated they had been reported to
FMCSA before June 25, 2004. Because of these discrepancies, it was
necessary to calculate the accident safety evaluation area values
using our match of crashes and then recalculate the SafeStat
carrier scores for June 25, 2004, using our accident safety
evaluation area values and the original driver, vehicle, and
safety management safety evaluation area values.^8 We used our
accident safety evaluation area values and the original driver,
vehicle, and safety management safety evaluation area values in
the regression model analysis.
Using the revised accident safety evaluation area values and
FMCSA's original driver, vehicle, and safety management safety
evaluation area values, the SafeStat model identified 4,989
carriers that pose high crash risks. For each regression model, we
input the safety evaluation area data for the carriers in our
analysis data set and used the regression model to calculate the
predicted mean number of crashes. We then sorted the predicted
scores and selected the 4,989 carriers with the worst predicted
values as the set of high-risk carriers identified by the
regression model. Next, we used MCMIS to determine the crash
history of these 4,989 carriers between June 26, 2004, and
December 25, 2005, and compared the aggregate crash history with
the aggregate crash history of the carriers identified by the
SafeStat model during the same period of time.
^8Our calculation of the accident safety evaluation area differed slightly
from that used by FMCSA. We did not add 1 to the severity weights for
crashes with an associated hazardous materials release due to the rarity
of this event.
The regression models do not categorize carriers by letter; the
regression models produce a predicted crash risk for each carrier.
The regression models make use of the safety evaluation area
values, but they differ from the SafeStat model in this respect.
The results show that a negative binomial regression model
estimated with the safety evaluation area values outperforms the
current SafeStat model in terms of predicting future crashes and
the future crash rate among identified carriers that pose high
crash risks. (See table 3.) That is, our negative binomial and
Poisson models show 111 and 109 crashes per 1,000 vehicles per 18
months, respectively, compared with the 102 crashes per 1,000
vehicles per 18 months estimated by the current SafeStat model.
The Poisson model is not as appropriate since the crash counts for
carriers have variability that is significantly different from the
mean number of crashes.^9 The empirical Bayes method optimizes the
selection of future crashes; however, it does so by selecting the
largest carriers. The largest carriers have a lower crash rate per
1,000 vehicles per 18 months than the carriers that pose high
crash risks identified by the SafeStat model or by the negative
binomial regression model. Since the primary use of SafeStat is to
identify and prioritize carriers for FMCSA and state compliance
reviews, the empirical Bayes method did not identify carriers with
the highest safety risk.
^9The equality of the variability in the number of crashes to the average
number of crashes is an assumption of the Poisson regression model. This
assumption does not hold for the MCMIS data we analyzed.
Table 3: Results for SafeStat Model and Regression Models
Number of crashes in
Method Crash rate 18 months Number of vehicles
SafeStat category A & 102 10,076 98,619
B
Negative binomial 111 19,580 175,820
Poisson 109 21,532 198,396
Empirical Bayes 59 56,705 965,070
Source: GAO analysis of FMCSA data.
Note: As discussed in the text, the zero inflated Poisson, the
zero inflated negative binomial, and the logistic regression
approaches did not provide useful results.
(541027)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
www.gao.gov/cgi-bin/getrpt?GAO-07-585 .
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Sidney H. Schwartz at (202)
512-7387 or [email protected], or Susan A. Fleming at (202) 512-2834 or
[email protected].
Highlights of [38]GAO-07-585 , a report to congressional requesters
June 2007
MOTOR CARRIER SAFETY
A Statistical Approach Will Better Identify Commercial Carriers That Pose
High Crash Risks Than Does the Current Federal Approach
The Federal Motor Carrier Safety Administration (FMCSA) has the primary
federal responsibility for reducing crashes involving large trucks and
buses that operate in interstate commerce. FMCSA decides which motor
carriers to review for compliance with its safety regulations primarily by
using an automated, data-driven analysis model called SafeStat. SafeStat
uses data on crashes and other data to assign carriers priorities for
compliance reviews.
GAO assessed (1) the extent to which changes to the SafeStat model could
improve its ability to identify carriers that pose high crash risks and
(2) how the quality of the data used affects SafeStat`s performance. To
carry out its work, GAO analyzed how SafeStat identified high-risk
carriers in 2004 and compared these results with crash data through 2005.
[39]What GAO Recommends
GAO is recommending that FMCSA use a negative binomial regression model to
identify carriers that pose high crash risks.
In commenting on a draft of this report, the Department of Transportation
agreed that the use of a negative binomial regression model looked
promising for selecting carriers for compliance reviews, but expressed
some reservation about the greater sensitivity of this approach to
problems with reported crash data.
While SafeStat does a better job of identifying motor carriers that pose
high crash risks than does a random selection, regression models GAO
applied do an even better job. SafeStat works about twice as well as
(about 83 percent better than) selecting carriers randomly. SafeStat is
built on a number of expert judgments rather than using statistical
approaches, such as a regression model. For example, its designers decided
to weight more recent motor carrier crashes twice as much as less recent
ones on the premise that more recent crashes were stronger indicators of
future crashes. GAO estimates that if FMCSA used a negative binomial
regression model, FMCSA could increase its ability to identify high-risk
carriers by about 9 percent over SafeStat. Carriers identified by the
negative binomial regression model as posing a high crash risk experienced
9,500 more crashes than those identified by the SafeStat model over an 18
month follow-up period. The primary use of SafeStat is to identify and
prioritize carriers for FMCSA and state compliance reviews. FMCSA measures
the ability of SafeStat to perform this role by comparing the crash rate
of carriers identified as posing a high crash risk with the crash rate of
other carriers. Using a negative binomial regression model would further
FMCSA's mission of reducing crashes through the more effective targeting
of compliance reviews to the set of carriers that pose the greatest crash
risk.
Late-reported, incomplete, and inaccurate data reported to FMCSA by states
have been a long-standing problem. However, GAO found that late reported
data had a small effect on SafeStat's ability to identify carriers that
pose high crash risks in 2004. If states had reported all crash data
within 90 days after occurrence, as required by FMCSA, a net increase of
299 carriers (or 6 percent) would have been identified as posing high
crash risks of the 4,989 that FMCSA identified. Reporting timeliness has
improved, from 32 percent of crashes reported on time in fiscal year 2000,
to 89 percent in fiscal year 2006. Regarding completeness, GAO found that
data for about 21 percent of the crashes (about 39,000 of 184,000)
exhibited problems that hampered linking crashes to motor carriers. Having
complete information on crashes is important because SafeStat treats
crashes as the most important factor for assessing motor carrier crash
risk, and crash information is also the crucial factor in the statistical
approaches that we employed. Regarding accuracy, a series of studies by
the University of Michigan Transportation Research Institute covering 14
states found incorrect reporting of crash data is widespread. GAO was not
able to quantify the effect of the incomplete or inaccurate data on
SafeStat's ability to identify carriers that pose high crash risks because
it would have required gathering crash records at the state level--an
effort that was impractical for GAO. FMCSA has acted to improve crash data
quality by completing a comprehensive plan for data quality improvement,
implementing an approach to correct inaccurate data, and providing grants
to states for improving data quality, among other things.
References
Visible links
38. http://www.gao.gov/cgi-bin/getrpt?GAO-07-585
*** End of document. ***