    Highway Safety: Further Opportunities Exist to Improve Data on
    Crashes Involving Commercial Motor Vehicles (18-NOV-05,
    GAO-06-102).

    Large trucks make up 3 percent of the nation's registered
    vehicles, but they were involved in 11 percent of all fatal
    crashes in 2003. To reduce the fatality rate, the Federal Motor
    Carrier Safety Administration (FMCSA) sets national safety goals
    and works in partnership with states to reach them. Crash data
    collected by states and submitted to FMCSA is key to these
    efforts, and to be fully useful, this data must be complete,
    timely, accurate, and collected in a consistent manner. GAO
    addressed (1) what is known about the quality of commercial motor
    vehicle crash data, and what states are doing to improve it, and
    (2) the results of FMCSA's efforts to help states make
    improvements.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-102
        ACCNO:   A41780
      TITLE:     Highway Safety: Further Opportunities Exist to Improve
    Data on Crashes Involving Commercial Motor Vehicles
         DATE:   11/18/2005
      SUBJECT:   Data collection
    	     Data integrity
    	     Federal/state relations
    	     Highway safety
    	     Motor carriers
    	     Motor vehicles
    	     Safety regulation
    	     Safety standards
    	     Traffic accidents
    	     Transportation safety
    	     FMCSA Safety Data Improvement Program

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-102

         * Report to Congressional Committees
              * November 2005
         * HIGHWAY SAFETY
              * Further Opportunities Exist to Improve Data on Crashes Involving
                Commercial Motor Vehicles
         * Contents
              * Results in Brief
              * Background
              * States Continue to Have Problems with CMV Crash Data Quality, but
                Are Pursuing a Variety of Improvement Efforts
                   * Challenges Remain in Meeting Data Quality Standards
              * Data Quality Problems Often Reflect Difficulties in Collection
                and Processing of Crash Reports
                   * Misinterpretation of Criteria and Definitions by Officers
                     Filling Out Crash Reports
                   * Crash Report Processing Issues
                   * States Are Pursuing a Variety of Efforts to Address Data
                     Quality Issues
                   * Data Analysis to Identify Problems
                   * Reduction of Crash Database Backlogs
                   * Using Electronic Systems to Expedite CMV Crash Data
                   * Improving Law Enforcement Training
              * FMCSA's Efforts Have Contributed to CMV Data Quality Improvements
              * FMCSA's SaDIP Has Supported State Efforts to Improve Data Quality
                   * SaDIP has Yielded Several Positive Results
                   * FMCSA Has Made Improvements in the Administration of SaDIP;
                     However, Management of Grant Awards Raises Concerns
              * Data Quality Map Has Spurred Improvements, but Limitations Curb
                Map's Continued Usefulness
                   * The State Safety Data Quality Map is an Incentive for States
                     to Make CMV Crash Data Improvements
                   * The Data Quality Map is Important in Addressing Data Quality
                   * The State Safety Data Quality Map Has Limitations that May
                     Affect its Effectiveness as a Tool for Monitoring and
                     Measuring CMV Crash Data Quality
              * Conclusions
              * Recommendations for Executive Action
              * Agency Comments and Our Evaluation
         * Objectives, Scope and Methodology
         * State Safety Data Quality Map Limitations
              * Completeness: Overall Completeness is Based on Fatal Crashes Only
              * Timeliness: Timeliness is Not Based on All Reported Crashes
              * Accuracy: Accuracy is Based on Only One Variable
         * FMCSA Reportable Crash, CMV, and Carrier Identification Visor Cards
         * SaDIP Grant and Cooperative Agreement Distribution by State
         * SaDIP Case Study States
              * Georgia
              * SaDIP Projects
              * Crash Data Collection/Reporting Issues
              * State/Federal Coordination Issues
              * Data Quality Map
              * Minnesota
              * SaDIP Projects
              * Crash Data Collection/Reporting Issues
              * State/Federal Coordination Issues
              * Data Quality Map
              * North Carolina
              * SaDIP Projects
              * Crash Data Collection/Reporting Issues
              * State/Federal Coordination Issues
              * Data Quality Map
              * Ohio
              * SaDIP Projects
              * Crash Data Collection/Reporting Issues
              * State/Federal Coordination Issues
              * Data Quality Map
              * Oklahoma
              * SaDIP Projects
              * Crash Collection/Reporting Issues
              * State/Federal Coordination Issues
              * Data Quality Map
              * Washington
              * SaDIP Projects
              * Crash Collection/Reporting Issues
              * State/Federal Coordination Issues
              * Data Quality Map
         * GAO Contact and Staff Acknowledgments

    Report to Congressional Committees

    November 2005

    HIGHWAY SAFETY

    Further Opportunities Exist to Improve Data on Crashes Involving
    Commercial Motor Vehicles

    Contents

    Tables

    Figures

    November 18, 2005Letter

    The Honorable Christopher S. Bond Chairman The Honorable Patty Murray
    Ranking Minority Member Subcommittee on Transportation, Treasury, the
    Judiciary, Housing and Urban Development, and Related Agencies Committee
    on Appropriations United States Senate

    The Honorable Joe Knollenberg Chairman The Honorable John W. Olver Ranking
    Minority Member Subcommittee on Transportation, Treasury, and Housing and
    Urban Development, the Judiciary, District of Columbia Committee on
    Appropriations House of Representatives

    Large trucks make up only 3 percent of the nation's registered vehicles,
    but they were involved in 11 percent of all fatal crashes in 2003, the
    last year for which complete data is available. That year, large trucks
    were involved in more than 430,000 crashes, killing approximately 5,000
    people.  1 In 1999, Congress established the Federal Motor Carrier Safety
    Administration (FMCSA) within the Department of Transportation2 and
    mandated it with reducing crashes, injuries, and fatalities involving
    large trucks and buses. Currently, FMCSA has the goal of reducing
    commercial motor vehicle crash fatalities to 1.65 fatalities per 100
    million miles of travel by 2008. As of fiscal year 2003, the commercial
    motor vehicle fatality rate was 2.3 fatalities per 100 million miles
    traveled, the lowest recorded since the Department of Transportation
    initiated tracking in 1975-but still 40 percent above the 2008 goal.

    FMCSA works in partnership with states to reach commercial motor vehicle
    safety goals. States are the gatekeepers for the collection and reporting
    of commercial motor vehicle crash information. They receive crash reports
    completed by law enforcement personnel in local jurisdictions, compile
    them, and then submit crash reports to FMCSA. At the federal level, FMCSA
    manages a database which provides data that is used in rating motor
    carriers according to various safety indicators. Based on this rating,
    motor carriers are selected for safety inspections and reviews as part of
    FMCSA's enforcement efforts. While the data collected is primarily for
    federal use, states use the information to assist overall crash safety
    efforts and in setting commercial motor vehicle safety goals for
    themselves. Because the data is used in both federal and state
    decision-making on a variety of safety-related issues, it is important
    that it adequately meets data quality standards.

    To be useful to both federal and state decision-makers, crash data must be
    complete, timely, accurate, and collected in a consistent manner. However,
    there have been concerns about the quality of the information FMCSA and
    the states use to direct their efforts. Beginning with the Motor Carrier
    Safety Improvement Act of 1999, Congress directed the Department of
    Transportation to improve the collection and analysis of data on
    commercial motor vehicle crashes. This resulted in the creation of a
    commercial motor vehicle data improvement program.3 The program was
    reauthorized in 2005.4 Since 2002, about $21 million has been awarded to
    states to improve their crash data quality, but data quality problems have
    persisted.5 In February 2004, the Department of Transportation Inspector
    General released a report discussing limitations of the commercial motor
    vehicle crash data.6

    In a Senate report accompanying the fiscal year 2005 appropriation for the
    Department of Transportation,7 Congress asked that we review FMCSA's
    program for helping states improve their commercial motor vehicle crash
    data. The report directed us to describe the benefits obtained through the
    program, identify what can be done to improve the effectiveness of the
    program, and address concerns regarding crash data raised in the February
    2004 Department of Transportation Inspector General's report. Accordingly,
    this report examines (1) what is known about the quality of commercial
    motor vehicle crash data and what states are doing to improve it, and (2)
    the results of FMCSA's efforts to facilitate the improvement of the
    quality of commercial motor vehicle crash data sent to the federal
    government.

    To describe the quality of commercial motor vehicle crash data, we
    reviewed a number of sources, including data reported by FMCSA and
    existing studies on the quality of commercial motor vehicle crash data. We
    interviewed officials from FMCSA, contractors that develop FMCSA crash
    data tools, and commercial vehicle industry researchers and public
    interest organizations to gain their perspective on commercial motor
    vehicle crash data quality. To provide information on states' efforts to
    improve commercial motor vehicle crash data, we reviewed grant
    documentation for 34 states that participated in FMCSA's Safety Data
    Improvement Program (SaDIP) as of September 2005.8 We also conducted case
    studies in six states that participated in the program.9 States were
    chosen based on a wide variety of factors including crash data quality and
    level of crash reporting. Additionally, we conducted phone interviews with
    states that did not participate, or are no longer participating, in the
    program.10 To provide results of FMCSA efforts to facilitate the
    improvement of commercial motor vehicle crash data quality, we conducted
    interviews with officials from FMCSA on the administration and management
    of the SaDIP program. We also analyzed the guidance and support FMCSA
    provided to states and assessed FMCSA's role in coordinating commercial
    motor vehicle data quality initiatives. We reviewed studies conducted by
    the University of Michigan Transportation Research Institute and
    determined that the methodologies used in assessing the quality of the
    data states submit to FMCSA were sound and that the studies provided
    sufficiently reliable results for our purposes. Through site visits, a
    review of grant applications, interviews with relevant stakeholders and
    experts, and these studies, we were able to determine shortcomings in the
    reliability of FMCSA's commercial motor vehicle crash data. However, we
    determined that the data was sufficiently reliable for the purpose of case
    study selection. Our work was conducted from February 2005 through
    November 2005 in accordance with generally accepted government auditing
    standards. See appendix I for more details regarding our objectives,
    scope, and methodology.

    Results in Brief

    Overall, commercial motor vehicle crash data does not yet meet general
    data quality standards of completeness, timeliness, accuracy, and
    consistency. For example, according to FMCSA, as of fiscal year 2004
    nearly one-third of commercial motor vehicle crashes that states are
    required to report to the federal government were not reported, and those
    that were reported were not always accurate, timely, or consistent. Data
    quality problems most often stem from errors or omissions either by law
    enforcement officers at the scene of a crash or in the processing of crash
    reports to a state level database. To address data quality problems, a
    number of states are undertaking four major types of data improvement
    activities:

    o Analyzing existing data to identify where problems are and to develop
    plans for addressing them;

    o Reducing backlogs of data that have not been entered into state-level
    databases in order to create more complete state crash files, through
    steps such as hiring contract employees;

    o Developing and implementing electronic data systems for collecting and
    processing crash information in a more timely, accurate, and consistent
    manner; and

    o Providing training, such as educating law enforcement officers on the
    definitions and criteria for commercial motor vehicle crashes, to create
    more accurate and consistent data.

    These state efforts are resulting in some progress. Based on analysis of
    FMCSA data, a great number of crashes are being reported to FMCSA.
    Overall, the total number of commercial motor vehicle crashes being
    reported to FMCSA has increased by 59 percent between fiscal year 2000 and
    fiscal year 2004, while the length of time it takes states to report these
    crashes to FMCSA has decreased as well.

    FMCSA has two main efforts to support states in improving their reporting
    of commercial motor vehicle crash information-a commercial vehicle crash
    data improvement program and a data quality rating system-and both appear
    to be beneficial. Through the data improvement program, FMCSA has provided
    nearly $21 million in discretionary grants to 34 states between 2002 and
    2005. These grants have ranged from $2,000 to $2 million and have helped
    states conduct all four data improvement activities previously described.
    The six states in our case studies generally improved their data quality,
    mainly through projects funded in whole or in part through the grant
    program. While we did not find problems with FMCSA's oversight of the
    program, we did note that FMCSA does not have formal guidelines for
    awarding funds to states. Because these grants are discretionary, and
    because more states are expected to participate in the program in the
    future, having formal guidelines and systems in place would likely assist
    in prioritizing states' requests and ensuring consistency in awarding
    funds. FMCSA's second major effort, the State Safety Data Quality map, has
    proven to be an important tool for states to use in improving their crash
    data as well. This map, created by FMCSA and the Volpe National
    Transportation Systems Center, is a color-coded display depicting the
    overall data quality for each state in one of three rating
    categories-"good" (green), "fair" (yellow), or "poor" (red). According to
    both FMCSA and state officials, the map and the underlying rating system
    serve as an incentive for states to make improvements to their crash data.
    Despite the map's utility thus far, we identified potential problems both
    in the methodology used for developing ratings and the risk of drawing
    erroneous conclusions from the map. One example of a problem with the
    current methodology is that the overall ratings combine information about
    crashes with information stemming from FMCSA's inspections of motor
    carriers. Combining ratings for both crash and inspection data quality
    tends to make it difficult to determine how states are doing specifically
    with their crash data. In addition, some states with a "good" rating in
    completeness are not reporting all crashes to FMCSA. Rating states as
    "good" when in fact they have problems may discourage states from
    continuing to devote attention and resources to areas needing improvement
    and possibly misdirect program efforts. FMCSA is aware of many of the
    limitations of the map, but has not yet developed and implemented a formal
    plan to improve it. Addressing these limitations will strengthen the data
    quality map as a tool for improving commercial motor vehicle crash data.

    To ensure that FMCSA is able to target limited funds as effectively as
    possible, we are recommending that FMCSA create specific guidelines and
    criteria for awarding commercial motor vehicle crash data improvement
    funding. We are also recommending that FMCSA develop a plan to improve the
    data quality map, including assessing the methodology for developing
    ratings, providing results in greater detail, and documenting any
    limitations associated with the map. These enhancements will provide users
    with a more useful tool to view the condition and progress made in states'
    commercial motor vehicle crash data.

    Background

    FMCSA was established as a separate administration within the U.S.
    Department of Transportation (DOT) on January 1, 2000, pursuant to the
    Motor Carrier Safety Improvement Act of 1999. FMCSA issues and enforces
    the federal motor carrier safety regulations that govern many aspects of
    specified commercial trucking and bus operations, including the interstate
    operation and maintenance of commercial motor vehicles (CMV). Regulations
    promulgated by FMCSA specify requirements that must be met by drivers of
    these vehicles. FMCSA conducts compliance reviews11 of truck and bus
    companies, and performs safety audits of new entrants12 into the industry.
    In addition, FMCSA trains inspectors to conduct safety audits, while
    states and local authorities are responsible for conducting the
    inspections and submitting the results to FMCSA.13 This partnership
    between FMCSA and the states annually results in about 3 million truck and
    bus inspections, 7,000 to 13,000 compliance reviews, and more than 19,000
    new-entrant safety audits.

    CMV crash data is key to FMCSA's efforts. CMV crash data is collected by
    local law enforcement, sent to the state, and then processed and uploaded
    by the state into FMCSA's data system. FMCSA maintains a database
    management system in each state so they can submit crash reports into
    FMCSA's central data system, the Motor Carrier Management Information
    System (MCMIS). FMCSA uses the information in its Motor Carrier Safety
    Status Measurement System, also known as SafeStat, to target carriers for
    compliance reviews to ensure that they are following safety regulations.14
    SafeStat uses a variety of data to rank carrier safety, but it places the
    heaviest weight on crash data.

    Federal and state data quality guidelines call for CMV crash data to meet
    four basic quality standards:15

    o Completeness: To support adequate decision-making for identifying
    problems and developing appropriate countermeasures, data should be
    collected for all reportable CMV crashes in the state, and data on all
    appropriate crash variables such as the carrier's identification number,
    should be submitted to FMCSA.

    o Timeliness: To make decisions about current safety problems, identify
    trends, or target carriers that pose immediate threats to highway safety,
    CMV crash data should be available for state and federal analytical
    purposes within a useful timeframe.

    o Accuracy: To adequately assess CMV crash problems and target the
    appropriate carriers for enforcement, all data within reportable CMV crash
    records should be accurate and reliable.

    o Consistency: To target carriers nationwide and to compare state results,
    CMV crash data should be collected uniformly using the same standards and
    applying the same criteria across jurisdictions.

    FMCSA has provided more specific guidelines and criteria for meeting each
    of these standards. On timeliness, for example, FMCSA calls for states to
    submit all CMV crash data to MCMIS within 90 days of when the crash
    occurs. In order to facilitate complete reporting of CMV crashes, FMCSA
    recommends data elements, such as the identity of the carrier, vehicle
    configuration, cargo body type, etc., as the minimum information to be

    collected in order to have complete information on CMV crashes.16 FMCSA
    has also created criteria to assist in consistent reporting of crash
    information. These published criteria are used for identifying reportable
    CMV crashes to be submitted to MCMIS (see fig. 1).

    Figure 1: Criteria for Selecting Truck and Bus Crashes to Report to FMCSA

    Note: Exceptions include crashes that involve: 1) a personally-owned truck
    or passenger vehicle meant for personal use only as the sole vehicle
    meeting the criteria above, or 2) a driver with a disease condition (e.g.,
    stroke, heart attack, diabetic coma or epileptic seizure) and no other
    injury or damage occurs, or 3) deliberate intent (suicide, self-inflicted
    injury, homicide, etc.), with no unintentional injury or damage. Of the
    430,000 CMV crashes that occur each year, about 150,000 meet reporting
    criteria.

    With the Motor Carrier Safety Improvement Act of 1999, Congress
    established a program to improve the collection and analysis of CMV crash
    data.17 This resulted in the creation of the Commercial Motor Vehicle
    Analysis Reporting System (CVARS). CVARS, which is now known as SaDIP was
    originally intended to be a standalone data collection system. After
    determining that a separate system would duplicate existing efforts,
    however, FMCSA decided to use SaDIP as a federal funding tool to support
    state efforts to collect and report CMV crash data.18

    Besides changes in scope, SaDIP has also changed greatly in how the
    program is administered. Since fiscal year 2001, FMCSA has received
    funding from the Congress to implement this program. As a new agency,
    FMCSA did not have the appropriate contracting infrastructure in place to
    administer the funds. Therefore it transferred funds and the
    administrative duties of the program to the National Highway Traffic
    Safety Administration (NHTSA), which awarded several grants to states in
    fiscal year 2002. NHTSA also provided some of the funding to the General
    Services Administration (GSA) to enter into cooperative agreements with
    several states to fund multi-year data improvement projects. In fiscal
    year 2003, however, FMCSA assumed responsibility for all oversight of
    SaDIP funding.

    FMCSA provides SaDIP funds to states through two different methods. States
    can receive 1-year grants to fund specific projects, or they can enter
    into multi-year cooperative agreements in order to fund multiple efforts
    that are necessary to identify and reduce data quality problems. States
    can receive both of these types of funds from FMCSA and can apply for
    grant funding multiple times. SaDIP funding is discretionary in nature,
    allowing states to request the amount of funding they need to conduct data
    improvement projects. Since 2002, SaDIP funding has been provided to 34
    states, and as of September 2005, awards ranged from $2,000 to $2 million
    totaling approximately $21 million. (See app. IV for funding distribution
    by state.)

    In February 2004, the Department of Transportation Inspector General
    issued a report on SafeStat, the system FMCSA uses to target its
    compliance reviews.19 The report identified a number of problems with
    SafeStat, much of which stemmed from the quality of the crash data being
    used in calculating carrier ratings. The State Safety Data Quality map,
    discussed later in the report, was developed in response to the
    recommendations in the 2004 report.

    States Continue to Have Problems with CMV Crash Data Quality, but Are
    Pursuing a Variety of Improvement Efforts

    When measured against generally accepted standards for completeness,
    timeliness, accuracy, and consistency, states continue to have challenges
    with the quality of their CMV crash data. Many of these challenges are
    based in the collection and processing of data at the local level, and
    often reflect broader crash data quality problems for all types of vehicle
    crashes. To address remaining limitations, and improve the completeness,
    timeliness, accuracy, and consistency of the data, states are undertaking
    four main types of efforts: analyzing data to identify problem areas,
    eliminating backlogs of data not yet entered, creating electronic systems
    to expedite data entry, and training law enforcement officers in ways to
    improve the data they submit.

    Challenges Remain in Meeting Data Quality Standards

    The completeness, timeliness, accuracy, and consistency of CMV crash data
    is currently not meeting generally accepted data quality standards. Table
    1 provides examples of some of the overall problems we (and others) have
    identified. Appendix V, which contains summaries of the six states we
    visited, provides more specific examples.

    Table 1: Examples of Problems with Commercial Vehicle Crash Data Quality

    Completeness    As of 2004, FMCSA estimates that about one-third of the
                    reportable CMV crashes are not being submitted to their data
                       system-the Motor Carrier Management Information System
                    (MCMIS).a In addition, FMCSA estimates 20 percent of nonfatal
                      crashes are not being reported. Further, studies indicate
                    that even those crashes reported to MCMIS often have missing
                                               data.b
    Timeliness   The average length of time from when a crash occurs to when
                    the crash data is uploaded to MCMIS is 99 days-9 days over
                    the required time limit. While this is not far from the goal,
                    there is substantial variation in timeliness among states.
                    For example, our analysis of state CMV crash data shows
                    timeliness ranges from 13 days to 339 days.c
    Accuracy     FMCSA assesses accuracy by determining the number and
                    percentage of interstate crashes uploaded to MCMIS without
                    enough information to determine a carrier's DOT number (known
                    as a non-match). As of fiscal year 2004, 15 percent of CMV
                    crash records in MCMIS can not be matched to a carrier's DOT
                    number.d
    Consistency  According to an analysis by FMCSA, 33 of 50 states have crash
                    reports that do not adequately follow the criteria for
                    reporting commercial motor vehicle crashes to FMCSA.e

    aMCMIS contains information on the safety fitness of commercial motor
    carriers and hazardous material shippers subject to the Federal Motor
    Carrier Safety Regulations and the Hazardous Materials Regulations. States
    upload their crash information to MCMIS via FMCSA's SafetyNet.

    bSource: FMCSA

    cSource: Analysis of MCMIS data from fiscal year 2000 through fiscal year
    2004.

    dSource: Analysis of fiscal year 2004 MCMIS data.

    eSource: Comparison of State Crash Reports with SafetyNet Selection
    Criteria and Key Data Secondary Elements Chart, FMCSA, August 12, 2005.

    Data Quality Problems Often Reflect Difficulties in Collection and
    Processing of Crash Reports

    CMV crash data quality problems often stem from issues that occur when
    data is initially collected at the scene of a crash and later processed
    through the state. We reviewed reports on crash data quality including
    individual state reviews conducted by the University of Michigan
    Transportation Research Institute (UMTRI).20 We also discussed these
    matters with state officials. We identified two key causes of poor data
    quality: (1) problems in interpreting how to fill out crash reports at the
    scene and (2) crash report processing issues ranging from competing
    priorities at the local level to complex routing of crash reports.

    Misinterpretation of Criteria and Definitions by Officers Filling Out
    Crash Reports

    According to studies and our discussions with state officials, data
    collected at the scene of a crash can be flawed because of law enforcement
    misinterpretation of reporting criteria and definitions.21
    Misinterpretation can occur for several reasons, such as infrequent
    opportunities for officers to receive training on filling out crash
    reports or unfamiliarity resulting from infrequent occurrences of CMV
    crashes in an officer's jurisdiction. Below are common problems with
    properly reporting CMV crashes:  22

    Identifying reportable crashes. While crashes that result in a fatality
    are easily identifiable as a reportable crashes, tow-away or injury
    crashes are more difficult to identify. For example, UMTRI's review of
    eight states showed that five of those states experienced problems with
    reporting crashes that did not involve a fatality. According to UMTRI,
    this is likely due to a lack of understanding of criteria for reporting
    CMV crashes to FMCSA (see fig. 1).

    Identifying reportable commercial motor vehicles. FMCSA is responsible for
    enforcing safety regulations for interstate carriers, but collects crash
    information on a variety of CMVs that meet certain criteria. According to
    UMTRI reports, states had problems identifying smaller trucks and they
    were less likely to be reported because law enforcement officials are less
    likely to recognize them as qualifying CMVs. Further, law enforcement
    officials may be confused about reporting interstate and intrastate
    carriers to FMCSA. UMTRI's review of 8 states showed that five of the
    states encountered problems in identifying appropriate vehicle type for
    reporting to FMCSA (see fig. 1).

    Crash Report Processing Issues

    Several other issues relate to the processing of crash reports. These tend
    to fall into the following three main categories:

    Competing priorities at the officer level. Data collection is complicated
    because at the scene of a crash, an officer's first priority is to ensure
    the safety of those involved; data collection is often a lesser concern.
    Only after the situation has been stabilized can the officer fill out the
    crash report. Competing priorities can make it difficult for law
    enforcement to adequately complete a crash report. According to a report
    by NHTSA, crash reporting is often a low priority when traffic safety data
    is not perceived as relevant to the work of law enforcement. This was
    confirmed in states we visited. For example, one state official with whom
    we spoke said that submitting crash data is often dependent on how much
    priority the local police chief places on data quality.

    Manual crash report forms. Typically, law enforcement officers complete
    handwritten crash reports, which are then manually submitted for data
    processing at a local or state agency. According to an FMCSA official,
    nearly all states use manual crash reporting forms to some extent.
    Completing crash reports by hand and manually processing the information
    has the potential to lead to errors in the data. Data quality may be
    further compromised by the use of a supplemental CMV crash report. States
    may have a general crash report for use in all crashes and a supplemental
    CMV crash report to be completed if a CMV is involved. For example, four
    of the six case study states we reviewed used a supplemental form to
    report a CMV accident. When data has to be captured in a separate form,
    law enforcement officers may not always complete the form, and sometimes
    it may be separated from the original crash report or even lost. UMTRI
    found in one state that use of the supplemental crash report may be
    leading to CMV crashes not being submitted to FMCSA.

    Complex processing. Even with a correctly completed crash report that is
    properly identified as a CMV crash, states may have a complicated process
    in place to route CMV crash data to FMCSA and this may contribute to
    lengthy delays. UMTRI state assessments found that processing issues in
    some states led to delays and resulted in incomplete submission of crash
    data. One official told us that all law enforcement agencies in his state
    send crash reports to the state patrol office where the report is scanned
    into electronic format. The electronic version of the report is then sent
    to the state department of transportation where additional location data
    is entered into the report. From here, the report enters the state
    database and then is periodically uploaded to FMCSA. This process can take
    significant time, especially when the original crash report is missing
    information.

    States Are Pursuing a Variety of Efforts to Address Data Quality Issues

    States are undertaking a variety of projects in order to improve the
    completeness, timeliness, accuracy, and consistency of CMV crash data.
    These projects encompass four general types of efforts: research and
    analysis, reduction of reporting backlogs, electronic processing of crash
    records, and training. Figure 2 shows state participation in each of these
    four efforts. These projects directly affect the completeness, timeliness,
    accuracy, and consistency of the states' CMV crash data.

    Figure 2: State Participation in CMV Crash Data Improvement Efforts from
    Fiscal Year 2002 through Fiscal Year 2005

    Note: Data available for the 34 states that have participated in FMCSA's
    data improvement program. Several states are participating in more than
    one effort.

    Data Analysis to Identify Problems

    Eleven of the 34 states have research projects to evaluate crash data
    collection and reporting practices.23 Assessing data collection and
    reporting processes allows states to determine how well their data meet
    each of the data quality standards of completeness, timeliness, accuracy,
    and consistency, providing them with a useful tool for determining where
    they are going to concentrate their improvement efforts.

    Out of 34 states for which we have data, 11 have conducted research to
    identify data problems.  During our case studies, we found examples of
    states conducting data analysis to identify their CMV crash data
    collection and reporting problems. (See app. V for more information on our
    case study states.) For example, one state conducted an analysis in order
    to determine why approximately half of the eligible crashes in the state
    crash file were not being reported to FMCSA. The analysis found that the
    state's crash file was not in the correct format to be correctly
    transferred to FMCSA. In addition, frequent errors in the state or zip
    code fields, and incomplete or inaccurate data, were leading to problems.
    A federally funded study of the state's crash data found approximately 50
    percent of the state's crash reports for one year (2003) were not reported
    to the federal crash file due to problems at the state level coding and
    preparing crash data for transfer to FMCSA. Another state conducted a
    traffic records assessment that found the major deficiency in the state's
    crash file was the lack of timely data. It also identified the lack of
    effective quality controls during data entry and non-reporting of eligible
    crashes as deficiencies. Following the state's assessment, researchers
    made recommendations to help the state prioritize projects, including
    planning for the eventual implementation of electronic crash data
    collection.

    Reduction of Crash Database Backlogs

    Fourteen of the 34 states have projects to reduce the backlog of crash
    reports that need to be entered into state crash files. In most states,
    until crashes are included in the state file, they are not reported to
    FMCSA. Hence, backlog reduction is essential for creating a complete state
    crash file, and with providing FMCSA with complete data about CMV crashes.
    A complete data file is critical for performing crash and trend analysis,
    for making informed policy decisions, and for gaining a better
    understanding of CMV safety issues.

    Of the 34 states for which we have data, 14 are participating in backlog
    reduction projects, including four of our six case study states. In these
    four states, temporary employees were hired to eliminate data backlogs.
    For example, in one state, contract employees were working to reduce a
    backlog of 600,000 crash reports. Data entry work began in July 2004 and
    state officials expect it to be completed by September 2005. Another state
    used temporary employees to eliminate a backlog of approximately 2 million
    crash records. This effort began in 2002, and as of June 2005, the data
    backlog has been completely eliminated and the state is now processing
    crash reports within 3 days of receipt.

    Using Electronic Systems to Expedite CMV Crash Data

    Twenty-seven of the 34 states have projects to collect or input CMV crash
    data through electronic systems. Electronic reporting allows for more
    accurate and timely transfer of crash data to state and federal crash
    files. Electronic transfer of crash data can reduce duplication and data
    entry error, because paper-based data collection systems are subject to
    human error and time delays. While these projects enhance the quality of
    CMV crash data, often they are large in scope and require law enforcement
    agencies to purchase hardware. Uniform electronic crash reporting systems
    are heavily dependent on acceptance from all stakeholders and often some
    jurisdictions may already have their own systems.

    Of the 34 states for which we have data, 27 are participating in
    electronic crash record projects, including all six of the case study
    states. For example, one state has developed software that facilitates a
    nightly transfer of CMV crash records from the Division of Driver and
    Vehicle Services to the state patrol agency. The state patrol agency then
    submits the crash reports to FMCSA. The state is also working to provide
    computer hardware and Internet access so that state patrol officers can
    complete crash reports online. Another state is developing electronic
    crash reporting capabilities using handheld computers, which will
    eliminate handwritten crash reports, and the need to manually enter crash
    report data into an electronic system.

    Improving Law Enforcement Training

    States are providing training to law enforcement officers in CMV crash
    data collection. Training for law enforcement officers on how to correctly
    identify a CMV and how to accurately complete a police accident report can
    help to improve the completeness, accuracy, and consistency of CMV crash
    data since many of the mistakes in reporting are made at the crash

    site.24 Training is also an opportunity to highlight the link between data
    collection and its end use in planning and prevention.

    Seventeen of the 34 states have projects aimed at providing law
    enforcement training initiatives, and all six of our case study states had
    some form of CMV training. In addition, all case study states we visited
    recognized problems with the quality of CMV crash data resulting from
    issues at the collection point. For example, one state provides on-going
    CMV training to teach officers to properly identify and report commercial
    vehicle crashes. This state also tries to show law enforcement officers
    how the data is used in planning and prevention in order to show the
    significance of good data. However, state officials told us that there has
    not been a lot of participation in the training from local law enforcement
    agencies. Another state is planning to develop a training video and visor
    cards25 for statewide dissemination. The visor cards will assist officers
    in CMV identification and provide information that will help police
    properly complete all the necessary crash reporting documentation.

    These state efforts are resulting in some progress. An analysis of FMCSA
    data shows that a greater number of crashes are being reported to FMCSA
    since 2000. Overall the total number of commercial motor vehicle crashes
    being reported to FMCSA has increased by 59 percent between fiscal year
    2000 and fiscal year 2004, while the length of time it takes states to
    report these crashes to FMCSA has decreased as well.

    FMCSA's Efforts Have Contributed to CMV Data Quality Improvements

    FMCSA's efforts appear to improve CMV crash data quality and have had a
    positive impact on state crash data. Specifically two efforts, the Safety
    Data Improvement Program (SaDIP) and the State Safety Data Quality map,
    have contributed to the changes. SaDIP funding has allowed states to
    improve data completeness, timeliness, accuracy, and consistency by
    supporting the implementation of new activities or increasing the scope
    and timeliness of existing projects. While FMCSA has provided adequate
    administration and oversight of SaDIP, the management of the program with
    regard to awarding grants raises some concerns. There are no formal
    guidelines in place for awarding funding to the states, and while this has
    not yet presented a problem, it may in the future. Funding continues to be
    made available, and more states continue to request funds for new
    projects. The State Safety Data Quality map is an evaluation tool that
    provides ratings for states' crash and inspection data quality, and
    displays the ratings so that a state's performance relative to other
    states is apparent. The map, which is publicized on FMCSA's Analysis and
    Information (A&I) Online Web site, allows state officials and the public
    to view the status of states'

    CMV crash and inspection data quality.26 The map is being used as an
    indicator of states' progress in improving the completeness, timeliness,
    and accuracy27 of crash data and by virtue of its public nature it is an
    incentive for states to improve their crash information. However, the
    methodology used for the ratings has limitations that may hinder the map's
    effectiveness in monitoring the status of CMV crash data quality.

    FMCSA's SaDIP Has Supported State Efforts to Improve Data Quality

    As of July 2005, FMCSA had provided about $21 million to 34 states in
    order to assist in improving the quality of CMV crash data.28 Several
    states have received funding more than once, and in fiscal year 2005,
    FMCSA made available to states almost twice the amount of money than it
    had at the beginning of the program ($3.9 million in fiscal year 2000
    versus $7.3 million in fiscal year 2005). Additionally, recently passed
    transportation legislation authorizes a total of $11 million to be used
    for SaDIP funding for

    fiscal years 2006-2009.29 The states that have participated in SaDIP
    account for about two-thirds of all CMV crashes occurring between April 1,
    2004, and March 31, 2005, reported to FMCSA's MCMIS database, and 70
    percent of the 2003 fatalities reported to MCMIS. FMCSA's goal is to
    provide funding for projects that will have the largest impact on
    improving state data describing CMV crashes. FMCSA also encourages states
    that are participating in data improvement projects that are already
    funded to apply for SaDIP funds in order to provide additional assistance
    to these larger efforts. (See table 2 for the annual distribution of SaDIP
    funds and app. IV for fund distribution by state.)

    Table 2: Annual Distribution of SaDIP funds

         Fiscal year         Funds budgeted for SaDIP     Funds awarded to states
    2000                                $3,865,000                          $0
    2001                                 2,748,000                           0
    2002                                 5,462,614                   5,207,014
    2003                                 4,967,500                     854,732
    2004                                 4,913,990                   5,449,467
    2005                                 7,340,800                   9,180,181
    Total                              $29,297,904                 $20,691,394

    Source: FMCSA.

    Note: These data are current as of September 2005. In addition to funds
    provided directly to states, FMCSA provided $1,200,000 to contractors to
    assist states in improving data quality, bringing the total amount spent
    on state data improvement efforts to $21,891,394.

    SaDIP has Yielded Several Positive Results

    States have used the SaDIP funds to conduct a variety of projects,
    including those discussed earlier in this report. These funds have
    benefited states in several different ways, including increased focus on
    CMV data quality and advancement and expansion of ongoing broader data
    quality projects.

    SaDIP increased national attention to the problems associated with the
    quality of CMV crash data reported to the federal government. Several
    state officials we interviewed stated that they have noticed an increase
    in the amount of focus given to CMV crash data issues. For example,
    presentations made by FMCSA at several national conferences and workshops
    have highlighted the importance of data quality, and informed states of
    the various types of assistance, such as training and funding, that are
    available to them. During our case studies, we were told the following:

    o One state official said that by providing funds to be used for specific
    purposes, SaDIP had the effect of focusing attention on data quality
    improvement. SaDIP has sustained a high level of interest in data quality
    and has been a catalyst to improving traffic records coordination across
    state agencies.

    o Another state official said SaDIP helped to improve communication
    between the state patrol and the state's department of vehicle services.
    Because grants were provided to both agencies, analysts at the agencies
    are working together and have a better understanding of each other's data
    needs and share access to their respective databases.

    In half of our case study states, SaDIP has also allowed states to
    expedite data improvement projects that were already planned for
    implementation.

    o One state official told us that the state had been considering a plan to
    improve traffic safety data, including CMV crash data, and SaDIP funds
    provided it with the means to implement this plan. The long-term funding
    provided by the SaDIP cooperative agreement to support a full time
    employee was also a crucial element in gaining support from state
    decision-makers.

    o One state official said the SaDIP grant made it significantly easier to
    prioritize and expedite a data project that the state was already
    considering implementing. The additional funding allowed the state to
    specifically address CMV crash data and use other funding to continue to
    address broader crash data issues.

    Finally, SaDIP has allowed states to increase the scope of ongoing
    projects or develop new initiatives. Two of our six case study states have
    used SaDIP funds in addition to other resources to develop comprehensive
    and long-term data quality initiatives to address the completeness,
    timeliness, accuracy, and consistency of the CMV crash data they report to
    FMCSA.

    o Officials in one state told us that SaDIP funding supports elements of
    the state's electronic traffic information processing initiative. SaDIP
    funding has allowed the state to resolve a considerable backlog in crash
    records, creating a more complete crash file, and work has begun to
    develop a new electronic crash report that will be used-first by the state
    police-and then by local law enforcement in the rest of the state. Without
    resolution of this backlog, the state was unable to fully implement the
    CMV data component of its electronic traffic information processing
    initiative. Doing this allowed the state to have more timely, accurate,
    and consistent crash data.

    o SaDIP funding has helped another state develop an electronic crash
    report and traffic records system. As previously stated, this will allow
    the state to have more timely, accurate, and consistent crash data. The
    state's goal is to develop the new system by 2008 and the state is
    evaluating and using multiple sources of funding to achieve this goal.
    Early in the process, the state made a strategic decision to target all
    crash data, not just commercial vehicle data. While the state uses SaDIP
    to develop electronic crash reporting software, SaDIP funding has also
    been leveraged to help the state accomplish its data quality improvement
    goals more quickly. For example, the new crash reporting capabilities
    developed through the SaDIP grant allow for much better crash analysis and
    targeting of safety efforts along major commercial vehicle corridors in
    the state.

    FMCSA Has Made Improvements in the Administration of SaDIP; However,
    Management of Grant Awards Raises Concerns

    SaDIP has evolved over time and FMCSA has made several efforts to improve
    the administration of the program. For example, beginning in fiscal year
    2006, FMCSA will be implementing a new application package for states to
    use in applying for SaDIP. While SaDIP had some preexisting application
    requirements for states, this application package provides a uniform
    application for states to use, and it is the first time states are going
    to be required to submit quantifiable project objectives and program
    measures. This will allow FMCSA to begin to measure the effectiveness of
    state improvement efforts. Additionally, SaDIP will be posted on
    Grants.gov ( http://www.grants.gov ) in 2006.30 This federal website
    provides a single source for grant applicants to electronically find and
    apply for federal funds. FMCSA will also be updating its State Program
    Manager's manual to include guidelines for the roles and responsibilities
    of state program managers in administering SaDIP. Finally, FMCSA will be
    issuing funds at designated times during the year. Currently, funds are
    awarded to states on a rolling basis, but beginning with the
    implementation of the new application package the funds will be awarded on
    specific dates to all applicants. This is expected to improve the
    program's organization and FMCSA's ability to keep track of grant progress
    as the number of program participants increases. Throughout these
    administrative changes, FMCSA has maintained sufficient oversight of the
    states participating in SaDIP. FMCSA has contracted with a company that is
    responsible for monitoring SaDIP participant states, and ensuring that
    they are submitting quarterly progress reports containing sufficient
    detail to FMCSA. This contractor also has regular conversations with the
    FMCSA Division Administrators in the SaDIP states, and it maintains copies
    of SaDIP-related paperwork.

    While these efforts are positive steps, we have concerns with FMCSA's lack
    of guidelines for awarding funds to states. FMCSA has not yet established
    formal guidelines for determining how much money a state should receive,
    or if the state should receive the funds in the form of a grant or a
    cooperative agreement. Since the beginning of the program, funds awarded
    to states have ranged from $2,000 to $2 million for projects ranging from
    specific activities to broader efforts that span three or more years. (See
    app. IV for funds awarded by state.) These awards have taken the form of
    both grants and cooperative agreements between the state and the federal
    government. Currently these decisions are made on a state-by-state basis,
    informally by a small review panel. The state's application, data quality
    history, discussions with the state's FMCSA division administrator, and
    any other relevant and available data on the state are consulted when
    making funding awards.

    While the absence of guidelines has not proven to be problematic to date,
    having formal guidelines will better ensure consistency and discipline in
    managing the grant program among states, particularly as states' needs
    become more targeted or more states decide to participate in the program.
    It will also add integrity to the grant management system and assist in
    providing better administration and oversight of SaDIP projects. For
    example, such guidelines would likely allow FMCSA to better assess and
    prioritize states' funding requests, including more formally considering
    whether proposed activities adequately address problems states identify in
    the proposal, whether the amount of funding requested is appropriate for
    the proposed activities, and whether multiple-year versus one-year funding
    is appropriate. Further, formal guidelines would provide a more structured
    framework to evaluate the effectiveness of different SaDIP project
    activities and assist in guiding future state improvement efforts.

    Data Quality Map Has Spurred Improvements, but Limitations Curb Map's
    Continued Usefulness

    The State Safety Data Quality map has encouraged states to improve their
    CMV data quality, but limitations exist that may hinder the map's
    usefulness as a tool for monitoring and measuring CMV crash data quality.
    The State Safety Data Quality map was created by FMCSA and the Volpe
    National Transportation Systems Center principally to provide context for
    both crash and inspection data used in the SafeStat system; however, it
    has

    evolved into a tool to evaluate state-reported crash and inspection
    data.31 It is based on a system that rates crash and inspection data
    quality-completeness, timeliness, and accuracy-as "good," "fair," or
    "poor."32 The map has proven to be a major incentive for states to
    initiate CMV crash data improvements, for gaining support in implementing
    these improvements, and as a tool to monitor CMV crash data quality.
    However, we have identified some important limitations that can affect the
    data quality map's future usefulness.

    The State Safety Data Quality Map is an Incentive for States to Make CMV
    Crash Data Improvements

    Since the State Safety Data Quality map is accessible by the public and
    presents data quality ratings in a simplified form, the map is a motivator
    for states to improve their CMV crash data. According to officials at
    FMCSA, they said the map has been very influential in encouraging states
    to improve their CMV crash data. Corroboratory comments came from the
    state level, where many of the officials in case study states provided
    anecdotes of how the data quality map served as an impetus for initiating
    improvements and gaining support.33 Here are examples of how the map
    assisted states in initiating or expediting improvements in CMV crash
    information.

    o One state includes the data quality map ratings in weekly status reports
    to agency heads and in reports to the governor. An official in this state
    said that once the data quality map was shown to the governor, it raised
    the importance of improving CMV crash data. This state now posts the data
    quality map on state agency Web sites and uses it as a tool to compare its
    data quality efforts to the rest of the nation.

    o One state official told us that when agency leaders understand that
    their state has a "poor" rating, they are likely to make data improvement
    a priority and focus attention and resources on the issue.

    o Another official said that the map helped officials to "see the light."
    In this state, the data quality map helped initiate a process that led to
    improved communication and coordination for data quality at the highest
    levels for all safety and security projects within the state.

    The Data Quality Map is Important in Addressing Data Quality

    FMCSA and states we reviewed use the State Safety Data Quality map to
    measure states' CMV data quality and progress being made in their data
    improvement efforts. Our review indicated that although the map was
    created as a tool for providing context to FMCSA's SafeStat system, both
    FMCSA and the states were using it specifically for monitoring the status
    of CMV crash data quality. State officials were also using it to identify
    data quality problems and assist states in targeting state improvement
    efforts. Here are examples from the states we contacted:

    o In one state an official reported that it was very helpful for law
    enforcement officials and department heads to instantly recognize their
    state's data quality status on the map.

    o In another state, officials said they knew they had a data quality
    problem, but were unable to identify the specific issue. The data quality
    map indicated that the state's accuracy was poor, and the problem was
    specifically with matching carriers in the state database with crashes in
    MCMIS.

    o In another state, officials said the data quality map focused attention
    on the state's data quality spending and results, and it helped state
    officials re-prioritize spending.

    Both FMCSA and state officials in many of the states we spoke with
    recognized the map as an important tool in measuring progress in their
    crash data improvement efforts. Further, FMCSA officials said the map was
    used as an assessment of a state's success in the SaDIP program. While
    officials also said they reviewed data in the MCMIS system to monitor
    state progress, an important measure of success in the program was a
    state's status in the State Safety Data Quality map and the underlying CMV
    crash measures.

    The State Safety Data Quality Map Has Limitations that May Affect its
    Effectiveness as a Tool for Monitoring and Measuring CMV Crash Data
    Quality

    Given the importance that FMCSA and the states attach to the data quality
    map, it appears to be a good first step in monitoring states' data
    quality. However, we found limitations with many features of the map that
    diminish its usefulness as an effective tool for specifically monitoring
    and measuring state progress in their improvement efforts. Specifically,
    we identified several limitations with the methodology used to develop the
    data quality map's ratings, as well as with the measures themselves that
    should be addressed. Below are some of the key limitations. A more
    detailed discussion of limitations is located in appendix II.

    o The data quality map measures used to calculate the completeness,
    timeliness, and accuracy of CMV crash data falls short of providing a
    complete measure of CMV crash data quality. While each data quality
    measure has some limitations, one key measure-completeness-has a number of
    difficulties. Most importantly, the completeness measure is limited in the
    data it is assessing. The completeness measure only evaluates fatal CMV
    crashes, which represents about 3 percent of all reportable CMV crashes.34
    In addition, the completeness measure does not assess the completeness of
    the information contained within the crash report.

    The completeness measure's methodology relies solely on comparing CMV
    fatal crash data in MCMIS to data states submit to the Fatality Analysis

    Reporting System (FARS).35 This particular approach provides additional
    limitations to the completeness measure. First, there are some
    definitional differences between FARS and MCMIS data that can account for
    about 4 percent of the crash records. Secondly, FARS data is not timely.
    For example, the June 2005 map relies on 2003 FARS data; thus the
    completeness measure does not reflect the current status of CMV crash
    data. In addition, since FARS data is only released once a year, quarterly
    issuances of the map do not necessarily reflect changes in the number of
    fatal crashes. Table 3 below describes in more detail the limitations for
    each crash data quality measure.

    Table 3: Comparison of Data Quality Standards and State Safety Data
    Quality Map Measures

    Data quality  State Safety                      Limitations
         standard    Data Quality
                      Map measure
    Completeness All reportable  Percentage of                  Includes only
                    CMV crashes in  Fatal Crash                    reported,
                    the state and   Records Reported               fatal crashes;
                    data on all                                    reflects only
                    appropriate     Compares the                   about 3
                    crash variables number of large                percent of all
                    are submitted   trucks in crashes              CMV crashes
                    to FMCSA.       involving a                    required to be
                                    fatality in MCMIS              reported to
                    FMCSA           versus those in                FMCSA.
                    recommends 20   the FARS.b
                    CMV crash data                                 Bases
                    variables that                                 completeness
                    should be                                      on FARS data,
                    reported on.a                                  though some
                                                                   differences
                                                                   exist in state
                                                                   definitions of
                                                                   a CMV fatal
                                                                   crash and may
                                                                   result in an
                                                                   inflated or
                                                                   deflated
                                                                   rating.

                                                                   Does not
                                                                   represent
                                                                   current status
                                                                   of
                                                                   completeness.

                                                                   Does not
                                                                   measure the
                                                                   completeness
                                                                   of CMV
                                                                   information
                                                                   within a crash
                                                                   report
                                                                   (missing
                                                                   variables).
    Timeliness   All reportable  Percentage of                  Only reflects
                    CMV crash       Crash Records                  records that
                    records are     Reported within 90             are uploaded
                    available for   Days                           into MCMIS and
                    state and                                      have not been
                    federal         The percentage of              changed or
                    analytical      State-reported                 edited.d
                    purposes in a   fatal and
                    useful          non-fatal crash                Backlogs of
                    timeframe.      records reported               crash reports,
                                    within 90 days in              once entered,
                    FMCSA           the MCMIS database             can negatively
                    recommends that for carriers over              affect
                    CMV crash data  a 12-month time                timeliness
                    be reported     period.c                       rating.
                    within 90 days
                    of the crash
                    occurrence.
    Accuracy     All data within Percentage of                  Only measures
                    reportable CMV  Matched Crash                  accuracy of
                    crash records   Records                        one variable
                    are accurate                                   (identity of
                    and reliable.   The percentage of              the motor
                                    State-reported                 carrier-the
                                    fatal and                      U.S. DOT
                                    non-fatal crash                number);
                                    records in the                 neglects to
                                    MCMIS database for             measure the
                                    interstate                     accuracy for
                                    carriers and                   other
                                    intrastate                     recommended
                                    hazardous material             variables.
                                    carriers over a
                                    12-month time
                                    period that were
                                    matched to a motor
                                    carrier in MCMIS.
    Consistency  Crash data      None                           N/A
                    should be
                    collected
                    uniformly.
                    Officials
                    should use the
                    same standards
                    and apply the
                    same criteria
                    uniformly
                    across
                    jurisdictions.

                    FMCSA provides
                    guidelines and
                    criteria for
                    reporting CMV
                    crashes.

    Source: FMCSA and Volpe National Transportation Systems Center. Data
    Quality Standards are based on GAO's review of data quality guidelines
    from a variety of sources including FMCSA, NHTSA, and Data Nexus Inc., and
    includes references to the Model Minimum Uniform Crash Criteria, National
    Governors' Association Elements and Criteria, and the American National
    Standard Manual for Classification of Motor Vehicle Traffic Records.

    aIn 1992, FMCSA adopted the National Governors' Association (NGA)
    recommended data elements, requiring these data to be collected and
    reported on motor carrier crashes.

    bFMCSA created an "Crash Consistency Overriding Indicator" to indicate
    consistency in reporting of non-fatal CMV crashes. The indicator flags
    states that may be experiencing major problems in reporting crash data.
    The Crash Consistency Overriding Indicator is the percentage of
    state-reported non-fatal crashes as compared to a three-year average of
    reported non-fatal crashes. States that have reported fewer than 50
    percent of non-fatal crash records for the current year based on the
    previous three-year average of non-fatal crash records are flagged and
    receive a rating of "poor" regardless of their ratings in any of the other
    data quality indicators. This indicator is also limited because it only
    identifies extreme cases of under-reporting and does not assess if there
    is a substantial increase in reported non-fatal crash records.

    c Includes both interstate carriers (those carriers that operate between
    states) and intrastate carriers (carriers that operate only within one
    state) though FMCSA is only responsible for regulating interstate
    carriers.

    dFMCSA's MCMIS currently does not have the ability to track the original
    upload date when a crash record is edited or changed. Therefore, records
    that are changed after the original upload are not used in the calculation
    for timeliness. For additional information see appendix II.

    o The data quality map is limited in its ability to meaningfully monitor
    and track CMV crash data quality over time. Since the State Safety Data
    Quality map was first issued, a large majority of states have been rated
    as "good" or "fair" for the completeness, timeliness, and accuracy
    measures. Since the first issuance of the map, over 90 percent of the
    states have been rated "good" or "fair" for completeness, and about three
    quarters of the states have been rated "good" or "fair" for crash
    timeliness and crash accuracy. However, our review shows that states need
    to continue to make improvements. In addition, since the majority of
    states are rated as "good" or "fair" in measures, it is also difficult to
    measure any progress made because many states have already reached the
    highest rating. For example, as of the June 2005 data quality map, 41
    states were rated "good" in crash completeness. Because these states have
    already reached the highest rating for this measure, it may prevent
    measuring any subsequent progress in this data quality standard.

    There are other problems with using the data quality map to track trends.
    Based on our review, a state's rating can temporarily decline for a
    variety of reasons-even for implementing improvements. Hence, for that
    period of time, the data quality map is not accurately reflecting a
    state's true data quality status. Officials in one state told us that the
    state was implementing a new electronic system which in the long term
    would improve its data quality greatly. However, current CMV crash
    reporting was slowed during implementation, and the state's rating went
    down from "fair" to "poor."

    o The data quality map's ratings for overall data quality combine data
    from crashes with data from FMCSA's inspections, making it difficult for
    map users to obtain an overall picture based solely on crash data. For the
    overall data quality rating, individual crash measures for completeness,
    timeliness, and accuracy are combined with inspection measures for
    timeliness and accuracy. For each of the five individual crash and
    inspection measures a state will receive a rating of "good," "fair," or
    "poor."36 A state's overall rating depends on how well the state did
    across all five measures (see table 4).

    Table 4: Overall Data Quality Rating

    Rating             Thresholds for overall data quality rating
    Good   No "poor" and a minimum of one "good" in a crash or inspection
              measure
    Fair   Maximum of one "poor" in a crash or inspection measure
    Poor   Two or more "poor" ratings in crash or inspection measures or a red
              flag in the Crash Consistency Overriding Indicator

    Source: FMCSA.

    Currently, users can only view CMV crash data quality by individual
    measures (completeness, timeliness, and accuracy). Separating the
    inspection data and presenting a specific overall CMV crash data quality
    rating (a combined rating composed of completeness, timeliness, and
    accuracy for crash data only) would enhance a state's ability to
    understand its crash data status and to monitor progress in improving the
    information.

    o While the State Safety Data Quality map provides a description of the
    methodology used, it does not identify limitations to the methodology.
    While many of these limitations to the map are acknowledged by FMCSA, they
    are not publicly displayed on the State Safety Data Quality map Web site.
    The absence of this information limits users' understanding of the map's
    data and increases the potential for incorrect deductions and improper
    map-based decisions.

    FMCSA officials are aware of many of the limitations that we have
    identified and we recognize their efforts to improve the State Safety Data
    Quality map to date. However, they do not have a formal plan in place to
    implement improvements. Further, it is important to acknowledge these
    issues so that users understand the limitations of the data quality map as
    a tool. During our review we learned that not only was the data quality
    map consulted in awarding SaDIP funds, it was also consulted when awarding
    Motor Carrier Safety Assistance Program (MCSAP) High Priority funds.37
    Further, according to officials at FMCSA and Volpe National Transportation
    Systems Center, there has been some discussion of expanding the usage of
    the map in the future for CMV crash data efforts. As the data quality map
    gains wider use, it will become even more important that these limitations
    are addressed.

    Conclusions

    The grant program and FMCSA's collaborative efforts with states have had a
    positive impact on improving the quality of states' crash data, therefore
    ultimately enhancing the ability of both federal and state governments to
    make highway planning and safety enforcement decisions. While states have
    made progress in improving the quality of this data-in terms of
    timeliness, completeness, accuracy, and consistency-much remains to be
    done. With additional funding through the reauthorization of the Safe,
    Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for
    Users and as states refine and target areas needing further improvement
    with respect to their crash data, it is expected that additional states
    will participate in the program. FMCSA will need a more formal framework
    to better ensure that the decision-making process for awarding funds to
    SaDIP program applicants is conducted uniformly.

    FMCSA's efforts to improve the quality of commercial motor vehicle crash
    data have brought considerable attention to the issues associated with
    poor commercial motor vehicle crash data. Providing states with funding to
    improve their CMV data quality and publicizing a rating through a data
    quality map are incentives that work in tandem to maximize states'
    efforts. It is clear that states pay attention to their ratings of "good,"
    "fair," or "poor" on the data quality map. However, the limitations we
    identified highlight some important concerns with the data quality map's
    ability to measure progress and accurately portray states' commercial
    motor vehicle crash data quality. As FMCSA continues to make improvements,
    it will be important for these ratings to become more precise, so that
    FMCSA and the states can obtain the clearest picture possible of the
    progress being made.

    Recommendations for Executive Action

    We recommend that the Secretary of Transportation direct the Administrator
    of FMCSA do the following:

    o Establish specific guidelines for assessing state proposals for SaDIP
    grants in order to better assess and prioritize states' funding requests
    and provide uniformity in awarding funds.

    o Increase the State Safety Data Quality map's usefulness as a tool for
    monitoring and measuring commercial motor vehicle crash data by ensuring
    that it adequately reflects the condition of the states' commercial motor
    vehicle crash data and continues to motivate states in their improvement
    efforts. Specifically, FMCSA should develop a plan for assessing and
    improving the data quality map's methodology. In addition, FMCSA should
    display an overall crash data rating separately from the inspection
    rating, and provide information on the limitations of the State Safety
    Data Quality map and the underlying ratings on FMCSA's Analysis and
    Information (A&I) Online Web site.

    Agency Comments and Our Evaluation

    We provided a draft of this report to the Department of Transportation for
    its review and comment. The department agreed with our findings and
    recommendations in this report. Department officials provided some
    technical comments and some minor additions to provide more detail on
    FMCSA's training efforts.

    We will send copies of this report to the interested congressional
    committees, the Secretary of Transportation, and other interested parties.
    We will make copies available to others upon request. In addition, the
    report will be available at no charge on GAO's Web site at h
    ttp://www.gao.gov.

    If you or your staff has any questions about this report, please call me
    at (202) 512-6570. Contact points for our Offices of Congressional
    Relations and Public Affairs may be found on the last page of this report.
    Key contributors to this report are listed in appendix VI.

    Katherine Siggerud Director, Physical Infrastructure

    Objectives, Scope and MethodologyAppendix I

    Congress asked us to review the Federal Motor Carrier Safety
    Administration's (FMCSA) program for helping states improve their
    commercial motor vehicle crash data. As part of this review, we were asked
    to describe the benefits obtained through the program, identify what can
    be done to improve the effectiveness of the program, and address concerns
    regarding crash data raised in a February 2004 Department of
    Transportation (DOT) Inspector General's report. The specific objectives
    of this report were to explain (1) what is known about the quality of
    commercial motor vehicle crash data and what states are doing to improve
    it, and (2) the results of FMCSA's efforts to facilitate the improvement
    of the quality of commercial motor vehicle crash data sent to the federal
    government.

    To provide information on the quality of states' commercial motor vehicle
    (CMV) crash data and efforts to improve it, we reviewed grant applications
    and project information from 39 states. Of these 39 states, 34
    participated in the Safety Data Improvement Program (SaDIP) and 5 states
    either chose not to participate in the program or their proposals were not
    accepted. We also conducted site visits to six of these states--Georgia,
    Minnesota, North Carolina, Ohio, Oklahoma, and Washington. We chose our
    case study states based on a variety of criteria, including participation
    in the SaDIP grant program, the type of agencies with which the state
    works under SaDIP, the number of CMV crashes in the state, the number of
    reported CMV crash fatalities, and data quality map ratings. To help
    ensure that our states reflected a variety of experiences, we chose states
    that had different combinations of these criteria. To understand the
    insights and experiences of states that no longer participated-or had
    never participated-in SaDIP, we also interviewed officials in Michigan,
    Missouri, New Hampshire, and New Jersey.1  While the results from the case
    studies and interviews cannot be projected to the universe of states, they
    are nonetheless useful in illustrating the uniqueness and variation of CMV
    crash data systems and the challenges states face in improving them.
    During our case study visits we met and discussed the status of state
    crash data systems with a variety of traffic safety data officials.2  The
    discussions included gathering information on FMCSA's data quality
    criteria3  but also included, for those participating in the program,
    state objectives and progress made with SaDIP funds. For additional
    perspective, we also interviewed key experts from organizations
    responsible for the development of crash data systems and models used by
    FMCSA, carrier industry officials, and public interest organizations.
    Finally, we conducted a literature review of studies published by the
    University of Michigan Transportation Research Institute (UMTRI). The
    Institute plans to conduct studies in all states to determine where
    problems are occurring in the collection of CMV crash data and in the
    reporting of this data to FMCSA. As of September 2005, studies had been
    conducted in eight states. We reviewed the studies and determined the
    methodologies used in determining whether Motor Carrier Management
    Information System (MCMIS) and state CMV crash data were sound. We
    reviewed the studies conducted by UMTRI and determined that the
    methodologies used in assessing the quality of the data states submit to
    FMCSA were sound and that the studies provided sufficiently reliable
    results for our purposes. Through site visits, a review of grant
    applications, interviews with relevant stakeholders/experts, and these
    studies, we were able to determine shortcomings in the reliability of
    FMCSA's CMV crash data. However, we determined that the data was
    sufficiently reliable for our purpose of case study selection.

    To provide results of FMCSA's efforts to facilitate the improvement of the
    state CMV crash data quality, we conducted interviews with officials from
    participating states and from FMCSA concerning the administration and
    management of SaDIP. We also analyzed the guidance and support provided by
    FMCSA to states for CMV data improvement efforts and assessed FMCSA's role
    in coordinating CMV data quality initiatives. In addition, we reviewed
    FMCSA's State Safety Data Quality map and assessed the methodology used by
    FMCSA in evaluating states' crash data quality. We interviewed officials
    and key experts at FMCSA and the Volpe National Transportation Systems
    Center responsible for developing and managing the data quality map. We
    also interviewed state officials from states participating-and not
    participating-in the SaDIP program to find out their views on the data
    quality map and its use as a monitoring tool for CMV crash data
    improvements.

    We conducted our review from February 2005 through November 2005 in
    accordance with generally accepted government auditing standards.

    State Safety Data Quality Map LimitationsAppendix II

    This appendix explains, in greater detail than the body of our report, the
    concerns we (and others) have raised about the limitations in the
    methodology FMCSA uses to develop ratings for the State Safety Data
    Quality map. The measures FMCSA employs to measure the completeness,
    timeliness, and accuracy of CMV crash data quality are limited, and do not
    provide comprehensive measurements of these attributes as established by
    the general data standards discussed in the body of this report. As a
    result, the ability to draw conclusions about the actual quality of a
    state's data is limited.

    The definitions FMCSA uses for each of the crash measures are shown in
    table 5, together with the criteria that constitute a rating of "good,"
    fair," or "poor." In the sections that follow we explain the limitations
    associated with each measure, followed by other limitations that stem from
    the current methodology.

    Table 5: State Safety Data Quality Map Measures for CMV Crashes

    Data quality                Measure                       Criteria
         standard
    Completeness The measure compares CMV fatality data Good = match to FARS
                    in the MCMIS database against those in is greater than or
                    the Fatality Analysis and Reporting    equal to 90 percent
                    System (FARS). The FARS data is
                    available through 2003.                Fair = match to FARS
                                                           is between 80 and 89
                                                           percent

                                                           Poor = match to FARS
                                                           is below 80 percent
    Timeliness   This measure reflects the percentage   Good = the percentage
                    of state-reported crash records        of reported records
                    uploaded to the MCMIS database within  within 90 days is
                    the 90-day standard.                   greater than or equal
                                                           to 85 percent

                                                           Fair = the percentage
                                                           of reported records
                                                           within 90 days is
                                                           between 60-84 percent

                                                           Poor = the percentage
                                                           of reported records
                                                           is less than 60
                                                           percent
    Accuracy     This measure reflects the percentage   Good = the percentage
                    of state-reported crash records (fatal of matched records is
                    and non-fatal) that were matched to a  greater than or equal
                    motor carrier in MCMIS over a 12-month to 95 percent
                    period.
                                                           Fair = the percentage
                                                           of matched records is
                                                           between 85 and 94
                                                           percent

                                                           Poor = the percentage
                                                           of matched records is
                                                           below 85 percent

    Source: FMCSA.

    Completeness: Overall Completeness is Based on Fatal Crashes Only

    A key limitation in FMCSA's measure of crash data completeness is the
    inability to evaluate completeness against nonfatal CMV crashes.
    Currently, the total number of nonfatal crashes occurring on a
    state-by-state basis is

    unknown; no baseline exists against which to measure these records.1
    Consequently, FMCSA is limited to measuring crash data completeness with
    fatal CMV crashes, which is approximately 3 percent of all CMV crashes.2

    Even within this narrowed dataset, the use of FARS3 as the basis of
    comparison poses other limitations:

    o There are notable differences in definitions used in the two databases.
    Most importantly, MCMIS can be subject to individual state definitions
    that may differ from FARS. According to officials at the Volpe National
    Transportation Systems Center, the range of these state-driven
    definitional differences is unknown. However, based on anecdotal evidence
    received from states and reported by Volpe, these differences can vary by
    state and can account for about 4 percent of crash records.4 Some examples
    are below:

    o A crash fatality resulting from the private use of a large truck may not
    meet the criteria as a reportable crash for MCMIS, but it is considered a
    large truck crash fatality by FARS.

    o MCMIS defines a large truck as any truck greater than a 10,000 gross
    vehicle weight rating. Many states do not collect the gross vehicle weight
    rating of vehicles and instead define a large truck based on an older
    FMCSA reporting criterion of greater than six tires or another definition
    of their choosing. As such, counts of large trucks derived from state
    crash databases (and reported to FARS) may be inconsistent with counts of
    vehicles that were reported to MCMIS.5

    The effect of these definitional differences is evident when comparing
    MCMIS fatal CMV crashes to FARS data-the state's index on completeness for
    crash data rises above 100 percent. The extent to which this occurs is
    substantial: in our analysis of the June 2005 issuance of the map, 24 of
    51 states (47 percent) had a completeness measure of greater than 100
    percent.  6

    o FARS data is not current. FARS is released once per year, detailing the
    prior calendar year's crash statistics. The lag between the most recently
    available FARS data and the most recent map issuance can be considerable.
    For example, the 2004 FARS annual report is not scheduled to be released
    until fall 2005 and as a result, the baseline for completeness data in the
    June 2005 map is based upon 2003 crash data.

    Timeliness: Timeliness is Not Based on All Reported Crashes

    The timeliness measure currently relies on a subset of records states
    submit to MCMIS. Any record that has been edited or changed since it was
    originally entered into MCMIS is not included in the calculations.7 If a
    record is edited, the initial upload date is replaced with the date it was
    updated. Because FMCSA's timeliness rating is based on the percentage of
    crash reports uploaded within 90 days of the crash, and edited records no
    longer reflect their initial upload date, they cannot be used in the
    timeliness calculation without distorting a state's rating. The
    consequence, however, is that timeliness is not measured against the
    entire universe of crashes in MCMIS-the more records a state edits, the
    fewer records its timeliness rating represents. FMCSA has acknowledged
    this problem with the edited records and is taking steps to resolve it.

    Another limitation with this measure is that efforts to reduce backlogs of
    crash records-a positive effort-can have a negative effect on the
    timeliness rating. If a state submits a backlog of reports from CMV
    crashes that occurred more than 90 days previously, and the crash took
    place during the period of time which the FMCSA rating covers, then the
    state's timeliness rating will be negatively affected. Conversely, if a
    state has a large backlog, its current timeliness rating may not be a
    meaningful representation of timeliness because the methodology has no way
    to capture those records that are accruing in the backlog.

    Accuracy: Accuracy is Based on Only One Variable

    The measure for accuracy on the data quality map is based only on a match
    of a CMV crash report in MCMIS, against a registered carrier in MCMIS. The
    U.S. Department of Transportation (DOT) assigns each interstate motor
    carrier a unique identifier number-the DOT number. For this measure,
    accuracy is evaluated primarily based on the degree to which a carrier's
    DOT number is the same as identified for a crash in MCMIS.8 Therefore, a
    crash report with a missing or invalid DOT number may be considered
    inaccurate, even if the rest of the information on the report provides
    accurate information on the crash. Currently, the data quality map does
    not assess the accuracy of any other data elements; however, FMCSA has
    been working with states to improve their collection of CMV crash
    information. For example, recently FMCSA distributed visor cards to state
    officials that explain how to determine who the carrier is, and where the
    correct carrier's DOT number can be found. See appendix III for copies of
    these cards.

    FMCSA Reportable Crash, CMV, and Carrier Identification Visor
    CardsAppendix III

    The following are copies of visor identification cards that FMCSA created
    as educational tools for law enforcement officers. The cards can be placed
    in officer's sun visor and referenced to determine whether a CMV crash
    should be reported, for identifying a vehicle as a CMV, and for
    identifying the correct carrier involved in the crash. FMCSA provided
    these to states to distribute to enforcement officers with the intention
    they will increase the officers' ability to properly identify a CMV and a
    reportable CMV crash.

    Figure 3: FMCSA Reportable Crashes

    Figure 4: Reportable Commercial Motor Vehicle Configurations and Cargo
    Body Type

    Figure 5: Responsible Carrier and Correct DOT Number Identification

    SaDIP Grant and Cooperative Agreement Distribution by StateAppendix IV

    Table 6: Distribution of SaDIP Grants by State

            State           FY2002    FY2003      FY2004      FY2005        Total
    Alaska             $198,000         0           0    $218,626     $416,626
    California                0              $100,000           0      100,000
    Connecticut         234,056         0           0           0      234,056
    Georgia             269,820         0     409,000     553,733    1,232,553
    Indiana             179,321   242,423           0           0      421,744
    Kansas                    0         0           0     373,436      373,436
    Kentucky            199,000         0           0           0      199,000
    Maryland            145,600         0           0           0      145,600
    Massachusetts       100,000   249,972           0           0      349,972
    Michigan                  0   115,845           0           0      115,845
    Minnesota           363,000         0     620,000           0      983,000
    Montana                   0         0           0     328,564      328,564
    Nebraska                  0         0       2,000       3,342       $5,342
    Nevada              427,443         0           0     350,000      777,443
    New Hampshire             0    98,068           0           0       98,068
    New Mexico                0         0           0     250,000      250,000
    New York                  0         0     300,000           0      300,000
    North Carolina            0         0     193,350           0      193,350
    Oklahoma                  0         0     131,111           0      131,111
    Pennsylvania              0         0           0   2,000,000    2,000,000
    Rhode Island         25,000         0      26,000           0       51,000
    South Carolina      350,000         0           0           0      350,000
    South Dakota        270,000         0           0           0      270,000
    Tennessee                 0         0     436,027     149,334      585,361
    Texas                     0   148,424     533,611           0      682,035
    Utah                      0         0     433,500     128,853      562,353
    Vermont                   0         0           0     134,000      134,000
    Washington                0         0     803,935           0      803,935
    West Virginia       157,500         0           0     603,943      761,443
    Total            $2,918,740  $854,732  $3,988,534  $5,093,831  $12,855,837

    Source: FMCSA.

    Note: This data is current as of September 2005.

    Table 7: Distribution of SaDIP Cooperative Agreements by State

          State            FY2002   FY2003       FY2004       FY2005        Total
    Colorado                 0        0            0   $1,460,000   $1,460,000
    Georgia                  0        0            0      612,051      612,051
    Iowa              $728,065        0            0      226,400      954,465
    Louisiana          829,625        0            0      569,000    1,398,625
    Ohio               406,330        0      $91,000      224,400      721,730
    Oklahoma                 0        0    1,150,390            0    1,150,390
    Tennessee          281,954        0      219,543      426,382      927,879
    Vermont                  0        0            0      136,500      136,500
    Virginia            42,300        0            0      431,617      473,917
    Total           $2,288,274       $0   $1,460,933   $4,086,350   $7,835,557

    Source: FMCSA.

    Note: This data is as of September 2005.

    SaDIP Case Study StatesAppendix V

    As part of our work, we conducted six case studies to examine how states
    are working to improve commercial motor vehicle crash data. Our visits to
    the six states yielded additional information about crash data quality
    improvement activities, the nature of their efforts, and the extent of
    progress made. States were chosen on a wide variety of factors, including
    crash data quality and participation in the SaDIP program.

    Georgia

    Georgia received a total of $1,844,604 between 2002 and 2005 (see table 8)
    to conduct crash data improvement projects. During this period, Georgia
    made significant improvement in its crash data quality, despite undergoing
    a major state government reorganization process.

    Table 8: Georgia SaDIP Funding History

          Award year      Funds awarded      Agency            Award type
    2005             $612,051           FMCSA       Cooperative agreement
    2005             $553,733           FMCSA       Grant
    2004             $409,000           FMCSA       Grant
    2002             $269,820           NHTSA       Grant

    Source: FMCSA.

    SaDIP Projects

    Georgia received its first SaDIP grant from the National Highway Traffic
    Safety Administration (NHTSA), and this grant was used to accomplish
    several projects, including: (1) hiring temporary employees to extract CMV
    crash reports from backlogged paper copies and microfilm them; (2)
    precoding the paper crash reports in preparation for their entry into the
    state repository system; (3) renting a mobile trailer to house temp
    employees; (4) developing a system to electronically transfer crash data
    from the state repository to the national database; (5) adding edit checks
    to the state crash database; (6) purchasing a new microfilm
    scanner/reader; and, (7) hiring a contractor to update the state's crash
    report manual. Progress on these projects was reported quarterly to NHTSA.

    Georgia received two additional grants after FMCSA became the lead agency
    for the SaDIP program. The first grant was used to maintain the temporary
    employees. Additionally, these funds were used to hire a statistician as a
    SaDIP advocate. This grant allowed Georgia to eliminate its reporting
    backlog from 2003 to the present. The second grant that Georgia received
    was used to fund its electronic crash records system. This system will
    electronically transfer crash records from local jurisdictions to the
    state repository system. This project has been put on hold following a
    state government reorganization.

    Crash Data Collection/Reporting Issues

    Georgia historically has had the worst CMV crash data quality rating in
    the nation (see crash data quality statistics, table 9). This is due to
    the fact that until recently, Georgia was not submitting crash data to
    FMCSA. Georgia's current overall rating is "good," and it is rated "good"
    for completeness and "fair" for timeliness and accuracy. For all other
    rating periods prior to March 2004, Georgia had an overall rating of
    "poor." This improvement in its data quality is due in large part to the
    SaDIP program. Georgia's non-reporting was due to a technical problem, but
    the state was able to correct it in a timely manner because of the SaDIP
    funding it received.

    Table 9: Georgia Crash Data Quality Statistics (Percentages)

                        March     June     September    December    March    June
                         2004     2004          2004        2004     2005    2005
    Completeness       117      115            95          95       95      95
    Timeliness           6        8            27          48       65      80
    Accuracy            84       82            82          83       94      93

    Source: FMCSA.

    SaDIP funds are the only federal funds currently being used to address
    data quality improvement in Georgia. Because the state's data quality was
    so poor, both FMCSA and the Federal Highway Administration indicated they
    would withhold federal funding unless Georgia improved its reporting. It
    was at this point that Georgia applied for SaDIP, and began focusing on
    its data quality. As of July 1, 2005, Georgia began a major state
    government reorganization process, and officials were unsure of how this
    would affect the quality of its crash data because all collection and
    reporting functions are moving to new departments. Officials expect to see
    a decrease in the state's rating during the transition period, but do not
    expect long-term problems.

    State/Federal Coordination Issues

    State and federal coordination on issues related to traffic safety data
    has not always been effective in Georgia. Between 1998 and 2002, Georgia
    had only submitted 250 fatal CMV crash records to FMCSA, yet between 2002
    through 2003 Georgia submitted 410 CMV fatalities to FARS. Georgia's
    Department of Public Safety, the agency responsible for crash reporting,
    did not have a functional computer system to upload the state's crash file
    to FMCSA. The problem was discovered in 2002 and FMCSA offered assistance
    to fix the problem, but the Department of Public Safety did not accept the
    offer. FMCSA then threatened to withhold Georgia's highway safety funds
    until the data issues were resolved; at this point the Department of
    Public Safety applied for-and received-SaDIP funding, and began work to
    resolve the crash data problems.

    Data Quality Map

    Georgia officials stated that the data quality map provided an incentive
    for the state to make data quality improvements in order to raise its
    rating from "poor" to "good." State and federal officials acknowledge that
    the map is one indication of the progress the state has made in improving
    its CMV data quality.

    Georgia officials never disagreed with their state's rating on the data
    quality map. However, officials felt that the map did not recognize
    improvements that were taking place within the state to improve crash
    data. The state plans to continue to make improvements to the crash
    records system even after the state reaches the "good" rating.

    Minnesota

    Minnesota received a total of $983,000 in 2002 and 2004 (see table 10) to
    conduct crash data improvement projects. During this period, Minnesota
    received safety data improvement grants from NHTSA and FMCSA.

    Table 10: Minnesota SaDIP Funding History

          Award year           Funds awarded           Agency       Award type
    2004                         $370,000        FMCSA        Grant
    2004                         $250,000        FMCSA        Grant
    2002                         $363,000        NHTSA        Grant

    Source: FMCSA.

    SaDIP Projects

    Minnesota received its first SaDIP grant while NHTSA was administering the
    program. This grant went to the Department of Public Safety, Office of
    Traffic Safety-the agency that receives NHTSA highway safety funds. The
    grant was used to fund two projects, including an electronic post-crash
    inspection form (Minnesota conducts in-depth post-crash inspections for
    crashes involving CMVs), and to purchase software that will allow for
    electronic transfer of CMV crash records between Driver and Vehicle
    Services and the state patrol (both agencies within the Department of
    Public Safety). In Minnesota, Driver and Vehicle Services collects crash
    reports from law enforcement agencies, and the state patrol is the office
    in charge of reporting crashes to FMCSA.

    The Minnesota state patrol received subsequent funding after FMCSA became
    the lead agency on the SaDIP program. These funds have been used to
    provide computer hardware so that state patrol officers can access the
    Internet and submit their crash reports electronically through an online
    form. The second component of the grant is to provide ongoing CMV crash
    training to law enforcement officers throughout the state. This training
    will focus on correctly identifying CMVs and teaching officers why CMV
    crash data is important, and what it is used for at the state and national
    levels.

    Crash Data Collection/Reporting Issues

    Before SaDIP funds were received, the Minnesota state patrol had limited
    access to Driver and Vehicle Services records. As a result, the state
    patrol was only able to report crashes to FMCSA that were investigated by
    their officers or were sent directly to the state patrol by local
    jurisdictions. Now all CMV records that are available at the Driver and
    Vehicle Services are reported to FMCSA (see crash data quality statistics,
    table 11).

    Table 11: Minnesota Crash Data Quality Statistics (Percentages)

                        March     June     September    December    March    June
                         2004     2004          2004        2004     2005    2005
    Completeness       117      117           116         123      121     121
    Timeliness          72       59            58          50       67      81
    Accuracy            78       74            82          86       91      93

    Source: FMCSA.

    The State of Minnesota encountered difficulty receiving crash reports from
    local law enforcement agencies. In addition, law enforcement officers
    incorrectly identified CMVs on crash reports. If law enforcement officers
    do not correctly identify the vehicle involved in the crash as a CMV, then
    the crash may not be extracted from the state crash file for submission to
    FMCSA.

    State/Federal Coordination Issues

    Minnesota officials said that the state now has a crash data users group
    as a subsection of its Traffic Records Coordinating Committee. This group
    allows all users of crash data to discuss their data needs and
    limitations, and provides a forum for making collaborative recommendations
    regarding crash data improvements.

    Minnesota officials and the FMCSA Division Administrator work very closely
    with each other to monitor and improve CMV crash data quality. Neither of
    these groups feels that the FMCSA data quality rating map is an accurate
    portrayal of a state's data quality. For example, Minnesota's rating
    decreased due to the fact that it was implementing its electronic data
    transfer software, and during that period crash records were not reported
    to FMCSA in a timely fashion. The map showed the state's rating as "poor"
    during this period, which did not reflect the fact that the state was
    making strides in improving its data quality.

    Minnesota uses several sources of funding to improve its overall crash
    data and CMV crash data. In addition to FMCSA funding, NHTSA and state
    funds have been used to improve the state's data quality.

    Data Quality Map

    Minnesota crash data has gone from an overall "fair" rating to "poor" to
    its current rating as "good." The drop in the state's rating was actually
    a result of the state not being able to report crashes during the
    implementation of SaDIP-funded improvement projects. After the electronic
    systems were fully implemented, the state's rating improved to "good" for
    completeness, and "fair" for timeliness and accuracy.

    Officials in Minnesota recognize that the map is an effective tool for
    getting the attention of decision-makers. While the map may not be an
    accurate assessment of the state's current data quality, both state and
    federal officials recognize that it does help to spotlight Minnesota's
    data quality status. Federal and state officials use the data quality map
    as a measure of the state's progress in improving data quality. State
    officials also use the map as an indicator of the success of the state's
    SaDIP project. Finally, officials in Minnesota said that the data quality
    map has been used to identify data quality problems in the state,
    including timeliness and accuracy, and to create projects to improve those
    measures.

    North Carolina

    North Carolina received a total of $193,350 in 2004 (see table 12) to
    conduct crash data improvement projects. While North Carolina has
    excellent crash data reporting at the state level, North Carolina
    continues to have complications reporting that data to FMCSA. North
    Carolina is a priority state for FMCSA given the fact that it is one of
    the top five states for commercial motor vehicle accidents in the nation.

    Table 12: North Carolina SaDIP Funding History

          Award year           Funds awarded           Agency       Award type
    2004                         $193,350        FMCSA        Grant

    Source: FMCSA.

    SaDIP Projects

    North Carolina is using its SaDIP grant to conduct two projects. First,
    the state conducted an analysis to determine what differences existed
    between the state data file and the federal crash file, and to try to
    determine why these differences were occurring. The completed analysis did
    not appear to fully address North Carolina's biggest problem, which is the
    data transfer between the two files.

    North Carolina is also using SaDIP to update a backlog it has in entering
    crash reports into the state data file by providing overtime pay to
    employees.

    Crash Data Collection/Reporting Issues

    Historically, North Carolina has had poor CMV crash data (see table 13).
    North Carolina is also one of FMCSA's priority states because of the large
    amount of crashes that take place in the state. It was rated as having
    "poor" data for each period since FMCSA began rating states.

    Table 13: North Carolina Crash Data Quality Statistics (Percentages)

                       March    June     September    December   March      June
                        2004    2004          2004        2004    2005      2005
    Completeness       82      82            77          77      77 77
    Timeliness          0     100             0           0       1  2
    Accuracy           81     100            81          80      80 81

    Source: FMCSA.

    North Carolina is unique in that it does not have a problem receiving
    crash reports from localities across the state. In other states, this is
    the biggest problem contributing to the states' data quality issues.1 In
    North Carolina the problem is getting records from the state crash file
    into the FMCSA data systems. FMCSA officials worked extensively with North
    Carolina to fix the file compatibility issues that are creating the
    problem exporting the data to FMCSA, but North Carolina has failed to
    correct its information technology problems. According to state officials,
    the biggest issue in North Carolina is backlog. Even if the state corrects
    its backlog, the state's data rating will not improve because the records
    are not transferring correctly into the federal database. Additionally,
    North Carolina does not require certain elements to be captured on its
    crash report or in its state database that are required in the federal
    database. Specifically, North Carolina does not require that motor carrier
    DOT numbers are collected, which may contribute to the state's "poor"
    accuracy rating.

    While North Carolina has very good data at the state level, the state's
    largest problem is reporting that data to the federal database. It appears
    that North Carolina's SaDIP projects are not focusing on this issue.

    State/Federal Coordination Issues

    North Carolina coordination between state agencies involved in CMV crash
    data has not always been effective. A major reorganization of the state's
    highway safety agency contributed to this lack of coordination, but the
    state is working to improve cooperation among these agencies with regards
    to traffic records. The largest factor contributing to North Carolina's
    poor crash data quality appears to be a lack of understanding by some
    state officials regarding how to convert a crash data file to the correct
    format in order to submit it to FMCSA. The SaDIP grant has provided the
    state the opportunity to review its crash data problems, a project that
    the state would not have conducted otherwise. The state is also using its
    own funds to implement an electronic crash reporting system in order to
    get crash data into the state data file more quickly.

    Data Quality Map

    State and federal officials in North Carolina use the data quality map to
    measure the progress the state is making with its SaDIP grant and with
    data improvements overall.

    North Carolina state officials use the map as an incentive for
    implementing data improvements and they appear motivated to change North
    Carolina's rating for the better. The map has also been used in North
    Carolina to identify data quality problems and target improvement efforts,
    although more still needs to be done.

    Both federal and state officials expect to continue to make improvements
    to the state's traffic records infrastructure after the state achieves the
    highest data rating.

    Ohio

    Ohio received a total of $721,730 between 2002 and 2005 (see table 14) to
    conduct crash data improvement projects. Ohio has been very proactive in
    addressing traffic safety data concerns and the state has paid particular
    attention to using crash data for planning.

    Table 14: Ohio SaDIP Funding History

        Award year      Funds awarded         Agency            Award type
    2005                  $224,400     FMCSA          Cooperative agreement
    2004                   $91,000     FMCSA          Cooperative agreement
    2002                  $406,330     NHTSA/FMCSA    Cooperative agreement

    Source: FMCSA.

    SaDIP Projects

    Ohio was one of the first 5 pilot states to participate in the SaDIP
    program. It received funds through a cooperative agreement with the
    General Services Administration (GSA) and NHTSA. Through this agreement,
    the state created electronic crash reporting capabilities, purchased
    handheld devices, modified crash reporting software, and is providing
    training to law enforcement officers to help them properly identify CMV
    crashes.

    Until recently, when FMCSA gained authority for the SaDIP cooperative
    agreements, Ohio had not submitted any progress reports to NHTSA on the
    status of these projects.

    Crash Data Collection/Reporting Issues

    Ohio has a "fair" overall data rating, with "good" ratings for
    completeness and accuracy, but a "poor" rating for timeliness (see crash
    data quality statistics, table 15). Ohio has been very aware of the
    importance of CMV crash data and state officials have been working to
    improve it for a long time, even before the SaDIP program.

    Table 15: Ohio Crash Data Quality Statistics (Percentages)

                        March    June September 2004    December    March   June
                         2004    2004                       2004     2005   2005
    Completeness        95      95             95          95       95     95
    Timeliness          15      17             11          23       40     46
    Accuracy            84      97             97          98       98     98

    Source: FMCSA.

    The state of Ohio has over 1,000 local law enforcement jurisdictions that
    are responsible for reporting CMV accidents. Both large and small
    jurisdictions are likely responsible for Ohio's "poor" crash data
    timeliness rating. The state has reduced the time lag in receiving crash
    reports from local jurisdictions from 62 days in 2004 to 30 days in 2005.

    State/Federal Coordination Issues

    Ohio state agencies involved in the collection and reporting of CMV crash
    data appear coordinated with each other, and have an active Traffic
    Records Coordinating Committee, which includes participation by FMCSA
    division officials. Ohio uses multiple sources of funding to address data
    quality issues, including state funds, and the state is proactive in data
    improvement projects.

    Data Quality Map

    State and federal officials said that the map provided one gauge of Ohio's
    data quality, but they felt the map would be more useful if it were
    updated more often. The map has brought more attention to data quality
    issues in Ohio and is included in reports to state leaders.

    State and federal officials in Ohio said that data quality improvements
    would still be a top priority regardless of the data quality map. Ohio
    officials also stated that the inaccuracies in the map had a big effect on
    staff morale. Local officials said they are working hard to improve their
    state's data quality, but the map does not accurately capture that
    improvement. State officials recognize the map provides a major incentive
    for implementing data quality improvements and maintaining the state's
    standing as a leader in traffic data quality.

    Oklahoma

    Oklahoma received a total of $1,281,501 in 2004 (see table 16) to conduct
    crash data improvement projects. Oklahoma's traffic records coordinating
    committee has taken the lead in the coordination of crash data projects in
    the state.

    Table 16: Oklahoma SaDIP Funding History

         Award year       Funds awarded        Agency          Award type
    2004                  $1,150,390      FMCSA      Cooperative agreement
    2004                    $131,111      FMCSA      Grant

    Source: FMCSA.

    SaDIP Projects

    Oklahoma began participation in the SaDIP program in July 2004 when it
    received a grant to purchase computer equipment, and then entered into a
    four year cooperative agreement with FMCSA in December 2004 to support its
    long term data quality improvement plans. This includes conducting a
    traffic records assessment, hiring a SaDIP coordinator, funding data entry
    for the record backlog, revising the state's crash report, and initiating
    a mobile data collection pilot project.

    Oklahoma officials said that the projects that are taking place using
    SaDIP funds would not have been funded otherwise. The SaDIP grant and
    long-term cooperative agreement have allowed the state to focus on CMV
    data and to begin to make data-driven decisions in its highway safety
    planning.

    Crash Collection/Reporting Issues

    Until recently Oklahoma had a nine month crash report backlog to be
    entered into the state system (see crash data quality statistics, table
    17). The backlog was primarily due to insufficient state resources.
    Overtime hours funded by SaDIP has helped to alleviate this backlog.

    Table 17: Oklahoma Crash Data Quality Statistics (Percentages)

                        March     June     September    December    March    June
                         2004     2004          2004        2004     2005    2005
    Completeness       104      104           102         102      101     101
    Timeliness          93       92            93          92       90      83
    Accuracy            84       86            87          86       86      84

    Source: FMCSA.

    State/Federal Coordination Issues

    Oklahoma has been working with the GSA to receive payments on its
    cooperative agreement, and officials said that this has caused the state
    some confusion.

    Oklahoma's crash data divisions are housed in the state's Department of
    Public Safety and Department of Transportation. While crash records staff
    in both agencies work closely together to make sure that the state's crash
    file is complete, Oklahoma officials were unaware of the value of CMV
    crash data at the national level, or of the criteria that FMCSA uses to
    rate CMV crash data.

    Oklahoma is using funds from several sources to improve its entire crash
    data system, including funds provided by NHTSA and the Federal Highway
    Administration. SaDIP funds are the only funds that the state is using
    specifically for CMV data improvements.

    Data Quality Map

    While high-ranking officials in Oklahoma's Department of Public Safety and
    the Highway Patrol are familiar with the data quality map, state officials
    involved with the SaDIP program were less familiar. We found that state
    officials involved with the SaDIP program had a detailed understanding of
    their state's data quality, but had not used the data quality map as an
    indicator of progress for the SaDIP cooperative agreement.

    Oklahoma officials stated that they would undertake their data improvement
    program regardless of whether or not the state was ranked "good." State
    officials, for example, recognized that more improvements could be made in
    the state's timeliness measure even though the state had a "good" rating
    in this category.

    Washington

    Washington received a total of $803,935 in 2004 (see table 18) to conduct
    crash data improvement projects. The state's traffic records coordinating
    committee is leading an electronic information processing initiative
    designed to reduce crash reporting inefficiencies and help the state meet
    national traffic safety goals.

    Table 18: Washington SaDIP Funding History

          Award year           Funds awarded           Agency       Award type
    2004                         $188,460        FMCSA        Grant
    2004                         $615,475        FMCSA        Grant

    Source: FMCSA.

    SaDIP Projects

    SaDIP is primarily being used to implement an electronic data feed between
    the Washington Department of Transportation and the state patrol. This
    will allow records to be submitted instantly to the state patrol when they
    are entered into the state department of transportation database. It will
    also make records searchable so that eligible CMV crashes that are
    misidentified and not sent to the state patrol can be identified as CMV
    crashes, improving the completeness of the state's CMV crash data.

    SaDIP is also being used to eliminate a six-month backlog of crash reports
    that need to be entered into the state's database housed at the Washington
    Department of Transportation. Until reports are entered into the state
    Department of Transportation system, they cannot be transferred to the
    state patrol electronically, nor can those reports incorrectly identified
    as non-CMV crashes be easily identified. Until this takes place,
    Department of Transportation employees identify CMV reports among all
    crash reports and provide the state patrol with paper copies of the
    reports.

    Crash Collection/Reporting Issues

    The largest problem that Washington State has with its crash data is
    receiving the entire crash report, including supplemental form, from the
    law enforcement offices that generate them (see crash data quality
    statistics, table 19).

    Table 19: Washington Crash Data Quality Statistics (Percentages)

                        March     June     September    December    March    June
                         2004     2004          2004        2004     2005    2005
    Completeness        96       96           103         103      108     108
    Timeliness          99       98            98          99       99      99
    Accuracy            95       96            96          97       97      97

    Source: FMCSA.

    Washington State also has a problem with incorrect identification of CMVs
    on police accident reports. If police do not correctly identify the
    vehicle involved in the crash as a CMV, then it may not get extracted from
    the state crash file for submission to FMCSA.

    State/Federal Coordination Issues

    Washington State has very good cooperation among state agencies involved
    in crash data collection and reporting. The state also has a good working
    relationship with its FMCSA division office. It also uses FMCSA and state
    funds to address data quality issues, making decisions on how to use these
    funds effectively through an active Traffic Records Coordinating
    Committee.

    While SaDIP initiatives in Washington State are a topic of discussion at
    the Traffic Records Coordinating Committee meetings, the committee
    coordinator was not specifically aware of FMCSA's grant-making process and
    how those grants are accessed and then allocated by the state. Whereas
    NHTSA funds are accessed via the Traffic Records Coordinating Committee
    forum, FMCSA funds are processed within the Washington State Patrol's
    Commercial Vehicle Division. To date there has been no problem targeting
    safety data funding to immediate priorities as identified by the Traffic
    Records Coordinating Committee, but this is an area that may need to be
    addressed in the future.

    Data Quality Map

    Washington State officials recognize that the data quality map is an
    important indicator of the state's progress in improving its crash records
    system. According to state officials, the data quality map has been used
    to measure the progress of the state's data quality improvements in
    general.

    State officials also indicated that they would continue to make
    improvements to their traffic records systems regardless of the state's
    data quality rating. Specifically, the Traffic Records Coordinating
    Committee is helping to coordinate the state's electronic reporting
    system.

    Even though the state has consistently ranked "good" on the map since its
    inception, state officials report that the map has been helpful for
    identifying areas where the state can make data quality improvements.

    GAO Contact and Staff AcknowledgmentsAppendix VI

    Katherine Siggerud (202) 512-6570 or siggerudk@gao.gov

    In addition to the contact named above, Randall Williamson, Assistant
    Director; Tida Barakat; Jonathan Carver; Bert Japikse; Sara Ann
    Moessbauer; Fred Swenson; Sharon Silas; and Stan Stenersen made key
    contributions to this report.

    (542054)

    www.gao.gov/cgi-bin/getrpt?GAO-06-102.

    To view the full product, including the scope

    and methodology, click on the link above.

    For more information, contact Katherine Siggerud, (202) 512-6570,
    Siggerudk@gao.gov.

    Highlights of GAO-06-102, a report to congressional committees

    November 2005

    HIGHWAY SAFETY

    Further Opportunities Exist to Improve Data on Crashes Involving
    Commercial Motor Vehicles

    Large trucks make up 3 percent of the nation's registered vehicles, but
    they were involved in 11 percent of all fatal crashes in 2003. To reduce
    the fatality rate, the Federal Motor Carrier Safety Administration (FMCSA)
    sets national safety goals and works in partnership with states to reach
    them. Crash data collected by states and submitted to FMCSA is key to
    these efforts, and to be fully useful, this data must be complete, timely,
    accurate, and collected in a consistent manner. GAO addressed (1) what is
    known about the quality of commercial motor vehicle crash data, and what
    states are doing to improve it, and (2) the results of FMCSA's efforts to
    help states make improvements.

    What GAO Recommends

    To ensure uniformity in awarding data improvement funds to states, FMCSA
    should establish specific guidelines for assessing and awarding state
    funding requests.

    Also, in order to address limitations in its data quality map, FMCSA
    should develop a plan for assessing and improving the map's methodology,
    and it should provide a crash specific data rating and limitations of the
    map on its Web site.

    The Department of Transportation agreed with our findings and
    recommendations in this report.

    Overall, commercial motor vehicle crash data does not yet meet general
    data quality standards of completeness, timeliness, accuracy, and
    consistency. For example, FMCSA estimates that nearly one-third of
    commercial motor vehicle crashes that states are required to report to the
    federal government were not reported, and those that were reported were
    not always accurate, timely, or consistent. States are undertaking four
    types of activities to improve data quality, including analyzing existing
    data to identify problems and develop plans for addressing them, reducing
    backlogs of data that have not been entered into state-level databases,
    developing and implementing electronic data systems, and providing
    training. As a result of these efforts, states have recently improved both
    the timeliness and the number of reportable crashes submitted to FMCSA.

    FMCSA has two main efforts to support states in improving their reporting
    of commercial motor vehicle crash information-a commercial vehicle crash
    data improvement program and a data quality rating system-and both appear
    to be beneficial. Through the data improvement program, FMCSA has provided
    nearly $21 million in discretionary grants to 34 states from 2002 through
    2005. These grants have ranged from $2,000 to $2 million and have helped
    states conduct a variety of data improvement activities. GAO did not find
    problems with FMCSA's oversight of the program, but we did note that FMCSA
    does not have formal guidelines for awarding grants to states. As state
    participation in the program increases, formal guidelines and systems
    would likely assist FMCSA in prioritizing states' requests and ensuring
    consistency in grant awards.

    FMCSA's second major effort, a tool for rating states' data quality, has
    proven to be an important tool for states to use in improving their crash
    data as well. These results are presented in a map that rates each state's
    data quality as "good," "fair," or "poor." According to both FMCSA and
    state officials, the map and the underlying rating system serve as an
    incentive for states to improve their crash data. While the map is useful,
    GAO identified problems in the methodology used for developing ratings.
    These problems may potentially lead to drawing erroneous conclusions about
    the extent of improvements that have been made, and discourage states from
    continuing to devote attention and resources to areas needing improvement.

    FMCSA's June 2005 data quality map showing each state's overall data
    quality rating for crash and inspection data.
    *** End of document. ***