Defense Transportation: Plan Needed for Evaluating the Navy Personal
Property Pilot (Letter Report, 06/23/1999, GAO/NSIAD-99-138).

The Defense Department (DOD) has been trying since 1994 to reengineer
the personal property program by simplifying processes, controlling
costs, ensuring quality by adopting best practices used by commercial
businesses, and relieving carriers of terms and conditions unique to the
military. Last year, the Navy began a pilot that allows servicemembers
to choose a carrier and arrange their own moves. Between January 1998
and March 1999, 223 servicemembers used the pilot program to arrange
their own move. Participation has been relatively low compared to the
three other pilots planned or under way, which involve substantially
more shipments--about 1,400 to 45,000 shipments annually. Surveys show
that participants are satisfied with the pilot and would use the option
again. The option of allowing members to arrange their own moves is not
offered in any of the other pilots, which are broader in scope and are
intended to replace the current program. GAO recommends that DOD
consider testing the Navy pilot and its unique features in conjunction
with one or more of the other pilots.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-99-138
     TITLE:  Defense Transportation: Plan Needed for Evaluating the
	     Navy Personal Property Pilot
      DATE:  06/23/1999
   SUBJECT:  Military personnel
	     Customer service
	     Employee transfers
	     Transportation costs
	     Personal property
	     Household goods
	     Program evaluation
IDENTIFIER:  Army Hunter Pilot Program
	     Navy Personal Property Pilot Project
	     DOD Full Service Moving Project

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

    United States General Accounting Office GAO                 Report
    to Congressional Committees June 1999           DEFENSE
    TRANSPORTATION Plan Needed for Evaluating the Navy Personal
    Property Pilot GAO/NSIAD-99-138 United States General Accounting
    Office
    National Security and Washington, D.C. 20548
    International Affairs Division B-282502 Letter June 23, 1999
    Congressional Committees Since 1994, the Department of Defense
    (DOD) has been engaged in initiatives to reengineer the personal
    property program to simplify current processes, control program
    costs, ensure quality of service by adopting commercial business
    processes characteristic of world-class businesses, and relieve
    carriers of DOD-unique terms and conditions.  We recently
    testified before the Military Readiness Subcommittee, House Armed
    Services Committee, on the results of the Army's pilot and DOD's
    plans to evaluate the other three pilots that are planned or under
    way.1  The statement of managers in the conference report on the
    1997 DOD Appropriations Act conference report directed us to
    validate the results and savings achieved from any personal
    property pilot program before DOD proposes further expansion of
    such programs.2  In January 1998, the Navy began a servicemember
    arranged move pilot that allows servicemembers to select a carrier
    and arrange their own moves. This report provides status
    information on the Navy pilot and our assessment of plans for
    evaluating the pilot's results. Results in Brief
    Between January 1998 and March 1999, 223 servicemembers have used
    the pilot program to arrange their own move rather than use the
    current program.  Participation has been relatively low compared
    to the other three pilots under way or planned, which involve
    substantially more shipments- approximately 1,400 to 45,000
    shipments annually.  Survey data indicate that participants are
    satisfied with the pilot and would use the option again. Because
    the pilot offers servicemembers a choice between the current
    program and arranging their own move, implementing the option
    increases the workload for local personal property officials. The
    member arranged move option is not featured in any of the other
    pilots, which are broader in scope and are intended to replace the
    current 1Defense Transportation:  Efforts to Improve DOD's
    Personal Property Program (GAO-T/NSIAD-99-106, Mar. 18, 1999).
    2House Report 104-863 (Sept. 28, 1996) p. 865. Page 1 GAO/NSIAD-
    99-138  Defense Transportation B-282502 program.  While the U.S.
    Transportation Command is responsible for evaluating all of the
    pilots to determine which one could provide better long-term
    results, its plan for doing so has not been finalized.  In
    addition, the U.S. Transportation Command's draft evaluation plan
    proposes to collect information on the extent the Navy pilot works
    as an option within the current program at a few Navy sites, which
    may not provide adequate data to assess the Navy pilot's
    feasibility or its compatibility with the other pilots' results.
    Consistent with the recommendation in our report on the Army's
    Hunter pilot (that DOD should develop a comprehensive strategy for
    testing each of the approaches), we are recommending that, in
    developing this strategy, the Secretary of Defense consider
    testing the Navy pilot and/or its unique features in conjunction
    with one or more of the other pilots.3 Background    The Navy
    pilot allows servicemembers the option of either participating in
    the current program or of selecting their own carrier from a list
    of approved small business carriers.4  Key features of the
    servicemember arranged move include shipment location information,
    direct claims settlement with the carrier, full replacement value
    for lost or damaged household goods, and payment via the
    government charge card.  Initiated in January 1998, the pilot
    included only shipments originating at Puget Sound, Washington,
    headed to San Diego, California; Norfolk, Virginia; New London,
    Connecticut; Pensacola, Florida; and Jacksonville, Florida.  To
    increase participation, the pilot's origin site was expanded from
    Puget Sound (including Whidbey Island and Everett, Washington) to
    include San Diego, Norfolk, and New London in July 1998.  However,
    due to potential competition with another pilot that will also
    serve the Florida area, Pensacola and Jacksonville remain as
    destination sites only.  The pilot does not have a specific end
    date; however, DOD established a target in the November 1997
    Defense Reform Initiative Report to offer the Navy option to every
    servicemember by January 1, 2000. The Navy pilot is one of four
    quality-of-life initiatives DOD is pursuing to change the way it
    is currently doing business, adopt commercial business 3Defense
    Transportation:  Army's Hunter Pilot Project Is Inconclusive but
    Provides Lessons Learned (GAO/NSIAD-99-129, June 23, 1999). 4In
    the motor freight and transportation industry, firms with annual
    gross revenues of $18.5 million or less are classified as small
    businesses.  See Federal Acquisition Regulation 19.102. Page 2
    GAO/NSIAD-99-138  Defense Transportation B-282502 practices, and
    achieve quality moving services for military families.  The other
    three pilots are described below. *The Army's pilot at Hunter Army
    Airfield, Savannah, Georgia, which was initiated in February 1996,
    is a quality-of-life effort to improve the relocation process and
    to test commercial practices in a military environment.  Services
    provided by the contractor include point-to-point move management
    with a single point of contact for the member, assistance in
    buying/selling a residence, full replacement value for lost or
    damaged household goods, direct claims settlement with the
    servicemember, and visibility of the shipment for approximately
    1,400 annual moves.  The contract will end on September 30, 1999.
*In January 1999, the Military Traffic Management Command (MTMC),
    which manages the current personal property program, began a pilot
    involving 50 percent of the moves in North Carolina, South
    Carolina, and Florida, which will total approximately 18,500 moves
    annually.  Key features of the pilot include the selection of
    carriers based on servicemember satisfaction and past performance
    rather than on price alone; achieving stronger carrier commitment
    through long-term contracts; offering full-value replacement
    protection and direct claims settlement to users.  The pilot is
    planned to run for 3 years, ending September 2002. *DOD announced
    on February 12, 1999, that it intended to begin the Full Service
    Moving Project as a fourth test.  This pilot would be similar to
    the pilot at Hunter Army Airfield, with modifications based on the
    Army's lessons learned, and it would involve a larger number of
    moves (approximately 45,000 annually).  The pilot would be tested
    in the National Capital (Washington, D.C.) Region, Georgia, and
    North Dakota. Like the Army pilot, it is intended to replace the
    existing program by using a contractor or contractors to provide
    both transportation and move management services.  No official
    start date has been set for this pilot program. Status of the Navy
    Pilot As of March 1999, 223 servicemembers have participated in
    the pilot.  In its second year of operation, the pilot has been
    expanded from one to four shipment sites.  Program results thus
    far show (1) participation has been relatively low compared to the
    other pilots that are under way or planned, (2) servicemembers are
    satisfied arranging their own move, according to limited survey
    data, and (3) workload would increase for personal property
    officials because this option adds an alternative to the current
    program. Page 3 GAO/NSIAD-99-138  Defense Transportation B-282502
    Servicemember           As of March 26, 1999, a total of 223
    members have participated in the pilot Participation Is Low    and
    132 moves have been completed.  Participation has been low, in
    part, because of the pilot's short duration and eligibility
    restrictions.  Although the Puget Sound office began to offer this
    option in January 1998, the other three sites did not offer it
    until late July 1998, and then only for Navy military members
    moving to one of six destinations.  The pilot's eligibility
    restrictions exclude civilians and members of the other services.
    Also, only certain types of domestic household goods shipments are
    eligible. The pilot excludes shipments of boats and mobile homes
    as well as shipments from non-temporary storage or from a mini-
    warehouse.  Shipments must weigh at least 3,000 pounds and are
    expected to cost between $2,500 and $25,000. As of the end of
    calendar year 1998, the participating personal property offices
    reported interviewing 1,083 Navy members that were moving to or
    from the six pilot sites to determine whether their shipments met
    pilot eligibility criteria.  Some 573, or 53 percent, of them had
    eligible shipments and 133, or 23 percent, selected the option.
    Over 90 percent of those determined as ineligible had household
    goods not weighing at least 3,000 pounds.  Site officials reported
    that about 70 percent of eligible members cited the effort
    involved in selecting a carrier as the primary reason that they
    did not use the option (see table 1 for participant information by
    site).5 Table 1:  Number of Participants by Site and Program
    Status, 1998 Puget Soundd          Norfolk       New London
    San Diego       Total Interviewsa                             420
    145                 247              271       1,083 Eligibleb
    149            93                115              216          573
    Select pilotc                            78            15
    19                21         133 Move completed
    54              5                  9               10           78
    Move in-process                          24            10
    10                11           55 a"Interviews" includes only
    those servicemembers who planned to move to a participating site.
    b"Eligible" includes servicemembers who had shipments that met the
    pilot's eligibility restrictions. c"Select pilot" includes
    servicemembers that participated in the pilot program. dThese
    figures include shipments from the Whidbey Island Naval Air
    Station.  Also, Puget Sound began offering the option in January
    1998, whereas the other sites began in July 1998. 5After October
    1998, pilot officials discontinued the requirement for sites to
    report the reasons for ineligibility and for members not selecting
    the pilot because data showed the same dominant reasons. However,
    sites still report the overall number of Navy members interviewed,
    eligible, and participating. Page 4
    GAO/NSIAD-99-138  Defense Transportation B-282502 Navy officials
    have recently modified eligibility requirements to increase
    participation in the pilot.  The major changes would increase
    eligibility by (1) reducing the weight minimum from 3,000 pounds
    to 1,000 pounds, (2) allowing boats, and (3) eliminating current
    restrictions to allow shipments to any domestic destination.
    Limited Customer Survey            To measure customer
    satisfaction with the pilot, the Navy uses a Results Indicate
    Satisfaction  nine-question customer survey that servicemembers
    are asked to fill out With Pilot                         and
    return after the move is completed.  The survey includes questions
    pertaining to pickup and delivery time, loss, damage, and overall
    satisfaction.  The survey asks, among other things, if the move
    was of a better quality than the servicemembers' prior move and if
    the customer would (1) choose the pilot again, (2) recommend it to
    someone else, and (3) use the same carrier again. As of February
    1999, the Navy had received 30 customer satisfaction surveys.
    These surveys are predominantly from the Puget Sound personal
    property office because Puget Sound was the first site to offer
    the pilot.  Of the 30 surveys, 23 indicate that the pilot was a
    better quality move than other military moves.  Eighteen of the 30
    shipments (60 percent) had no claims, and 12 had a claim.  Of the
    12 claims made, only 2 exceeded $500. One of these claims was for
    $1,900 for broken china.  Despite this damage, the customer
    responded that this was a better quality move than prior ones and
    stated that they would use the same carrier and the pilot again
    (see table 2 for selected survey responses).6 Table 2:  Summary of
    Survey Responses to Selected Questions Question
    Yes               No          No difference Better quality move?a
    23                 4                      1 Use pilot again?
    27                 3 Any claims?
    12               18 Recommend the pilot?
    27                 3 aThe numbers in this row do not add to 30
    because two surveys did not contain the question regarding whether
    the servicemember arranged move was a better quality move than
    other military moves. 6The carriers, not the government, settle
    the claims directly with servicemembers. Page 5
    GAO/NSIAD-99-138  Defense Transportation B-282502 Increased
    Workload at            While the pilot offers advantages to the
    servicemember, it would add to the Personal Property Offices
    workload of the personal property officials who are responsible
    for the bulk of day-to-day program management.  Among their
    additional duties, or duties that were previously handled by MTMC,
    are negotiating agreements with participating small business
    carriers, providing individual counseling to potential
    participants, maintaining up-to-date carrier information and
    performance data, and tracking customer satisfaction survey
    results. These are additional duties not previously handled by the
    personal property office. Local contracting officers at each
    participating installation enter into agreements with companies
    that offer acceptable discounts off of commercial rates.  The
    agreements are entered into with companies that are self-certified
    as small businesses and on MTMC's approved list.  These agreements
    provide for the use of a government charge card for simplified
    payment and for direct claims settlement with the carrier at full
    repair or replacement value at no additional cost to the
    government.  In addition, carriers are required to provide
    information on a shipment's location through the use of a toll-
    free help line and a pager for direct delivery notification, which
    are designed to improve service and reduce storage costs. The
    personal property office provides servicemembers with the names of
    participating movers that have been determined to have reasonable
    prices. The property offices also maintain carrier quality books
    that contain a carrier's performance history, the returned
    customer satisfaction surveys, and the carrier's marketing
    materials.  Participating servicemembers are required to contact
    at least three moving companies and document the basis for their
    preference of one of the carriers.  The contracting officer can
    then make the selection considering both price and the
    servicemember's recommendation concerning non-price factors such
    as quality.  The award is made to the firm selected under the
    simplified acquisition procedures contained in Federal Acquisition
    Regulation part 13. Plan for Evaluating the  The Navy does not
    plan to conduct an evaluation of the pilot program Pilot Is
    Incomplete              separate from its own evaluation.  The
    U.S. Transportation Command (USTRANSCOM) is currently in the
    process of developing and coordinating an evaluation plan with the
    services to evaluate the personal property pilot tests, and it
    will make a recommendation as to the follow-on course of action.
    In this regard, the Navy has modified its customer survey
    questions Page 6                                   GAO/NSIAD-99-
    138  Defense Transportation B-282502 so that the survey questions
    match the USTRANSCOM survey questions. Navy officials are also
    accumulating pilot project transportation costs and developing a
    methodology to compare these costs with those that the government
    would have otherwise paid. Presently, the pilot has only been
    tested as an option to the current program at a few Navy sites and
    not as an option at the other pilot sites. The other pilots are
    designed to test an approach to replace the current program.
    However, the USTRANSCOM draft evaluation plan does not address the
    feasibility or potential benefits of incorporating the pilot
    option into the pilots that may replace the current program.
    Further, the evaluation plan does not directly assess the unique
    aspects of the pilot, which include limiting carrier selection to
    small business carriers and using the government charge card for
    payment.  Consequently, DOD may not have the information it needs
    to craft a DOD-wide personal property program. Navy officials are
    concerned about several aspects of the USTRANSCOM evaluation plan.
    In March 1999, they stated that the data elements to be collected
    should be better defined, that a consistent evaluation time period
    should be established, and that expert advice should be sought.
    The officials also believe that cost comparisons of overhead will
    be difficult because the same personnel who administer the current
    program are implementing the Navy pilot.  In our recent testimony
    and report on the Army's Hunter pilot and in our comments on
    several draft evaluation plans, we have stated similar concerns
    about the current evaluation plan.  These concerns include the
    need to (1) develop a comprehensive strategy for testing the
    unique characteristics and/or processes of each pilot and (2) use
    expert advice in finalizing a methodologically-sound evaluation
    plan. Conclusions    The Navy pilot program differs from other
    ongoing or proposed pilot programs because it adds an option to
    the current program; it is not intended to replace the current
    program.  The pilot participation levels and results thus far
    provide general information about the program's potential benefits
    and customer satisfaction compared to the current program- which
    DOD is proposing to replace.  Since the Navy pilot option is not
    to be integrated and tested with the other pilots, information on
    the viability of providing this option will only be available in
    comparison to the current program.  Further, the draft evaluation
    plan does not identify a specific method for assessing the pilot's
    unique features. Page 7
    GAO/NSIAD-99-138  Defense Transportation B-282502 Recommendation
    Consistent with the recommendation in our report on the Army's
    Hunter pilot that the Secretary of Defense develop a comprehensive
    strategy for testing each of the approaches, we recommend that the
    Secretary of Defense, in developing this strategy, consider
    testing the Navy pilot and/or its unique features in conjunction
    with one or more of the other pilots. Doing so would test the Navy
    pilot in an environment that is more consistent with the changes
    being considered and likely to be implemented. Agency Comments
    DOD stated that it concurred with the report and its
    recommendations. Specifically, DOD stated that, as part of its
    plan to develop a comprehensive strategy for evaluating each of
    the pilots, it would determine-in concert with the services-how
    best to incorporate the features of the Navy pilot into the other
    pilots.  DOD's comments are reprinted in appendix I. Scope and
    To determine how the Navy plans to implement and evaluate the
    pilot Methodology        program, we reviewed available program
    documents and met with program management officials at the Naval
    Supply Systems Command, Mechanicsburg, Pennsylvania.  We discussed
    how the pilot is being implemented and managed, the contract terms
    and conditions, and the key characteristics of the pilot. Further,
    we discussed and reviewed the customer survey results and other
    data that will be provided to USTRANSCOM for its evaluation of the
    pilot.  We also discussed with Navy Supply Systems Command and
    Naval Audit Service officials their concerns with the USTRANSCOM
    evaluation plan. We visited and interviewed officials at the Fleet
    and Industrial Supply Center, Puget Sound, Bremerton, Washington,
    to determine how the pilot was implemented at its first location.
    We discussed the process used to establish and manage the pilot at
    Puget Sound while continuing to operate the current system.
    Additionally, we discussed their experiences and observations with
    the pilot.  We also visited and interviewed program, contracting,
    and government charge card officials at the Fleet and Industrial
    Supply Center, Norfolk, Virginia, to determine how implementation
    was progressing and to compare the pilot operations at Puget Sound
    to those of Norfolk. Page 8
    GAO/NSIAD-99-138  Defense Transportation B-282502 To provide
    information on the pilot's progress, we visited and interviewed
    Naval Supply Systems Command officials to determine how sites were
    selected for expansion after the pilot was established at Puget
    Sound. Further, we attended a "lessons learned" conference that
    included personal property officials from all of the pilot sites
    as well as from the Navy Transportation Support Center, Bureau of
    Naval Personnel, and the Defense Finance and Accounting Service to
    understand the problems these organizations encountered with the
    pilot and possible solutions. To obtain information on pilot
    participation, we reviewed weekly reports summarizing the number
    of military members that were eligible and those that were
    selected to participate.  We also discussed the reasons for their
    participation with site officials. To provide information on
    available cost data, we reviewed the cost comparisons developed by
    site officials that compare actual pilot transportation costs to
    estimated/constructed costs for the same move under the current
    system.  We did not independently verify any data reported by
    pilot sites or Naval Supply Systems Command headquarters. Our
    review was conducted between October 1998 and April 1999 in
    accordance with generally accepted government auditing standards.
    We are sending copies of this report to the Honorable William S.
    Cohen, Secretary of Defense; the Honorable Richard Danzig,
    Secretary of the Navy; General Charles T. Robertson, Jr.,
    Commander in Chief, USTRANSCOM; Rear Admiral Donald E. Hickman,
    Commander, Naval Supply Systems Command; Major General Mario F.
    Montero, Jr., Commander, MTMC; and the Honorable Jacob J. Lew,
    Director, Office of Management and Budget. We will also make
    copies available to others upon request. Page 9
    GAO/NSIAD-99-138  Defense Transportation B-282502 Please contact
    me at (202) 512-8412 if you or your staff have any questions
    concerning this report. Major contributors to this report are
    listed in appendix II. David R. Warren, Director Defense
    Management Issues Page 10
    GAO/NSIAD-99-138  Defense Transportation B-282502 List of
    Congressional Committees The Honorable John Warner Chairman The
    Honorable Carl Levin Ranking Minority Member Committee on Armed
    Services United States Senate The Honorable Ted Stevens Chairman
    The Honorable Daniel K. Inouye Ranking Minority Member
    Subcommittee on Defense Committee on Appropriations United States
    Senate The Honorable Floyd Spence Chairman The Honorable Ike
    Skelton Ranking Minority Member Committee on Armed Services House
    of Representatives The Honorable Jerry Lewis Chairman The
    Honorable John P. Murtha Ranking Minority Member Subcommittee on
    Defense Committee on Appropriations House of Representatives Page
    11                             GAO/NSIAD-99-138  Defense
    Transportation Contents Letter
    1 Appendix I
    14 Comments From the Department of Defense Appendix II
    15 Major Contributors to This Report Tables
    Table 1:  Number of Participants by Site and Program Status, 1998
    4 Table 2:  Summary of Survey Responses to Selected Questions
    5 Abbreviations DOD              Department of Defense MTMC
    Military Traffic Management Command USTRANSCOM U.S. Transportation
    Command Page 12                               GAO/NSIAD-99-138
    Defense Transportation Page 13    GAO/NSIAD-99-138  Defense
    Transportation Appendix I Comments From the Department of Defense
    Appendix I Page 14           GAO/NSIAD-99-138  Defense
    Transportation Appendix II Major Contributors to This Report
    Appendix II National Security and  Charles I. Patton, Jr.,
    Associate Director International Affairs          Nomi R. Taslitt,
    Assistant Director Robert L. Self, Senior Evaluator Division,
    Washington, D.C. Office of the General          John G. Brosnan,
    Assistant General Counsel Counsel, Washington, D.C. Norfolk Field
    Office           Daniel A. Omahen, Evaluator-in-Charge John R.
    Beauchamp, Senior Evaluator (709369)         Letter        Page 15
    GAO/NSIAD-99-138  Defense Transportation Ordering Information The
    first copy of each GAO report and testimony is free. Additional
    copies are $2 each.  Orders should be sent to the following
    address, accompanied by a check or money order made out to the
    Superintendent of Documents, when necessary, VISA and MasterCard
    credit cards are accepted, also. Orders for 100 or more copies to
    be mailed to a single address are discounted 25 percent. Orders by
    mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC
    20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts.
    NW) U.S. General Accounting Office Washington, DC Orders may also
    be placed by calling (202) 512-6000 or by using fax number (202)
    512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of
    newly available reports and testimony.  To receive facsimile
    copies of the daily list or any list from the past 30 days, please
    call (202) 512-6000 using a touchtone phone.  A recorded menu will
    provide information on how to obtain these lists. For information
    on how to access GAO reports on the INTERNET, send an e-mail
    message with "info" in the body to: [email protected] or visit
    GAO's World Wide Web Home Page at: http://www.gao.gov United
    States                       Bulk Rate General Accounting Office
    Postage & Fees Paid Washington, D.C. 20548-0001            GAO
    Permit No. GI00 Official Business Penalty for Private Use $300
    Address Correction Requested

*** End of document. ***